Tuesday, November 10, 2015

Application Services 6.x service blueprint deployment error - ERROR: C:\opt\vmware-appdirector\agent\appd.properties not found. Abort


This error is very common error and very irritating, the application deployment fails after very long wait on Node Setup stage and when you see the Deployed VM you would find the below error at C:\opt\vmware-appdirector\agent\logs

INFO 2015-11-10 16:56:17,468 - C:\opt\vmware-appdirector\agent\appd.properties not detected.
INFO 2015-11-10 16:56:17,468 - Wait for 2 seconds before retry ...

And after 150150 seconds, retrying every 2 seconds it aborts

INFO 2015-11-10 16:57:03,578 - Wait limit 150150 seconds over. Giving up.
INFO 2015-11-10 16:57:03,578 - ERROR: C:\opt\vmware-appdirector\agent\appd.properties not found. Abort.
INFO 2015-11-10 16:57:03,734 -
INFO 2015-11-10 16:57:03,734 -
INFO 2015-11-10 16:57:03,749 - Process exited with code 1

And on Application Services execution plan of Deployments will remain at Node Setup for several minutes and fails


There could be multiple reasons for this, before trying below steps make sure you have prepared the Windows Template as recommended in the below VMware document
 http://pubs.vmware.com/vra-62/topic/com.vmware.ICbase/PDF/vrealize-automation-62-using-application-services.pdf page 52

If correct JRE version, Vmware Cloud Application Director Agent Bootstrap service and VCACGuestAgentService (vCAC agent service) is installed and could be restarted without errors in the Template then stop both the services and proceed further.

Check: If the windows machine deployed is getting the proper IP and networking, sometime the VM doesn’t pickup the IP from Customization Specification. If that is the case then create a new customization specification and run again or try the blow steps to reconfigure the networking configuration again

You can find customization specification in vCenter Server at Home>Customization Specifications Manager

01: In the networking section, select Custom settings instead of Typical settings

02: If it was already on Custom settings then delete the default network interface and Click on New

03: By default it is set on DHCP, click on the setting button available on top right of network interface button

04: Now Select the DNS tab and update the DNS information, Save the settings and complete the Customization Specification Wizard

Now request Application Service from vRealize Automation service Catalog and see if get through that stage, if it still continues then follow the below steps

Ideally in the application services blueprint (Template), the vCAC guest agent is installed automatically by  VMware Cloud Application Director Agent Bootstrap service. So as per the Vmware documents, you would just  place the unzipped files to their designated location and run the Bootstrap service. In this method we would manually install the services. For that first lets uninstall all the services

Uninstalling the Services (Cleanup)

  • Connect to your Template (Convert to VM if Template or Power On the VM is using Linked Clone), and run the Command Prompt with Admin Privilege
  • Go to Temp directory where you extracted the Vmware Cloud Application Director Agent Bootstrap service  and run the below Batch file

This batch file  will remove the files and folder from "C:\opt\". Check the respective service has been removed, sometimes it stays on disabled mode, you may wait or restart the Windows if it stays at disabled mode.
  • Now go to uninstall the vCAC guest agent service by going to its extracted location and running the below command, i.e.,
winService.exe -u

At this point, I would also delete all the extracted files and start over again, delete the files from

Lets start from the beginning, reboot the Windows  and download the required files

Find all the agents on Application Services Appliance @ /home/darwin/tcserver/darwin/webapps/darwin/agent

  1. JRE 1.7 http://ip of Application Services appliance/agent/jre-1.7.0_72-win64.zip
  1. VMware Cloud Application Director Agent Bootstrap service http://ip of Application Services appliance/agent/vmware-appdirector-agent-bootstrap-windows_6.0.0.0.zip
  1. Windows vCAC Guest Agent  https://ip_of_vRealize_Automation_appliance:5480/installer

Important: Before extracting the downloaded files, disable the security by going to the properties of the zip file and click on "UNBLOCK", if you miss this step then certainly you will encounter above error message.

  1. Extract the JRE 1.7 files to "C:\opt\vmware-jre"
Confirm the Java version by below command

  1. Extract the VMware Cloud Application Director Agent Bootstrap service to Temporary folder  "C:\Temp", also place the "NTRights.exe" file in the same location. If you do not have NTRights.exe file then download "Windows 2003 Resource Kit", then copy NTRights.exe from this kit. If you do not use NTRights.exe for bootstrap service delegation then you are instructed to manually delegate .\darwin user to the Bootstrap service.

Now install this service by running below command

install.bat password=your_desired_password cloudProvider=vcac vcacServer=fqdn_of_IaaS_Server httpsMode=true

There shouldn’t be any error after the above command runs

Check if bootstrap service is installed and delegated to  .\darwin
You can reset the .\darwin user's password by going to Start>>RUN>>lusrmgr.msc, use this if the service doesn't start with the given password

  1. Extract Windows vCAC Guest Agent to "C:\" and
Cd C:\VRMGuestAgent
Run the below command
WinService -I -h fqdn_of_IaaS:443 -p ssl

Make sure the above services are installed and confirm if you can start the services. Stop both services now and power off the VM.

Create a new snapshot for linked clone or convert the VM to template for full clone and proceed deploying the application now. Hope these steps are helpful, as I was lost and spent much more time troubleshooting. 

Tuesday, August 25, 2015

Installing and Configuring vRealize Operations Manager 6.0 (vROM) AKA vCenter Operations Manager

I am deploying vRealize Operations Manager 6.0 which is completely different from all its previous versions. The configuration is completely changed, looks very simple and light. I am setting up single vROM, meaning the master node must also perform data collection and analysis because it is the sole node, and the only place where vRealize Operations Manager adapters are installed. For distributed or multi-node  installation those roles could designated to different nodes.

  1. Master Node: Required node in the cluster which performs all the actions.
  2. Data Node: (Optional) - which has adapters installed to perform collection and analysis. Suitable for larger Deployments.
  3. Replica Node: (Optional) - Replica of Master Node for High Availability. Suitable for larger Deployments.
  1. Remote Collector Node: (Optional) - Distributed deployments might require a remote collector node that can navigate firewalls, interface with a remote data source. Only gathers objects for inventory and doesn’t store or perform any analysis. Suitable for Distributed Installation.

vROM sizing calculator could be found here

vROM could be installed and configured on

  1. As an Vmware Appliance (From vCenter not through ESXi Host)
  1. Windows Operating System
  1. Linux Operation System.

Check the below guide for installing on Windows or Linux Operating System, I am deploying as an Appliance using .ova file.

Deploy the vROM Appliance:

  1. Connect to vCenter Server, click on File and select deploy from OVF Template and locate the ova file and select next.
  2.  Accept the License and give the name for vRealize Operations Manager Appliance.  Select the appropriate Cluster, Host and Datastore.
  3. Select the appropriate deployment Configuration and complete the wizard to start the deployment.
  4. Select appropriate Network Mapping and complete the network configuration details. It is always static IP for Master and Replica Node.

Additional nodes as mentioned above could be deployed following the same above steps. I am setting up this solution for smaller environment so I would only use single Master Node.

Master Node Configuration:

  1. Enter the FQDN or IP of the appliance in browser. You are presented with 3 options as mentioned below, the getting started tab for each of the option has very nice flow diagram which gives you a high level idea how its going to work

  1. Express Installation
  1. New Installation
  2. Expand Existing Installation

I am going with the New Installation, so I would be creating a new vRealize Operations Manager cluster with this node as the master node. 

  1. Set the Administrator Password and click on Next.
  2. I am using the default certificates which came with vROM. If you have server certificate of your own to be used for secure communication, you may upload here.
  3. Enter the name for Master Node and NTP server details. Click on Next and then Finish to complete the configuration.
  4. Now you can add more nodes or enable services for data collection on master node.

Import Data from Existing vCenter Operations Manager or Configure to collect data from environment

Existing vCenter Operations Manager can be imported to new deployment and when you turn off the earlier deployment, monitoring of the inventory and data is done solely by the new version of vRealize Operations Manager.

  1. I would choose "New Environment" option to monitor my other solutions.

This how it goes if you choose "Import Environment" option.

  1. Accept the License Agreement, enter the License Key and Click on Finish to complete the configuration.

Add Solutions to be managed by vRealize Operations Manager

  1. Click on Solutions, and then Configure. I'm adding 2 vCenter details here.
  2. IN the next step you define what all needs to monitored, I am going with all default options.
  3. Click on Finish to complete the configuration.

It would take some time to collect the performance Data from added vCenter Server. This concludes the configuration, the new version of Operations Manager is smooth and works like a charm. You just need one 1 IP to setup the Single Node environment, I hope this post was informative. 

Monday, August 24, 2015

Prepare Linux 6.5 (RHEL) Template for vRealize Automation Application Services (Application Director)

Regular vSphere Template cannot be directly used in vRealize Application Services, we need to install the agent and prepare it for to be used with Application Services. In this post, I am using RHEL 6.5, the steps would remain same for other flavours of linux

  1. Prepare a Linux VM, I am using RHEL 6.5. Login with root.
  1. Download the installer script http://Application_Services_Server_IP/tools/preparevCACTemplate.sh to the Linux machine. You can use the wget or scp commands to download the file.
  1. Ensure that the installer script has permission to execute.
chmod +x ./preparevCACTemplate.sh
  1. Run the installer script.
  1. Enter the vRealize Automation Appliance Host Name or IP
  2. Enter the IaaS  Manager Services Server Hostname or IP
  1. Enter the Application Services Server Hostname or IP
  1. Next prompt is to Check Certificate and Download & Install Java Runtime Environment 1.7.0_72

Make sure you get the confirmation
# Installation Completed Successfully                  #
# Ready to capture as a template                         #
  1. Make sure if the Application Director agent service is available
 rpm -qa | grep -i appd

  1. Shut down the Linux virtual machine.

For this VM to be added as Template on Application Services (Application Director), we need to publish this as Blueprint from vRealize Automation (vCAC). You could use the Full Clone or Linked Clone method in the Blueprint. I am using Linked Clone to save storage and time.

  1. Take a snapshot of this VM and give the name & description appropriate.
  2. Connect to your vRealize Automation console and got to Infrastructure>>Blueprints and enter respective information.
  3. Under Build Information, Select "Linked Clone" for Action and select the snapshot of respective VM from "Clone From" option. Rest of the options are self explainatory.
  4. Now hover the mouse over Blueprint name and click on Publish.
  5. Now go to Infrastructure>>Compute Resources, Hover the mouse on respective vCenter(Endpoint) and click on Data Collection. Click Request Now for Inventory, State and Performance.

This will make this template available for vRealize Automation Services (Application Director)