Coming soon on this blog, vRA pickup lines. Watch this space!

Deploying OVFs with vRealize Automation

vRA is good at only a few things out of the box – primarily VM template based blueprints.

One thing that doesn’t work at all out of the box is deploying OVF/OVA machines, which I will cover a way to do so in this post. There are many ways to do this thanks to vRO, but I think this way is fairly easy to digest, using appliances a vRA admin would typically have access to.

Use Case

For me it is simple – deploying a fully functional vSphere or vRealize environment requires a number of appliances. Being able to build a full reference VSAN environment with automation is pretty cool and nice to be able to hand back to other members of my team for testing or experimentation. Deploying vRO/vRA is a lot more complex, and with the 7.2 releases of these solutions, you can do the whole thing via API calls.

Caveats and Assumptions

Not all appliances are equal! IHV or ISV appliances often tend to only set an IP address and not enable post-configuration, so while this will showcase a pretty end-to-end solution, other stuff outside of VMware appliances may not work as well.

Also, this post assumes vRA is already set up and is working pointed to an existing vCenter Server for data collection.

This also assumes you have a method of allocating IP addresses in place, as well as a process to pre-create appropriate DNS records for the appliances  you build.

Setting the stage

For this particular post, I will detail deployment of a vCenter Server Appliance, version 6.0U2. Some parameters may change between versions but not too much.

When you download the ISO for the vCenter Server Appliance, extract the contents to a temporary folder. Search inside the extracted content in the folder VCSA to find the actual OVA file called vmware-vcsa.ova.

Launch the vSphere Client and deploy the OVA to your choice of datastore/host.

You’ll be prompted during the wizard to populate numerous values that would normally be handled during the VCSA UI Installer – leave them alone and continue.

Do not power on the appliance as part of the process!

Once the OVA has been deployed, simply convert the new appliance into a template.

Build your VCSA blueprint

In vRA, execute a data collection against your vCenter Server resources.

Once that is successful, go to Design->Blueprints and create a new composite blueprint.

Drag a vSphere Machine object onto the canvas, and set the names and descriptions as you like.

In the Build Information tab, specify these values

  • Blueprint Type – Server
  • Action – Clone
  • Provisioning Workflow – CloneWorkflow
  • Clone From – <The template>
  • Customization Spec – <blank>

Lastly, in case it is not part of your existing Property Groups (aka Build Profiles), you will want to pass this property in:

  • Extensibility.Lifecycle.Properties.CloneWorkflow = *,__*

Most documents emphasize passing the VMPSMasterWorkflow32 properties in as part of setting up Event Broker, so this is just calling out an additional requirement.

From here, set your custom properties and build profiles as you like. One recommendation I have for vRA admins is to add a custom property on your endpoint that denotes the vCenter Server UUID – this helps identify which vCenter your machine deployed to, which is needed for steps coming up. If you have a single vCenter, this is optional, but you may want to put it in place now, just in case!

Publish and entitle your blueprint.

Create vApp properties workflow in vRO

The key to making this work is all about the vApp properties of the virtual machine once deployed. Once the VM is cloned from the OVF template you made earlier, but before the VM starts up, you need to be able to inject your custom properties at the vCenter VM level so that the appliance can bootstrap itself and initiate firstboot scripts successfully.

In vRO, create a new workflow. Add an input parameter named payload of type Properties. This will hold the content that the vRA Event Broker will use to populate your cloned VM with what it needs to build properly.

Create an attribute called vc of type VC:SDKConnection and set the value to your vCenter Server.
Create an attribute called vcVM of type VC:VirtualMachine and leave it empty.
Create an attribute called iaasProperties of type Properties and leave it empty.

In the workflow schema, create a Scriptable Task and bind the payload input parameter to it, as well as the iaasHost, and vc attributes on the input side. Bind the vcVM and iaasProperties attributes on the output side of the task.

Insert the following code:

// create search query IaaS for the VM and its properties.
var properties = new Properties();
properties.put("VirtualMachineID", payload.machine.id);
// query IaaS for entity
var virtualMachineEntity = vCACEntityManager.readModelEntity(iaasHost.id, "ManagementModelEntities.svc", "VirtualMachines", properties, null);
// now get the properties you need for the applianceType value
var vmProperties = new Properties()
var props = virtualMachineEntity.getLink(iaasHost, "VirtualMachineProperties");
for each (var prop in props) { 
vmProperties.put(prop.getProperty("PropertyName"), prop.getProperty("PropertyValue"));
}
// get your output to an attribute
iaasProperties = vmProperties
// construct the Managed Object Reference to find the VM object in vCenter
var object = new VcManagedObjectReference()
object.type = "VirtualMachine"
object.value = payload.machine.externalReference

// query the vCenter for the actual object
vcVM = VcPlugin.convertToVimManagedObject(vc,object)

The above code should be fairly straightforward – it is pulling in your IaaS custom properties into a vRO object, and it is also finding the vCenter Virtual Machine so that you can manipulate it. Next will be the code that updates the OVF Properties of the VM in question.

Updating the OVF Properties

Create a Scriptable Task in the schema of your workflow.

On the input side, bind the values iaasProperties and vcVM.

Paste the following code into the task:

// create the properties variable to be called later
var ovfProps = new Properties()
ovfProps.put("guestinfo.cis.appliance.net.addr.family","ipv4")
ovfProps.put("guestinfo.cis.appliance.net.mode","static")
ovfProps.put("guestinfo.cis.appliance.net.addr",iaasProperties.get("VirtualMachine.Network0.Address"))
ovfProps.put("guestinfo.cis.appliance.net.prefix",iaasProperties.get("VirtualMachine.Network0.SubnetMask"))
ovfProps.put("guestinfo.cis.appliance.net.gateway",iaasProperties.get("VirtualMachine.Network0.Gateway"))
ovfProps.put("guestinfo.cis.appliance.net.dns.servers",iaasProperties.get("VirtualMachine.Network0.PrimaryDns"),iaasProperties.get("VirtualMachine.Network0.SecondaryDns"))
ovfProps.put("guestinfo.cis.appliance.net.pnid",iaasProperties.get("Hostname")+".domain.lab")
ovfProps.put("guestinfo.cis.appliance.net.ports","{}")
ovfProps.put("guestinfo.cis.vmdir.password","VMware1!")
ovfProps.put("guestinfo.cis.vmdir.domain-name","vsphere.local")
ovfProps.put("guestinfo.cis.vmdir.site-name","vRA-Provisioned-Default-Site")
ovfProps.put("guestinfo.cis.vmdir.first-instance","True")
ovfProps.put("guestinfo.cis.db.type","embedded")
ovfProps.put("guestinfo.cis.appliance.root.passwd","VMware1!")
ovfProps.put("guestinfo.cis.appliance.ssh.enabled","True")
ovfProps.put("guestinfo.cis.appliance.time.tools-sync","False")
ovfProps.put("guestinfo.cis.appliance.ntp.servers","ntp.domain.lab")
ovfProps.put("guestinfo.cis.upgrade.silent","True")
ovfProps.put("guestinfo.cis.kv.new","True")
ovfProps.put("guestinfo.cis.silentinstall","True")
ovfProps.put("guestinfo.cis.clientlocale","en")

// construct the reconfiguration specification, which will add the vApp Properties from IaaS to your VM.
var vmspec = new VcVirtualMachineConfigSpec()
vmspec.vAppConfig = new VcVmConfigSpec()

var newOVFProperties = new Array()
// get the existing VM's OVF property list.
// this is needed to match the 'key' value to the 'id' and subsequently 'value' parameters. The 'key' is required.
var vmProperties = vcVM.config.vAppConfig.property
for each(prop in vmProperties) {
// get the key value
var key = prop.key
// define new property spec, which will edit/update the existing values.
var newProp = new VcVAppPropertySpec()
newProp.operation = VcArrayUpdateOperation.fromString("edit")
newProp.info = new VcVAppPropertyInfo()
// assign the new key
newProp.info.key = key
// get the values from ovfProps - these are of type VcKeyValue, properties are .key and .value
newProp.info.id = prop.id
newProp.info.value = ovfProps.get(prop.id)
newProp.info.userConfigurable = true
// add the new OVF Property to the array
newOVFProperties.push(newProp)

// assign the properties to the spec
vmspec.vAppConfig.property = newOVFProperties
// now update the VM - it should be OFF or this will fail
try {
// reconfigure the VM
var task = vcVM.reconfigVM_Task(vmspec)
System.log("OVF Properties updated for "+vcVM.name+".")
} catch(e) {
throw("Error updating the OVF properties: "+e)
}

Save your workflow and exit.

With the workflow completed, now we need to go into the Event Broker and create a subscription so that this workflow will be executed at the correct time during the provisioning process.

Setting up the Event Broker Subscription

In vRA, go to Administration->Events->Subscriptions, and create a new subscription.

For the Event Topic, choose Machine Provisioning and click next.

For the Conditions, you will want to set it up with All of the following and at least these values:

  • Data->Lifecycle State->Name equals CloneWorkflow.CloneMachine
  • Data->Lifecycle State->Event equals CloneWorkflow.CloneMachine.EVENT.OnCloneMachineComplete
  • Data->Lifecycle State->Phase equals EVENT

For the Workflow, choose the workflow you created above.

Ensure this is a blocking subscription, otherwise it will execute asynchronously. Since this absolutely has to complete successfully to ensure a proper build, blocking on this one is important!

And finally, because it always seems to happen to me – make sure to PUBLISH the subscription! Friends don’t let friends troubleshoot an Event Subscription that is a draft…

Wrap-Up

At this point you can begin testing your deployments, and verify that the workflow is firing at the appropriate time.

Additionally, it may make sense to build an XaaS service that allows for you to customize the names, sizes of the appliances, or changing default domain, VLAN, etc.

Thanks for reading! If you have any questions feel free to ping me on Twitter.

If you’re curious as to the other available states you can use, check out the Life Cycle Extensibility – vRealize Automation 7.0 document, starting around page 33. Each release has a version of the document with the latest information, so find the one for your version.

 

 

VMware VSA Deepdive – Part 6 – Eventing So Hard Right Now

I sure have done a lot of blogging about how to power off a VM lately, don’t you think?

One more time, but this time it’s the best version of them all.

I’m Eventing SO HARD right now

I figured this out while investigating something related to VSA and connecting the dots. Here’s how it went down.

In my setup, we have the VSA Clustering Service running on a separate VM at the remote location to handle quorum. One problem that pops up about once a month is the ongoing struggle of Windows patching. The VMs patch, and reboot as you would expect. This throws an alarm on the vCenter Datacenter object that the cluster service is offline. The other issue is that it doesn’t clear itself once the box comes back up. I wanted to find a way to clear this automatically, so that my monitoring guys didn’t lose their minds when hundreds of these appeared at 2AM.

If you look at the settings for the Alarm in question, you’ll see that it uses Events rather than Conditions for these alerts. I hadn’t dug too hard into how these worked before, so time to get dirty.

The default VSA Cluster Service Alarm.
The default VSA Cluster Service Alarm.

I realized then that the alarms in question were bubbling up from VSA Manager to vCenter using AMQP, and based on prior experience I knew that WSCLI would show events when I used startListener – so I started to do some more testing.

Running wscli [cluster IP] startListener from the command-line, I verified it was listening, and then rebooted the VSA cluster service machine in my lab. And then something neat happened – the listener showed an event called MemberOfflineEvent fire. I then waited for the node to come back online and sure enough, I saw MemberOnlineEvent fire.

After adding the trigger of MemberOnlineEvent to the alarm, and having it set the status back to normal I performed another test. Sure enough, the alarm came up, and a couple of minutes later, it disappeared when the service came back online. Problem solved!

It begs the question – why wasn’t this built in to the standard alarm? I wish I knew. But at the very least, I can correct this one.

I started thinking about this newfound trick and wondered – could it apply elsewhere with the VSA solution? I know that custom alarms are created for each NFS datastore used by the VSA, and custom alarms for the VSA machines too. What could we do with those?

The default VSA VM offline alarm settings.
The default VSA VM offline alarm settings.

This may look familiar! You can make similar changes to this alarm and it will clear things up automatically there too. So what about datastores?

The default VSA Datastore Alarm.
The default VSA Datastore Offline Alarm.

OK, the event is a little different, but otherwise pretty straightforward. We just need to find the corresponding event that indicates all is well again.

Below is the list of alarms created by the VSA Manager during an installation, by object context and name.
Datacenter – VSA Cluster Service Offline
Datacenter – VSA Storage Cluster Offline
Virtual Machine – VSA Member Offline
Datastore – VSA Storage Entity Offline
Datastore – VSA Storage Entity Degraded

All of these only have alarms for triggering, but none are set up by default to clear themselves. After using WSCLI and lots of testing, here are the settings I feel make the most sense to give the best idea of what is actually happening. This list only shows what would be added to the existing alarm to automatically clear it out.

Alarm: VSA Cluster Service Offline, VSA Member Offline
Event: MemberOnlineEvent
Status: Green

This clears up most of the alarms very easily. The datastore alarms are a little more interesting – you can actually have both the Offline and Degraded alarms fired during an outage, which isn’t necessarily helpful. The below changes will ensure only one is showing at a time.

Alarm: VSA Storage Entity Offline
Event: StorageEntityDegradedEvent
Status: Green

Alarm: VSA Storage Entity Degraded
Event: StorageEntityOnlineEvent
Status: Green

With the above changes, the datastore will go from green, to yellow, to red, and back up the list in the proper order.

Eureka Moment

This is fun and all, but this entry is actually about how to Power Off a VSA VM isn’t it?
Once I had figured out the connection between the events sent to vCenter from VSA Manager, I remembered something from my many adventures in WSCLI regarding the shutdownSvaServer command:

This operation sends an SvaEnterMaintenanceModeEvent management event
when the node is marked for entering maintenance mode, and a
SvaShutdownEvent when the VSA service is ready to shut down.

I hope this is helping you catch on to where this is going.
Let’s do an experiment in our VSA Cluster – create a custom alarm on one of the VSA VMs like so:

Creating a custom Maintenance Mode Alarm.
Creating a custom Maintenance Mode Alarm.
Setting custom Maintenance Mode Alarm event values.
Setting custom Maintenance Mode Alarm event values.
Setting custom Maintenance Mode Alarm action values.
Setting custom Maintenance Mode Alarm action values.

This alarm object will Power Off the VSA VM automatically when the shutdownSvaServer argument is sent through WSCLI. It’s truly a beautiful thing!

The alarm works!
The alarm works!

And for those wanting to see what it looks like in the standard client:

05-alarm-maintenance-shutdown-client

There it is! It seems really simple in hindsight, but then again we did start out just trying to power off a VM.
NOTE: Disable HA before shutting the VSA down. If you don’t, your VSA will get restarted automatically. Obviously, once you are done patching and all of that, re-enable it!

This process is pretty easy to update for a single datacenter manually.

Next time, we’ll make a workflow that will go through every datacenter object and update all of the alarms.

Thanks for reading! The VSA onion continues to peel…