Author Archives: vapprentice

About vapprentice

Senior Cloud and Virtualisation Engineer

Cisco UCS and SolidFire ISCSI Boot

Note: At the time of writing Cisco has not certified booting UCS blades from Solidfire ISCSI boot disks.

In order to successfully boot a Cisco UCS B Series blade from a SolidFire ISCSI array you will need to create a custom ESXi ISO that includes a firewall rule to open ports 3261-3264.

Explanation :

It seems that Solidfire arrays will present volumes on ports 3260-3264, so you may get one server to see it’s boot disk without issue but the next one will not work.  The ESXi firewall by default is opened on port 3260 only, there is no rule for 3261,3262,3263 and 3264 so traffic coming back from the Solidfire array is blocked and as a result you will not see the bootdisk.

Workaround

1. Create a custom VIB that opens firewall ports, you will find a good guide here, alternatively PM me for 5.1 firewall bundle.

2. Create a custom ISO using powercli

Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

Add-EsxSoftwareDepot  d:\temp\firewall3261-offline-bundle.zip

Get-EsxImageProfile | Sort-Object “ModifiedTime” -Descending | format-table -property Name,CreationTime 

New-EsxImageProfile -CloneProfile ESXi-5.1.0-20130402001-standard -Name SFBoot-ESXi-5.1.0-20130402001-standard -AcceptanceLevel CommunitySupported

Add-EsxSoftwarePackage -ImageProfile SFBoot-ESXi-5.1.0-20130402001-standard -SoftwarePackage firewallrule

Export-EsxImageProfile -ImageProfile SFBoot-ESXi-5.1.0-20130402001-standard -ExportToISO -FilePath d:\temp\SFBoot-ESXi-5.1.0-20130402001-standard.iso

Hot Modify disks in vCloud Director using PowerCLI

I did not write this script, I found it on this excellent blog, I had a few problems getting the original code to work so thought I would share what I used to get this working. The script below increases the disk size to 70GB on a powered on VM.


Connect-CIServer vcloud.site.com -User administrator -Password password 

#Get VM called diskresize
$vm = Get-CIVM diskresize

#Define new size in MB
$newsize = "70240" 

#Modify the extension data with the new size
(($vm.ExtensionData.Section|where {$_-is  [VMware.VimAutomation.Cloud.Views.OvfVirtualHardwareSection]}).item|where {$_.ResourceType.value-eq17}).hostresource[0].AnyAttr[0]."#text" = $newsize

# Final step, update server data
($vm.ExtensionData.Section|where {$_-is[VMware.VimAutomation.Cloud.Views.OvfVirtualHardwareSection]}).updateserverdata()

‘Not free; Lock’ error messages and high CPU on ESXi host causes VMs to momentarily freeze

Background:

We’re running vCloud Director 1.5.2  and vsphere 5.0 update 1 in a 11 host cluster/PVDC, customers began complaining that virtual machines we’re locking up and losing pings for between 5 and 30 seconds, we were seeing Not free;Lock errors as below in /var/log/vmkernel.log.

2013-04-24T08:33:41.546Z cpu28:6869)DLX: 3901: vol DATALUN’: [Req mode: 1] Not free; Lock [type 10c00001 offset 207360000 v 123467, hb offset 3854336

gen 69, mode 1, owner 5176c50b-7452087c-b21a-mtime 133636 nHld 0 nOvf$

2013-04-24T08:33:48.541Z cpu21:6869)DLX: 3394: vol : [Req mode 1] Checking liveness of [type 10c00001 offset 207360000 v 123467, hb offset 3854336

gen 69, mode 1, owner 5176c50b-7452087c-b21a- mtime 133636 nHld 0$

2013-04-24T08:33:52.552Z cpu21:6869)DLX: 3901: vol : [Req mode: 1] Not free; Lock [type 10c00001 offset 207360000 v 123467, hb offset 3854336

gen 69, mode 1, owner 5176c50b-7452087c-b21a-mtime 133636 nHld 0 nOvf$

This was a real head scratcher, we spent the best part of a week troubleshooting with VMware and EMC, we were seeing the error messages in /var/log/vmkernel.log on random esxi hosts in the same cluster.  When the messages appeared virtual machines running on the datastore and esxi host would momentarily lock up, you can work out which ESXi hosts is causing the lock as the mac address is visible in the error message, we also noticed that the CPU would shoot up to 100% on the host holding the lock.  We went through the storage configuration on the hosts and at the backend, we found some performance issues which we adressed but it did not fix the problem.  At first we thought it might be a LUN zoning problem but this all checked out and everything appeared to be in order.  So we went back to VMware and after a week or so of extensive troubleshooting they confirmed we were hitting the bug described below. The fix is to upgrade to ESXi 5.0 Update 2.

https://www.vmware.com/support/vsphere5/doc/vsp_esxi50_u2_rel_notes.html

“ESXi hostd agent might consume very high CPU resulting in performance degradation”
“When vCloud Director fetches the screen shot of virtual machine desktop from the ESXi host, hostd agent might enter into an infinite loop resulting in 100% CPU usage and the CPU usage might not reduce until you restart hostd.

Hopefully this post will save you some time and hair!!

UCS Service Profile Template creation for SolidFire ISCSI Boot

Note: At the time of writing Cisco has not certified booting UCS blades from Solidfire ISCSI boot disks.

UCS Pre-Requisites:

1. VNIC Templates created

2. UUID Pool Created

3. Boot Policy Created  (can be created during service profile template creation)

4. Maintenance Policy Created

5.  Authentication Policy  Created (used to authenticate with the ISCSI target, can be created during service profile template creation )

6. Initiator IP address policy created (IP addresses assigned to the ISCSI vNICs)

7. Create dedicated ISCSI VLAN, use dedicated uplinks for the ISCSI VLAN

Create Service Profile Template

ISCSI 1

VNIC Creation Select ‘Expert’

ISCSI2

Click add to create the iscsi vnics from vnic templates

ISCSI3

Create ISCSI VNICS, click ‘add’ in the lower pane

ISCSI4

Name the vnic

choose the overlay vnic previously created

do not set iscsi adapter policy

Select ISCSI VLAN

Do not select MAC Address Assignment

ISCSI5

Should look like this

ISCSI6

No VHBA’s

ISCSI7

Leave default click next

ISCSI8

Leave default

ISCSI9

Create boot policy if not already created

ISCSI13

Select manintenance policy if required, recommend using user ack!

ISCSI14

Leave defaults

ISCSI15

Create Policies as required

ISCSI16

Service Profile template is created…

Create service profile from template

ISCSI17

Once created select the new Service Profile  in the left pane and browse to ‘boot order’ in the tab selections

ISCSI18

Select the ISCI vNIC, you will NOT be able to modify the ‘set iscsi boot parameters’ as the service profile is bound to a template.

Work around is to unbind the service profile from it’s template, Cisco say they will fix this, no date as yet!

Go to general tab and click ‘unbind’, now you will be able to modify iscsi boot parameters

ISCSI18

Choose authentication profile for accessing the iscsi target

Select IQN pool

Select Initator IP address pool, create if not there.

ISCSI20

Create iscsi target

Enter the IQN ID provided to you by the storage team

Do not select authentication profile

Enter IP address of ISCSI target.

ISCSI21

You are now ready to boot from the SolidFire Array, my next post will go through booting the ESXi host!

vShield Edge Static Routing between External networks

Config:

1 x VCNS Edge Gateway with Public IP Block – VLAN 200

External Network 1 – with Public IP Block – VLAN 201

External Network 2 – with Public IP Block – VLAN 202

I recently came across a customer requirement where they wanted to set up some static routes between External Network 1 (VLAN 201) and External Network 2 (VLAN 202) as above.  Both networks had been created in vCloud Director as external networks and did not have any Organisation Networks attached to them. 

We had an existing VCNS Edge Gateway with a public IP block assinged on VLAN 200, my assumption was that we add both external networks to the existing VCNS Edge Gateway and apply the static routes between them.  It turns out that it’s not that simple! when I attempted to apply the routes between Network 1 and Network 2  I got the following message… “Static routing between overlapping networks is not supported”, it seems that the because they are connected to vCloud Director as external networks it will not allow you to add static routes as VCD sees them as overlapping networks.

This was a misunderstanding on my part on how static routing works on VCNS Edge Gateways, it seems that I’m not the only one as several of my colleagues had the same misconception.  In the end we applied the static routes on an upstream switch as we could not get the routes to stick on the VCNS Gateway.  Static Routes work well on VAPP networks and Organisation networks on the internal side of the vShield Edge, but there is very little information on there on using them for external networks, I aim to do some more research in this area so will update the blog in due course.

vCloud Director and vRAM Datastore usage

Scenario:

VDC Storage Allocation = 120GB

1 VAPP with 2 virtual machines

4GB of RAM in each and a 40GB Disk per VM.

Total disk usage – 8OGB right?

Customer attempts to create a new VM with exactly the same specs as above, (4GB of RAM and 40GB disk) they are told that they will exceed there storage allocation!

The customer checked their storage usage and vCloud Director reports that they are using 88GB of their 120GB allocation, they have no templates in their catalog so cannot understand where the extra 8GB of storage has gone???

Explanation:

Whatever the size of the virtual machine memory, vCloud Director will reserve the equivalent data store space for the swapfile, so a virtual machine with 4GB of RAM will consume 4GB of disk space on the datastore.

vCloud Director does not take into account memory reservations, it effectively treats it as 100% memory reservation.  This is also irrespective of the power state of the virtual machine.

Memory of virtual machines stored in the catalog is not included in the space allocation.

Something worth taking into conisderation when sizing vCloud Director implementations!

Passwordless SSH between the UCS and a remote Linux Server

1. On the remote Linux/Unix server create the user

useradd -m ucsuser -c “UCS user” -d /home/ucsuser

2. Change the directory permissions

chown ucsuser /home/ucsuser

3. Generate SSH key for the new user

ssh-keygen (accept defaults and leave passphrase empty)

4. Copy the public Key to a text file.

cd .ssh/

cat id_rsa.pub

5. SSH to the UCS Manager, at the CLI type the following commands

scope security

create local-user ucsuser

set password

6. Copy the Public Key obtained in step 4 and paste it into the CLI with inverted commas.

set sshkeyssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAuo9VQ2CmWBI9/S1f30klCWjnV3lgdXMzO0WUl5iPw85lkdQqap+NFuNmHcb4K

iaQB8X/PDdmtlxQQcawclj+k8f4VcOelBxlsGk5luq5ls1ob1VOIEwcKEL/h5lrdbNlI8y3SS9I/gGiBZ9ARlop9LDpD

m8HPh2LOgyH7Ei1MI8=

commit-buffer

Now you should be able to SSH from the remote linux server to the UCS without entering a password.

VCOPS 5.6 vCloud Director Adapter – Configuration Guide

In this post we’ll go through the basic steps to get VCOPs running with the vCloud Director Adapter, the vCloud instance in this example is running vCloud Director 1.5.

1 .Create an IP Pool and allocate 2 IP addresses, 1 for the UIM server 1 for the Analytics server

 2. From the “Data Center” view select the IP Pools tab and click Add.

      Name the IP Pool and define the IP Subnet Mask and Gateway

 2

3. Enter DNS Details

3

4. Select the relevant Network and hit OK.

4

5. Deploy the OVA

5

6. Select small deployment

6

7. Select ‘Fixed’

7

8. Enter IP the addresses of the UIM and Analytics Virtual Machines and hit Next to finish the deployment.

8

9. Once the Deployment has completed you should see the VCOPS VAPP with 2 VMS.

9

10. Browse to the VCOPS Admin interface on UIM IP address https://10.255.224.28/admin/

Enter username = admin and password = admin to follow the initial set up wizard.

10

11. Enter vCenter Details

11

12. If using self-signed certs, click Yes to trust the vCenter Server certificate

13.  Enter new admin and root passwords for both VMs. Default passwords are Admin/admin and root/vmware

 12

14. Register your vCenter, leave Collector user and password blank.

 13

15. If you need to import data follow this… https://www.vmware.com/support/pubs/vcops-pubs.html

Otherwise click Next.

 14

16. If you are using linked vCenters, select the linked VC from the list, otherwise click Finish to complete the registration.

15

17. Once registration is complete the the VCOPS icon should appear in the solutions and Appplications view of the registered vCenter

16

Register the vCloud Adapter

18. Download the adapter installation PAK file anonymously from  ftp://ftp.integrien.com

19. Save the PAK file on your desktop

20. Browse to the UIM server admin interface –  https://10.255.224.28/admin/

21 . Go to the Update TAB and select the PAK file by browsing to your desktop

17

22. Hit Update and click OK to continue

18

23. The update will take several minutes to complete.

19

24. Log in to the Custom user interface as administrator. – https://10.255.224.28/vcops-custom

25. Browse to Admin>Support

20

26. On the Info tab, find the Adapters Info pane and click the Describe icon, highlighted in yellow on the screen grab below.

22

27. Select Yes to begin the describe process, it should take several minutes to complete.

23

28. The vCloud adapter should now appear in the Adapters Info window.

24

29. To create the adapter Instance – Go to Environment > Configuration > Adapter Instances

26

30. Select Add New Adapter Instance

27

31. Enter vCloud instance IP Address or FQDN

28

32. Add vCloud Director credential, Select Add

29

33. Add vCloud Director login Credentials and hit OK.

30

34.  When you get back to the ‘Add adapter Instance’ page, select ‘Test’ to test the connection to vCloud Director, if the test is successful hit OK to complete the config. VCOPs will access the API so bare this in mind if you’ve got Firewalls or Load Balancers in front of your VCD cells.

31

35. That completes the installation of the vCloud Director adapter.

VCOPs will begin collecting DATA so you can go ahead and start creating some custom dashboards.

32

Configure High Availability for VMware SSO using vShield Edge – Part 3

This guide assumes you have already installed the SSO servers in High Availability mode.

Once you have configured the vShield Edge load balancer, perform the steps below to complete the SSO high availability configuration. 

NOTE: We are not using certificates in this example.

  1. Stop Single Sign-On Services  on both SSO servers 

At Command Prompt execute:

  • SC stop ssoTomcat  
  • SC \\SSO2 stop ssoTomcat 

    2.Copy configuration files from SSO1 to SSO2

Copy <drive>:\Program Files\VMware\Infrastructure\SSOServer\security\server-identity.jks

to….

\\SSO2\<drive>$\Program Files\VMware\Infrastructure\SSOServer\security

Copy <drive>:\Program Files\VMware\Infrastructure\SSOServer\webapps\sso-adminserver\WEB-INF\WEB-INF\web.xml

to……

\\SSO2\<drive>$\Program Files\VMware\Infrastructure\SSOServer\webapps\sso-adminserver\WEB-INF\WEB-INF

3.  Save Keystore password

Open the file :\<drive>$\Program Files\VMware\Infrastructure\SSOServer\conf\server, search for line starting with <Connector SSLEnabled=”true”, on line search for parameter keystorePass and write down the password 

4. SSO2 configuration

Open a Command Prompt on SSO2

At Command Prompt execute:

  • CD <drive>:\Program Files\VMware\Infrastructure\SSOServer\utils
  • SSOCLI.cmd configure-riat -a configure-ssl –keystore-password <password saved above> –keystore-file “<drive>:\Program Files\VMware\Infrastructure\SSOServer\security\server-identity.jks” -m <SSO Administrator Password>

5. Start Single Sign-On Services

At Command Prompt execute:

 SC start ssoTomcat

Configure High Availability for VMware SSO using vShield Edge – Part 2

In Part 1 went through the process of deploying the vShield Edge, next we configure the SSO virtual servers

Configure vShield Edge SSO Virtual Server.

Select the newly deployed Edge and click Actions.

Select Manage

vs1

Select the Load Balancer tab.

Click the plus sign to add a pool.

On the Name & Description screen, enter SSO as the name for this pool.

Click Next.

addpool

On the Services screen, enable HTTPS.

Set the Balancing Method to LEAST_CONN.

Enter 7444 as the port number.

Click Next.

serv

On the Health Check screen, change the Monitor Port to 7444.

Click Next.

health

On the Members screen, click the plus sign to add members to this pool.

Enter the IP address of the SSO1 server.

Click Add, to add it to the pool.

Repeat this step for the SSO2 server.

Click Next.

memb

Click Finish to complete the pool creation.

IMPORTANT: Click Publish Changes in the green bar.

pub

Click the Enable button to enable to pool.

enable

IMPORTANT: Click Publish Changes in the green bar.

pub

Click the Virtual Servers link.

vss

Click the plus sign to add a virtual server.

Enter a name for the virtual server. E.g.. ssl001.localdomain

Enter SSO as the description.

Enter the IP address.

Select the SSO pool.

Enable HTTPS and change the port to 7444.

Select SSL_SESSION_ID as Persistent Method

Click Add.

.virt2

IMPORTANT: Click Publish Changes in the green bar.

pub

Make sure you create a DNS entry in local DNS for the SSO VIP

 

Configure vShield Edge Web Client Virtual Server.

We’re also going to configure the vShield Edge to act as a load balancer for Web Client Service.

Select the new deployed Edge and click Actions.

Select Manage.

vs1

Select the Load Balancer tab.

Click the plus sign to add a pool.

On the Name & Description screen, enter WebClient as the name for this pool.

Click Next.

webcli

On the Services screen, enable HTTPS.

Set the Balancing Method to LEAST_CONN.

Enter 9443 as the port number.

Click Next.

On the Health Check screen, change the Monitor Port to 9443.

Click Next.

ht2

On the Members screen, click the plus sign to add members to this pool.

Enter the IP address of the SSO1 server.

Click Add, to add it to the pool.

Repeat this step for the SSO2 server.

Click Next.
Note SSO1 and SSO2 must have vSphere WebClient software installed.

mem2

Click Finish to complete the pool creation.

IMPORTANT: Click Publish Changes in the green bar

pub

Click the Enable button to enable to pool.

enable

IMPORTANT: Click Publish Changes in the green bar.

pub

Click the Virtual Servers link

setts2

Click the plus sign to add a virtual server.

Enter a name for the virtual server. E.g.

Enter Webclient as the description.

Enter the virtual IP address.

Select the WebClient pool.

Enable HTTPS and change the port to 9443.

Select SSL_SESSION_ID as Persistent Method

Click Add.

wc2

IMPORTANT: Click Publish Changes in the green bar.

pub

 You should now be able to access the SSO service and Web Client using the Virtual IP address, you can check the health of the pools on the ‘Load Balancer’ tab.  In part 3 we will complete the configuration on the SSO servers.

Configure High Availability for VMware SSO using vShield Edge – Part 1

This guide assumes that you have already gone through the SSO server install in high availability, you should currently have 2 SSO servers and a VIP address for load balancing.

  Deploy the vShield Manager template.

 image1

After the OVF file is deployed, power on the vShield Manager virtual machine and open the console.

Log in to the console with the user name admin and password default

At the manager prompt, type enable.

At the Password prompt, type the password default to enable setup mode.

When setup mode is enabled, the prompt string changes to manager#

At the manager# prompt, type setup to begin the setup procedure.

Enter the IP address, subnet mask, default gateway and DNS details.

To change the hostname of vShield Manager.

Type configure terminal

Type hostname xxxxxx

Type exit to exit configure terminal mode

Type copy running-config startup-config

Type reboot to restart vShield Manager

Register vShield Manager with vCenter:

Login to the vShield Manager GUI and Click Settings & Reports.

In the vCenter Server section click on Edit.

Register vShield Manager with vCenter.

2

From vShield Manager GUI.

Click Datacenters.

Select  Datacenter.

Click on the Network Virtualization tab.

Click on the green plus sign to add a vShield Edge.

add vse

Enter Name and Description

Click Next.

4

Configure the credentials for CLI access and enable ssh

Click Next.

6

On the Edge Appliances screen, leave all options at default.

edge appliance

Click on the green plus sign to add the Edge appliance.

Enter vShield Edge placement details

8

Click Add

Click again on the green plus sign to add the Failover Edge appliance.

edge appliance

Select Failover vShield Edge placement details.
Note: Datastore and Host have to be different from the previous ones

10

On the Interfaces screen, click the green plus sign to add the uplink interface.

Enter vnic0 as the name of the interface.

Select the Network to bind ‘connected to’ this will be your management network.

Click the green plus sign to add the IP configuration.

11

Click the plus sign again to add the IP address.

Enter IP address and click OK.

Enter subnet mask and click Save.

Leave everything else default in the parent window.

Click Add.

12

Again on the Interfaces screen, click the green plus sign to add the Internal interface.

Enter int0 as the name of the interface.

Select the Network to bind.

Click the green plus sign to add the IP configuration

int0

Click the plus sign again to add the IP address.

Enter IP address and click OK.
(For IP address use 192.168.2.1)

Enter subnet mask (255.255.255.128) and click Save.

Leave everything else default in the parent window.

Click Add.

Click Next when back on the Interfaces screen.

On the Default Gateway screen, configure the default gateway.

Check Configure Default Gateway.

Enter gateway IP address.

Click Next.

15

On the Firewall & HA screen, check the Configure Firewall default policy checkbox.

Set the Default Traffic Policy to Accept.

We are not going to use firewall capabilities in the loadbalancer.

Click Next.

On the Summary screen, click Finish.

Wait for the Edge deployment process to finish.

fw

That’s the vShield Edge device deployed, in Part 2 we will configure the SSO Virtual Servers

Schedule Cisco UCS Backups

A quick and easy ‘expect’ script that runs from cron on a nightly basis, it’s not the most graceful script you’ll ever see, but it gets the job done.

I understand that scheduled backups are available in Cisco UCS release 2.1

1. Create a ‘backup’ user on the UCS and the remote Unix/Linux Server, in our case we used the VMA as the backup location.

2. Configure passwordless ssh between the UCS and the remote server, more details can be found here.
 
3. Copy the script below to the remote Linux/Unix server and put it the home directory of the the ‘backup’ user created in step 1.

cat /home/bckup/ucsbackup_full.sh

#!/usr/bin/expect -f

# Expect script to run full backup of the UCS

# Set Variables

set UCS 10.255.2.3 #UCS VIP Address
set DESTPASSWD ucsb@ckup   
set DESTIP 10.255.1.2 # IP Address of remote Unix/Linux Server
set DESTUSER backupuser
set DESTDIR /var/ftp/pub/ucsbackups/

 # Connect to UCS

spawn ssh $UCS

# Delete previous night backup as UCS allows only one object to exist

 send   “scope system\n”

send   “delete backup $DESTIP\n”

send   “commit-buffer\n”

send   “exit\n”

# Run New Backup and SCP to the shared area on the VMA

 send   “scope system\n”

send   “create backup scp://$DESTUSER@$DESTIP$DESTDIR full-state enabled \n”

expect “Password:”

send   “$DESTPASSWD\n”

send   “commit-buffer\n”

send   “exit\n”

expect  eof

4. Edit the crontab of the ‘backup’ user and schedule it to run on Sunday morning at 3am.

crontab -l
0 3 * * 7 /home/bckup/ucsbackup_full.sh > /home/bckup/full.txt