Category Archives: VMware

General VMWare posts

‘Not free; Lock’ error messages and high CPU on ESXi host causes VMs to momentarily freeze

Background:

We’re running vCloud Director 1.5.2  and vsphere 5.0 update 1 in a 11 host cluster/PVDC, customers began complaining that virtual machines we’re locking up and losing pings for between 5 and 30 seconds, we were seeing Not free;Lock errors as below in /var/log/vmkernel.log.

2013-04-24T08:33:41.546Z cpu28:6869)DLX: 3901: vol DATALUN’: [Req mode: 1] Not free; Lock [type 10c00001 offset 207360000 v 123467, hb offset 3854336

gen 69, mode 1, owner 5176c50b-7452087c-b21a-mtime 133636 nHld 0 nOvf$

2013-04-24T08:33:48.541Z cpu21:6869)DLX: 3394: vol : [Req mode 1] Checking liveness of [type 10c00001 offset 207360000 v 123467, hb offset 3854336

gen 69, mode 1, owner 5176c50b-7452087c-b21a- mtime 133636 nHld 0$

2013-04-24T08:33:52.552Z cpu21:6869)DLX: 3901: vol : [Req mode: 1] Not free; Lock [type 10c00001 offset 207360000 v 123467, hb offset 3854336

gen 69, mode 1, owner 5176c50b-7452087c-b21a-mtime 133636 nHld 0 nOvf$

This was a real head scratcher, we spent the best part of a week troubleshooting with VMware and EMC, we were seeing the error messages in /var/log/vmkernel.log on random esxi hosts in the same cluster.  When the messages appeared virtual machines running on the datastore and esxi host would momentarily lock up, you can work out which ESXi hosts is causing the lock as the mac address is visible in the error message, we also noticed that the CPU would shoot up to 100% on the host holding the lock.  We went through the storage configuration on the hosts and at the backend, we found some performance issues which we adressed but it did not fix the problem.  At first we thought it might be a LUN zoning problem but this all checked out and everything appeared to be in order.  So we went back to VMware and after a week or so of extensive troubleshooting they confirmed we were hitting the bug described below. The fix is to upgrade to ESXi 5.0 Update 2.

https://www.vmware.com/support/vsphere5/doc/vsp_esxi50_u2_rel_notes.html

“ESXi hostd agent might consume very high CPU resulting in performance degradation”
“When vCloud Director fetches the screen shot of virtual machine desktop from the ESXi host, hostd agent might enter into an infinite loop resulting in 100% CPU usage and the CPU usage might not reduce until you restart hostd.

Hopefully this post will save you some time and hair!!

Configure High Availability for VMware SSO using vShield Edge – Part 3

This guide assumes you have already installed the SSO servers in High Availability mode.

Once you have configured the vShield Edge load balancer, perform the steps below to complete the SSO high availability configuration. 

NOTE: We are not using certificates in this example.

  1. Stop Single Sign-On Services  on both SSO servers 

At Command Prompt execute:

  • SC stop ssoTomcat  
  • SC \\SSO2 stop ssoTomcat 

    2.Copy configuration files from SSO1 to SSO2

Copy <drive>:\Program Files\VMware\Infrastructure\SSOServer\security\server-identity.jks

to….

\\SSO2\<drive>$\Program Files\VMware\Infrastructure\SSOServer\security

Copy <drive>:\Program Files\VMware\Infrastructure\SSOServer\webapps\sso-adminserver\WEB-INF\WEB-INF\web.xml

to……

\\SSO2\<drive>$\Program Files\VMware\Infrastructure\SSOServer\webapps\sso-adminserver\WEB-INF\WEB-INF

3.  Save Keystore password

Open the file :\<drive>$\Program Files\VMware\Infrastructure\SSOServer\conf\server, search for line starting with <Connector SSLEnabled=”true”, on line search for parameter keystorePass and write down the password 

4. SSO2 configuration

Open a Command Prompt on SSO2

At Command Prompt execute:

  • CD <drive>:\Program Files\VMware\Infrastructure\SSOServer\utils
  • SSOCLI.cmd configure-riat -a configure-ssl –keystore-password <password saved above> –keystore-file “<drive>:\Program Files\VMware\Infrastructure\SSOServer\security\server-identity.jks” -m <SSO Administrator Password>

5. Start Single Sign-On Services

At Command Prompt execute:

 SC start ssoTomcat

Configure High Availability for VMware SSO using vShield Edge – Part 2

In Part 1 went through the process of deploying the vShield Edge, next we configure the SSO virtual servers

Configure vShield Edge SSO Virtual Server.

Select the newly deployed Edge and click Actions.

Select Manage

vs1

Select the Load Balancer tab.

Click the plus sign to add a pool.

On the Name & Description screen, enter SSO as the name for this pool.

Click Next.

addpool

On the Services screen, enable HTTPS.

Set the Balancing Method to LEAST_CONN.

Enter 7444 as the port number.

Click Next.

serv

On the Health Check screen, change the Monitor Port to 7444.

Click Next.

health

On the Members screen, click the plus sign to add members to this pool.

Enter the IP address of the SSO1 server.

Click Add, to add it to the pool.

Repeat this step for the SSO2 server.

Click Next.

memb

Click Finish to complete the pool creation.

IMPORTANT: Click Publish Changes in the green bar.

pub

Click the Enable button to enable to pool.

enable

IMPORTANT: Click Publish Changes in the green bar.

pub

Click the Virtual Servers link.

vss

Click the plus sign to add a virtual server.

Enter a name for the virtual server. E.g.. ssl001.localdomain

Enter SSO as the description.

Enter the IP address.

Select the SSO pool.

Enable HTTPS and change the port to 7444.

Select SSL_SESSION_ID as Persistent Method

Click Add.

.virt2

IMPORTANT: Click Publish Changes in the green bar.

pub

Make sure you create a DNS entry in local DNS for the SSO VIP

 

Configure vShield Edge Web Client Virtual Server.

We’re also going to configure the vShield Edge to act as a load balancer for Web Client Service.

Select the new deployed Edge and click Actions.

Select Manage.

vs1

Select the Load Balancer tab.

Click the plus sign to add a pool.

On the Name & Description screen, enter WebClient as the name for this pool.

Click Next.

webcli

On the Services screen, enable HTTPS.

Set the Balancing Method to LEAST_CONN.

Enter 9443 as the port number.

Click Next.

On the Health Check screen, change the Monitor Port to 9443.

Click Next.

ht2

On the Members screen, click the plus sign to add members to this pool.

Enter the IP address of the SSO1 server.

Click Add, to add it to the pool.

Repeat this step for the SSO2 server.

Click Next.
Note SSO1 and SSO2 must have vSphere WebClient software installed.

mem2

Click Finish to complete the pool creation.

IMPORTANT: Click Publish Changes in the green bar

pub

Click the Enable button to enable to pool.

enable

IMPORTANT: Click Publish Changes in the green bar.

pub

Click the Virtual Servers link

setts2

Click the plus sign to add a virtual server.

Enter a name for the virtual server. E.g.

Enter Webclient as the description.

Enter the virtual IP address.

Select the WebClient pool.

Enable HTTPS and change the port to 9443.

Select SSL_SESSION_ID as Persistent Method

Click Add.

wc2

IMPORTANT: Click Publish Changes in the green bar.

pub

 You should now be able to access the SSO service and Web Client using the Virtual IP address, you can check the health of the pools on the ‘Load Balancer’ tab.  In part 3 we will complete the configuration on the SSO servers.

Configure High Availability for VMware SSO using vShield Edge – Part 1

This guide assumes that you have already gone through the SSO server install in high availability, you should currently have 2 SSO servers and a VIP address for load balancing.

  Deploy the vShield Manager template.

 image1

After the OVF file is deployed, power on the vShield Manager virtual machine and open the console.

Log in to the console with the user name admin and password default

At the manager prompt, type enable.

At the Password prompt, type the password default to enable setup mode.

When setup mode is enabled, the prompt string changes to manager#

At the manager# prompt, type setup to begin the setup procedure.

Enter the IP address, subnet mask, default gateway and DNS details.

To change the hostname of vShield Manager.

Type configure terminal

Type hostname xxxxxx

Type exit to exit configure terminal mode

Type copy running-config startup-config

Type reboot to restart vShield Manager

Register vShield Manager with vCenter:

Login to the vShield Manager GUI and Click Settings & Reports.

In the vCenter Server section click on Edit.

Register vShield Manager with vCenter.

2

From vShield Manager GUI.

Click Datacenters.

Select  Datacenter.

Click on the Network Virtualization tab.

Click on the green plus sign to add a vShield Edge.

add vse

Enter Name and Description

Click Next.

4

Configure the credentials for CLI access and enable ssh

Click Next.

6

On the Edge Appliances screen, leave all options at default.

edge appliance

Click on the green plus sign to add the Edge appliance.

Enter vShield Edge placement details

8

Click Add

Click again on the green plus sign to add the Failover Edge appliance.

edge appliance

Select Failover vShield Edge placement details.
Note: Datastore and Host have to be different from the previous ones

10

On the Interfaces screen, click the green plus sign to add the uplink interface.

Enter vnic0 as the name of the interface.

Select the Network to bind ‘connected to’ this will be your management network.

Click the green plus sign to add the IP configuration.

11

Click the plus sign again to add the IP address.

Enter IP address and click OK.

Enter subnet mask and click Save.

Leave everything else default in the parent window.

Click Add.

12

Again on the Interfaces screen, click the green plus sign to add the Internal interface.

Enter int0 as the name of the interface.

Select the Network to bind.

Click the green plus sign to add the IP configuration

int0

Click the plus sign again to add the IP address.

Enter IP address and click OK.
(For IP address use 192.168.2.1)

Enter subnet mask (255.255.255.128) and click Save.

Leave everything else default in the parent window.

Click Add.

Click Next when back on the Interfaces screen.

On the Default Gateway screen, configure the default gateway.

Check Configure Default Gateway.

Enter gateway IP address.

Click Next.

15

On the Firewall & HA screen, check the Configure Firewall default policy checkbox.

Set the Default Traffic Policy to Accept.

We are not going to use firewall capabilities in the loadbalancer.

Click Next.

On the Summary screen, click Finish.

Wait for the Edge deployment process to finish.

fw

That’s the vShield Edge device deployed, in Part 2 we will configure the SSO Virtual Servers