Objective 3.4 – Determine Appropriate Compute Resources for a vSphere 5.x Physical Design
1. Describe best practices with respect to CPU family choices.
Best practice is to stick to identical hardware across clusters, if this is not possible then EVC mode can be enabled but remember vCenter will create a baseline which may limit some features if the CPU being added to the cluster is newer than the existing CPUs, see VMware KB 1003212 for more info.
Skills and Abilities
2. Based on the service catalog and given functional requirements, for each service:
- Determine the most appropriate compute technologies for the design.
- Implement the service based on the required infrastructure qualities.
- AMD – Vi (IOMMU) or Intel VT-d CPUs for direct I/O compatibiltiy
- Be careful when using CPU affinity on systems with hyper-threading. Because the two logical processors share most of the processor resources, pinning vCPUs, whether from different virtual machines or from a single SMP virtual machine, to both logical processors on one core (CPUs 0 and 1, for example) could cause poor performance.
General BIOS Settings
- Make sure you are running the latest version of the BIOS available for your system.
- Make sure the BIOS is set to enable all populated processor sockets and to enable all cores in each socket.
- Enable “Turbo Boost” in the BIOS if your processors support it.
- Make sure hyper-threading is enabled in the BIOS for processors that support it.
- Some NUMA-capable systems provide an option in the BIOS to disable NUMA by enabling node interleaving. In most cases you will get the best performance by disabling node interleaving (in other words, leaving NUMA enabled).
- Make sure any hardware-assisted virtualization features (VT-x, AMD-V, EPT, RVI, and so on) are enabledin the BIOS.
- Disable from within the BIOS any devices you won’t be using. This might include, for example, unneeded serial, USB, or network ports.
- Cache prefetching mechanisms (sometimes called DPL Prefetch, Hardware Prefetcher, L2 Streaming Prefetch, or Adjacent Cache Line Prefetch) usually help performance, especially when memory access patterns are regular. When running applications that access memory randomly, however, disabling these mechanisms might result in improved performance.
- If the BIOS allows the memory scrubbing rate to be configured, we recommend leaving it at the manufacturer’s default setting
3. Explain the impact of a technical design on the choice of server density:
- Scale Up
- Scale Out
- Auto Deploy
Few large servers, bigger impact if there is a failure, less management, cooling, power and heating.
Smaller servers, fewer VMs impacted when there is failure, scaling is more agile.
Suitable for large enterprises, requires extra infrastructure and can be more complex to manage, dependency on vCenter so needs extra consideration when designing vCenter availability.
Blade vs Server
- less space; less I/O slots; less RAM slots; non scalable cost; more
heating/cooling cost; vendor lockin; simpler cabling; shared chassis (SPOF?); expertise
- Rack = more I/O & RAM slots; take up more space.
4. Determine a consolidation ratio based upon capacity analysis data.
Again, some of the notes borrowed from the BrownBag DCD pdf, they do a nice job of making the notes concise.
- Cores per CPU: The number of cores per host must match or exceed the number of vCPUs of the Largest VM
- Depending on load typically you can run 4 to 6 VMs per core on quad core socket processor
- During current state analysis, determine total CPU and RAM required, then divide that out to the # of hosts required to meet the CPU/RAM requirement.
This will also be based on budget,as well as what the compute ‘density’ requirement is (i.e. scale up or scale out approach), if redundancy is one of the main requirements, then a scale out approach would be better.
5. Calculate the number of nodes in an HA cluster based upon host failure count and resource guarantees.
Taken from the vSphere Availability PDF.
The following recommendations are best practices for vSphere HA admission control.
- Select the Percentage of Cluster Resources Reserved admission control policy. This policy offers the most flexibility in terms of host and virtual machine sizing. When configuring this policy, choose a percentage for CPU and memory that reflects the number of host failures you want to support. For example, if you want vSphere HA to set aside resources for two host failures and have ten hosts of equal capacity in the cluster, then specify 20% (2/10).
- Ensure that you size all cluster hosts equally. For the Host Failures Cluster Tolerates policy, an
unbalanced cluster results in excess capacity being reserved to handle failures because vSphere HA reserves capacity for the largest hosts. For the Percentage of Cluster Resources Policy, an unbalanced cluster requires that you specify larger percentages than would otherwise be necessary to reserve enough capacity for the anticipated number of host failures.
- If you plan to use the Host Failures Cluster Tolerates policy, try to keep virtual machine sizing requirements similar across all configured virtual machines. This policy uses slot sizes to calculate the amount of capacity needed to reserve for each virtual machine. The slot size is based on the largest reserved memory and CPU needed for any virtual machine. When you mix virtual machines of different CPU and memory requirements, the slot size calculation defaults to the largest possible, which limits consolidation.
- If you plan to use the Specify Failover Hosts policy, decide how many host failures to support and then specify this number of hosts as failover hosts. If the cluster is unbalanced, the designated failover hosts should be at least the same size as the non-failover hosts in your cluster. This ensures that there isadequate capacity in case of failure.
Example: Admission Control Using Percentage of Cluster Resources Reserved Policy The way that Current Failover Capacity is calculated and used with this admission control policy is shown with an example. Make the following assumptions about a cluster:
The cluster is comprised of three hosts, each with a different amount of available CPU and memory resources. The first host (H1) has 9GHz of available CPU resources and 9GB of available memory, while Host 2 (H2) has 9GHz and 6GB and Host 3 (H3) has 6GHz and 6GB.
There are five powered-on virtual machines in the cluster with differing CPU and memory requirements. VM1 needs 2GHz of CPU resources and 1GB of memory, while VM2 needs 2GHz and 1GB, VM3 needs 1GHz and 2GB, VM4 needs 1GHz and 1GB, and VM5 needs 1GHz and 1GB.
The Configured Failover Capacity is set to 25%.
The total resource requirements for the powered-on virtual machines is 7GHz and 6GB. The total host resources available for virtual machines is 24GHz and 21GB. Based on this, the Current CPU Failover Capacity is 70% ((24GHz – 7GHz)/24GHz). Similarly, the Current Memory Failover Capacity is 71% ((21GB-6GB)/21GB).Because the cluster’s Configured Failover Capacity is set to 25%, 45% of the cluster’s total CPU resources and 46% of the cluster’s memory resources are still available to power on additional virtual machines.
6. Explain the implications of using reservations, limits, and shares on the physical design.
Resource Allocation Shares Shares specify the relative importance of a virtual machine (or resource pool). If a virtual machine has twice as many shares of a resource as another virtual machine, it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources.
Shares are typically specified as High, Normal, or Low and these values specify share values with a 4:2:1 ratio, respectively. You can also select Custom to assign a specific number of shares (which expresses a proportional weight) to each virtual machine.
Specifying shares makes sense only with regard to sibling virtual machines or resource pools, that is, virtual machines or resource pools with the same parent in the resource pool hierarchy. Siblings share resources according to their relative share values, bounded by the reservation and limit. When you assign shares to a virtual machine, you always specify the priority for that virtual machine relative to other powered-on virtual machines.
The following table shows the default CPU and memory share values for a virtual machine. For resource pools, the default CPU and memory share values are the same, but must be multiplied as if the resource pool were a virtual machine with four virtual CPUs and 16 GB of memory.
For example, an SMP virtual machine with two virtual CPUs and 1GB RAM with CPU and memory shares set to Normal has 2×1000=2000 shares of CPU and 10×1024=10240 shares of memory.
NOTE Virtual machines with more than one virtual CPU are called SMP (symmetric multiprocessing) virtual machines. ESXi supports up to 32 virtual CPUs per virtual machine.
The relative priority represented by each share changes when a new virtual machine is powered on. This affects all virtual machines in the same resource pool. All of the virtual machines have the same number of virtual CPUs. Consider the following examples.
- Two CPU-bound virtual machines run on a host with 8GHz of aggregate CPU capacity. Their CPU shares are set to Normal and get 4GHz each.
- A third CPU-bound virtual machine is powered on. Its CPU shares value is set to High, which means it should have twice as many shares as the machines set to Normal. The new virtual machine receives 4GHz and the two other machines get only 2GHz each. The same result occurs if the user specifies a custom share value of 2000 for the third virtual machine.
Limit specifies an upper bound for CPU, memory, or storage I/O resources that can be allocated to a virtual machine.
A server can allocate more than the reservation to a virtual machine, but never allocates more than the limit, even if there are unused resources on the system. The limit is expressed in concrete units (megahertz, megabytes, or I/O operations per second).
CPU, memory, and storage I/O resource limits default to unlimited. When the memory limit is unlimited, the amount of memory configured for the virtual machine when it was created becomes its effective limit.
In most cases, it is not necessary to specify a limit. There are benefits and drawbacks:
- Benefits – Assigning a limit is useful if you start with a small number of virtual machines and want to manage user expectations. Performance deteriorates as you add more virtual machines. You can simulate having fewer resources available by specifying a limit.
- Drawbacks – You might waste idle resources if you specify a limit. The system does not allow virtual machines to use more resources than the limit, even when the system is underutilized and idle resources are available. Specify the limit only if you have good reasons for doing so.
A reservation specifies the guaranteed minimum allocation for a virtual machine.
vCenter Server or ESXi allows you to power on a virtual machine only if there are enough unreserved resources to satisfy the reservation of the virtual machine. The server guarantees that amount even when the physical server is heavily loaded. The reservation is expressed in concrete units (megahertz or megabytes).
For example, assume you have 2GHz available and specify a reservation of 1GHz for VM1 and 1GHz for VM2. Now each virtual machine is guaranteed to get 1GHz if it needs it. However, if VM1 is using only 500MHz, VM2 can use 1.5GHz. Reservation defaults to 0. You can specify a reservation if you need to guarantee that the minimum required amounts of CPU or memory are always available for the virtual machine.
7. Specify the resource pool and vApp configuration based upon resource requirements.
Taken from vSphere Resource Management 5.5 PDF
A resource pool is a logical abstraction for flexible management of resources. Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources.
Each standalone host and each DRS cluster has an (invisible) root resource pool that groups the resources of that host or cluster. The root resource pool does not appear because the resources of the host (or cluster) and the root resource pool are always the same.
Users can create child resource pools of the root resource pool or of any user-created child resource pool. Each child resource pool owns some of the parent’s resources and can, in turn, have a hierarchy of child resource pools to represent successively smaller units of computational capability.
A resource pool can contain child resource pools, virtual machines, or both. You can create a hierarchy of shared resources. The resource pools at a higher level are called parent resource pools. Resource pools and virtual machines that are at the same level are called siblings. The cluster itself represents the root resource pool. If you do not create child resource pools, only the root resource pools exist.
In the following example, RP-QA is the parent resource pool for RP-QA-UI. RP-Marketing and RP-QA are siblings. The three virtual machines immediately below RP-Marketing are also siblings.
For each resource pool, you specify reservation, limit, shares, and whether the reservation should be expandable. The resource pool resources are then available to child resource pools and virtual machines.
A vSphere vApp allows packaging of multiple interoperating virtual machines and software applications that you can manage as a unit and distribute in OVF format.
A vApp can contain one or more virtual machines, but any operation carried out on the vApp, such as clone or power off, affects all virtual machines in the vApp container,
From the vSphere Web Client, you can access the vApp summary page with the current status of the vApp, and you can manage the vApp.
NOTE: Because the vApp metadata resides in the vCenter Server database, a vApp can be distributed across multiple ESXi hosts. This information can be lost if the vCenter Server database is cleared or if a standalone ESXi host that contains a vApp is removed from vCenter Server. Back up your vApps to an OVF package to avoid losing metadata.
vApp metadata for virtual machines within a vApp do not follow the snapshots semantics for virtual machine configuration. vApp properties that are deleted, modified, or defined after a snapshot is taken remain intact (deleted, modified, or defined) after the virtual machine reverts to that snapshot or any prior snapshots.
You can use VMware Studio to automate the creation of ready-to-deploy vApps with pre-populated application software and operating systems. VMware Studio adds a network agent to the guest so that vApps bootstrap with minimal effort. Configuration parameters that are specified for vApps appear as OVF
properties in the vCenter Server deployment wizard. For information about VMware Studio and for download, see the VMware Studio developer page on the VMware Web site
8. Size compute resources: Memory, CPU, I/O devices, Internal storage
Plan for 60-80% utilisation, start with one vCPU and add more as required, 4-6 virtual machines per core but should be driven by the results from the discovery phase, i.e. capacity planner, perfmon, top etc…
Plan for 70-90% utilisation, take in to account virtual machine memory overhead which is determined by the amount of RAM and number of vCPUs.
Minimum of four NICs on your host server. If using SCSI, use more than four NICs, note VMotion. Assign multiple NIC’s for redundancy and increased capacity.
Taken from vSphere 5.5 performance best practices..
- Make sure that end-to-end Fibre Channel speeds are consistent to help avoid performance problems.
- Configure maximum queue depth for Fibre Channel HBA cards.
- For the best networking performance, we recommend the use of network adapters that support the following hardware features:
- Checksum offload
- TCP segmentation offload (TSO)
- Ability to handle high-memory DMA (that is, 64-bit DMA addresses)
- Ability to handle multiple Scatter Gather elements per Tx frame
- Jumbo frames (JF)
- Large receive offload (LRO)
- On some 10 Gigabit Ethernet hardware network adapters, ESXi supports NetQueue, a technology that significantly improves performance of 10 Gigabit Ethernet network adapters in virtualized environments.
- In addition to the PCI and PCI-X bus architectures, we now have the PCI Express (PCIe) architecture. Ideally single-port 10 Gigabit Ethernet network adapters should use PCIe x8 (or higher) or PCI-X 266 and dual-port 10 Gigabit Ethernet network adapters should use PCIe x16 (or higher). There should preferably be no “bridge chip” (e.g., PCI-X to PCIe or PCIe to PCI-X) in the path to the actual Ethernet device (including any embedded bridge chip on the device itself), as these chips can reduce performance
Boot device should be a minimum of 1GB, when booting from a local disk or SAN/iSCSI LUN, a 5.2GB disk is required to allow for the creation of the VMFS volume and a 4GB scratch partition.
9. Given a constraint to use existing hardware, determine suitability of the hardware for the design.
Has to be on the VMware HCL otherwise you can run in to support issues from VMware, make sure it meets the infrastructure qualities, AMPRS!