Maximizing OpenEdge Performance in Vmware ® ESX John Harlow, President BravePoint Session 118

© 2009 Progress Software Corporation. All rights reserved. 2 About John Harlow & BravePoint John Harlow Unix user since 1982 Progress developer since 1984 Linux Desktop and Server user since 1995 VMware ® user since earliest beta in 1999 BravePoint is an IT Services Company Founded in employees Focus on OpenEdge,.Net,Business Intelligence Managed Database and D/R services QAD and manufacturing implementations We use virtualization extensively

© 2009 Progress Software Corporation. All rights reserved. 3 Assumptions and Background This presentation assumes that you have some familiarity with virtualization in general and VMware® specifically Virtualization at BravePoint Our production systems run in VMware® VMs Most Development/Test Servers run as Virtual Machines in a VMware® Server Farm Mac/Linux users use desktop VMs to run Windows Apps Support Desk and Developers use desktop VMs to deal with conflicting customer VPNs –Centralized VM server for VPN guests improves security Production systems D/R is done via VMs

© 2009 Progress Software Corporation. All rights reserved. 4 What is Virtualization? Definition Virtualization is an abstract layer that decouples the physical hardware from the operating system to deliver greater IT resource utilization and flexibility

© 2009 Progress Software Corporation. All rights reserved. 5 Benefits of Virtualization? Partitioning Multiple applications, operating systems and environments can be supported in a single physical system Allows computing resources to be treated as a uniform pool for allocation Decouples systems and software from hardware and simplifies hardware scalability

© 2009 Progress Software Corporation. All rights reserved. 6 Benefits of Virtualization? Isolation VM is completely isolated from the host machine and other VMs Reboot or crash of a VM shouldnt affect other VMs Data is not shared between VMs Applications can only communicate over configured network connections

© 2009 Progress Software Corporation. All rights reserved. 7 Benefits of Virtualization? Encapsulation Complete VMs typically exist in a few files –Easily backed up, copied, or moved The hardware of the VM is standardized –Compatibility is guaranteed Upgrades/changes in the real underlying hardware are generally transparent to the VM

© 2009 Progress Software Corporation. All rights reserved. 8 Why Use Virtualization At All? Lets look at a typical SMB computer systems SystemCPU Load Domain Controller10% Print Server20% File Server20% Exchange Server20% Web Server7% Database Server30% Citrix Server50%

© 2009 Progress Software Corporation. All rights reserved. 9 Why Use Virtualization? In the typical SMB setup Utilization is low Backup and recovery are complicated and hardware dependent Administration is complicated Many points of failure

© 2009 Progress Software Corporation. All rights reserved. 10 Why Use Virtualization? Less hardware Higher utilization Redundancy and higher availability Lower administrative workload Hardware upgrades are invisible to virtual systems The list goes on and on… Virtualized Servers

© 2009 Progress Software Corporation. All rights reserved. 11 Why ESX Best Practices? We already know how to administer and tune our real systems Besides, they dont even know that they are in a VM How different could a VM be from a real machine? Were going to look under the covers at these 4 areas Memory CPUs Networking Storage

© 2009 Progress Software Corporation. All rights reserved. 12 ESX Memory Management Concepts Each virtual machine believes its memory is physical, contiguous and starts at address 0 In reality no instance starts at 0 and the memory used by a VM can be scattered across the physical memory of the server Virtual memory requires an extra level of indirection to make this work ESX maps the VMs memory to real memory and intercepts and corrects operations that use memory This adds overhead Each VM is configured with a certain amount of RAM at boot This configured size cannot change while the VM is running The total RAM of a VM is its configured size plus a small amount of memory for the frame buffer and other overhead related to configuration This RAM can be reserved or dynamically managed

© 2009 Progress Software Corporation. All rights reserved. 13 Memory Overhead The ESX Console and Kernel use about 300 meg of memory Each running VM also consumes some memory The memory overhead of a VM varies based upon: Memory allocated to the VM Number of CPUs Whether it is 32 or 64 bit There is a table coming up with that info Note--total amount of configured RAM can exceed the physical RAM in the real ESX server

© 2009 Progress Software Corporation. All rights reserved. 14 VM Memory Overhead

© 2009 Progress Software Corporation. All rights reserved. 15 How VMware ® Manages RAM Memory Sharing - mapping duplicate pages of RAM between different VMs Since most installations run multiple copies of the same guest operating systems, a large number of memory pages are duplicated across instances Savings can be as much as 30% Memory Ballooning - using a process inside the VM to tie-up unused memory Guests dont understand that some of their memory might not be available The VMware ® Tools driver mallocs memory from the guest OS and gives it back to ESX to use for other VMs Physical-to-physical memory address mapping is also handled by VMware ® and adds overhead

© 2009 Progress Software Corporation. All rights reserved. 16 Memory Best Practices Make sure that the host has more physical memory than the amount used by ESX and the working sets of the running VMs Esxtop is a tool that helps you monitor this Reserve the full memory set size for your OpenEdge server Vmware ® cant take memory away from the guest and slow it down Use

© 2009 Progress Software Corporation. All rights reserved. 17 Memory Best Practices Use shadow page tables to avoid latency in managing mapped memory Allocate enough memory to each guest so that it does not swap inside its VM Vmware ® is much more efficient at swapping that the guest is Dont over commit memory Ram is cheap If you must over commit memory, be sure to place the ESX swap area on fastest filesystem possible

© 2009 Progress Software Corporation. All rights reserved. 18 ESX CPU Management Virtualizing CPUs adds overhead Amount depends on how much of the workload can run in the CPU directly, without intervention by VMware ® Work that cant run directly requires mode switches and additional overhead Other tasks like memory management and networking also add overhead

© 2009 Progress Software Corporation. All rights reserved. 19 CPU Realities A guest is never going to match the performance it would have directly on the underlying hardware For CPU intensive guests this is important For guests that do lots of disk i/o it doesnt tend to matter much When sizing the server and the workload, factor in losing 20-30% of CPU resources to virtualization overhead

© 2009 Progress Software Corporation. All rights reserved. 20 CPU Best Practices Use as few vCPUs as possible vCPUs add overhead Unused vCPUs still consume resources Configure UP systems with UP HAL Watch out for this when changing a systems VM hardware from SMP to UP Most SMP kernels will run in UP mode, but not as well Running SMP in UP mode adds significant overhead Use UP systems for single threaded apps

© 2009 Progress Software Corporation. All rights reserved. 21 CPU Best Practices Dont over commit CPU resources Take into account the workload requirements of each guest At the physical level, aim for a 50% CPU steady state load Whenever possible pin multi-threaded or multi-process apps to specific vCPUs There is overhead associated with moving a process from one vCPU to another If possible, use guests with low system timer rates This varies wildly by guest OS

© 2009 Progress Software Corporation. All rights reserved. 22 ESX Network Management Pay attention to the physical network of the ESX system How busy is the network? How many switches must traffic traverse to accomplish workloads? Are the NICs configured to optimal speed/duplex settings? Use all of the real NICs in the ESX server Use server class NICs Use identical settings for speed/duplex Use NIC teaming to balance loads Networking speed depends on the available CPU processing capacity Virtual switches and NICs use CPU cycles. An application that uses extensive networking will consume more CPU resources in ESX

© 2009 Progress Software Corporation. All rights reserved. 23 Networking Best Practices Install VMware ® tools in guests Use vmxnet driver, not e1000 that appears by default Optimizes network activity Reduces overhead Use the same vSwitch for systems that communicate directly Use different vSwitches for systems that do not communicate directly Use a separate NIC for administrative functions Console Backup

© 2009 Progress Software Corporation. All rights reserved. 24 VMware ® Storage Management For OpenEdge applications backend storage performance is critical Most performance issues are related to the configuration of the underlying storage system Its more about i/o channels and hardware than it is about ESX

© 2009 Progress Software Corporation. All rights reserved. 25 VMware ® Storage Best Practices Locate VM and swap files on fastest disk Spread i/o over multiple HBAs and SPs Make sure that the i/o system can handle the number of simultaneous i/os that the guests will generate Choose Fibre Channel SAN for highest storage performance Ensure heavily used VMs not all accessing same LUN concurrently

© 2009 Progress Software Corporation. All rights reserved. 26 VMware ® Storage Best Practices Avoid operations that require excessive file locks or metadata locks Preallocate VMDK files Avoid operations that excessively open/close files on VMFS file systems Use Virtual Infrastructure client to create VMFS partions since it will align them on 64k boundaries If you must use SAN/NAS make sure you adequate CPU to support the activity Use independent/persistent mode for disk i/o Non-persistent and snapshot modes incur performance penalties

© 2009 Progress Software Corporation. All rights reserved. 27 Other Resource Best Practices If you frequently change the resource pool (ie: adding or removing ESX servers) use Shares instead of Reservations This way relative priorities remain in tact Use a Reservation to set the minimum acceptable resource level for a guest, not the total amount Enable hyperthreading in the ESX server

© 2009 Progress Software Corporation. All rights reserved. 28 Advanced Topics Use the references on the following slide to learn more about these topics Setting maximum queue depth appropriately Monitor queue depth with ESXTOP Increasing VMs maximum outstanding disk requests if needed Guest systems use 64K as default the default i/o size Increasing this for applications that use larger block sizes

© 2009 Progress Software Corporation. All rights reserved. 29 Reference Resources Performance Tuning Best Practices for ESX Server 3 Performance Best Practices for VMware vSphere The Role of Memory in VMware ESX Server 3 Ten Reasons Why Oracle Databases Run Best on VMware

Maximizing OpenEdge Performance in Vmware ® ESX John Harlow, President BravePoint Session 118