vSphere NUMA高级技术架构分析

1、vSphere NUMA高级技术架构分析Concept of local and remote memory in a single serverCores connect to direct-attached memory through integrated memory controllerCores connect to “remote” memory through interconnectMultiple Compute Nodes in a Single SystemConnected with a high-speed interconnect2vCPU1 and VCPU2

2、of VM are scheduled on Node 1Data access flows through integrated memory controllerPredictable, consistent High Speed, low latency memory accessLocal Memory AccessUse of Memory Channel Bandwidth3vCPU2 now accesses memory attached to CPU2Data access flows through interconnectLocal: 2666MT/s = 128 GB/

3、s BW 75 nsRemote: Interconnect =41.6 GB/s BW | Remote latency 130 nsRemote Memory AccessDistance and Lower Bandwidth Interconnect Introduced4About the NUMA ClientESXi groups the vCPUs and memory of a VM into a single unit called a NUMA client.A NUMA client fits inside a NUMA home node.The ESXi NUMA

4、scheduler uses the NUMA client to initially place a VM on a node and to perform ongoing load- balancing operations.VM1vCPUvSocketVM3vSocketvCPUvCPUSocketNUMA Home NodeMemoryHTHTCoreHTHTCoreHTHTCoreHTHTCoreVM2vCPUvSocketNUMA ClientNUMAClientNUMAClientESXiLayervSocket7CPU, NUMA, and Memory SchedulersH

5、ow do these work together?Wide VM NUMA SupportSocketCoreCoreCoreMemoryCoreCoreCoreSocketCoreHome Node 0Home Node 1If a VM has more vCPUs than the number of cores in a NUMA node, the VM is split into a number of NUMA clients and it is referred to as a wide VM.For example, a 10-vCPU VM is considered a

6、 wide VM on a dual-socket, eight cores per socket system:Each NUMA client is assigned a home node.Only the cores count. Hyperthreading threads do not count.VMNUMA ClientMemoryCoreCoreCoreCoreCoreCoreCoreCoreCoreNUMA ClientvCPUvCPUvCPUvCPUvCPUvSocketvSocketvSocketvSocketvSocketvCPUvCPUvCPUvCPUvCPUvSo

7、cketvSocketvSocketvSocketvSocketVirtual NUMA TopologyWhen a wide VM is created, a virtual NUMA (vNUMA) topology is created and presented to the guest operating system.vNUMA enables NUMA-aware guest operating systems and applications to make efficient use of the underlying hardwares NUMA architecture

8、.The vNUMA topology presents a vNUMA node to the VM.vNUMA topology is exposed to the VM if two conditions are met:The VM contains nine or more vCPUs.The vCPU count exceeds the core count of the physical NUMA node.Node 0:6 Cores32 GB RAMNode 1:6 Cores32 GB RAMNUMA ServervNUMA-EnabledVM:12 vCPUs64 GB

9、RAMvNUMANodevNUMANodeOptimizing the Number of Cores per Socket for VMs (1)The number of vCPUs and cores per socket for a VM affect the ability of guest operating systems and applications to optimize their cache usage.You can configure the number of vCPUs and the number of cores per socket for a VM b

10、y editing the VMs CPU properties.To optimize guest operating system memory behavior, ensure that the number of virtual cores per socket does not exceed the number of physical cores per socket on your ESXi host.VMvSocketvCPUvCPUvSocketFor example, you have a VM that is currently configured with 10 vC

11、PUs and 1 core per socket(10 sockets). This VM runs on a dual-socket, eight cores per socket system.Optimizing the Number of Cores per Socket for VMs (2)Dual-Socket, Eight Cores per Socket SystemSocketCoreCoreCoreMemoryCoreCoreCoreSocketCoreNode 0Node 1VMvNUMA NodevCPUvCPUvCPUvCPUvCPUvSocketvSocketv

12、SocketvSocketvSocketvCPUvCPUvCPUvCPUvCPUvSocketvSocketvSocketvSocketvSocketMemoryCoreCoreCoreCoreCoreCoreCoreCoreCorevNUMA NodeTo optimize guest operating system memory behavior for this VM, change the number of cores per socket to five, which results in two sockets.With this new configuration, the

13、vNUMA topology presented to the VM aligns with the NUMA system on which the VM resides.Optimizing the Number of Cores per Socket for VMs (3)SocketCoreCoreCoreMemoryCoreCoreCoreSocketCoreNode 0Node 1VMvNUMA NodevCPUvCPUvCPUvCPUvCPUvSocketvCPUvSocketvCPUvCPUvCPUvCPUMemoryCoreCoreCoreCoreCoreCoreCoreCo

14、reCorevNUMA NodeDual-Socket, Eight Cores per Socket SystemMemory Considerations with vNUMA (1)When a vNUMA topology is calculated, only the compute dimensions are considered.The calculation does not consider the amount of memory configured for the VM, or the amount of memory available within each ph

15、ysical NUMA node.Therefore, you must manually account for memory.For example, an ESXi host has two sockets, 10 cores per socket, and 128 GB of RAM per NUMA node for a total of 256 GB of RAM on the host.SocketMemory 128 GBCCCCCCCCCCSocketMemory 128 GBCCCCCCCCCCNode 0Node 1Dual-Socket, 10 Cores per So

16、cket SystemMemory Considerations with vNUMA (2)Continuing with the example, you create a VM with 96 GB of RAM, 1 socket, and 8 cores per socket. For this VM, ESXi creates a single vNUMA node. The VM fits into a single physical NUMA node.Node 0SocketMemory 128 GBDual-Socket, 10 Cores per Socket Syste

17、mCCCCCCCCCCNode 1SocketMemory 128 GBCCCCCCCCCCVMvNUMA NodeCCCCCCCCvSocketMemory 96 GBMemory Considerations with vNUMA (3)If you create a VM with 192 GB of RAM, 1 socket, and 8 cores per socket, ESXi still creates a single vNUMA node. However, the VM must span 2 physical NUMA nodes, resulting in remo

18、te memory access.Node 0SocketMemory 128 GBCCCCCCCCCCNode 1SocketMemory 128 GBCCCCCCCCCCVMvNUMA NodeCCCCCCCCvSocketMemory 192 GBRemote Memory AccessDual-Socket, 10 Cores per Socket SystemMemory Considerations with vNUMA (4)Node 0SocketMemory 128 GBCCCCCCCCCCNode 1SocketMemory 128 GBCCCCCCCCCCIf the VM is memory-intensive, the optimal configuration for this VM is 2 sockets and 4 cores