kvm-qemu-libvirt-vdsm研究.ppt

1、KVM Qemu-kvm Libvirt VDSM,Andrew Xu xuan opzoon confidential | 2011 opzoon R&D,体系结构 KVM技术介绍 什么是KVM技术 KVM初始化 KVM API 内存管理 VirtIO Qemu-kvm 什么是Qemu-kvm Kvm和qemu的关系 Libvirt 什么是libvirt技术 Libvirt原理和代码示例 VDSM 什么是vdsm Vdsm的模块分析 参考资料,体系结构,什么是KVM技术,KVM 是作为内核模块实现的，因此 Linux 只要加载该模块就会成为一个hypervisor kvm.ko + kvm-

2、kvm-intel.ko（只支持全虚拟化，必须有Intel VT或者AMD虚拟化技术支持才行） 设备模拟由qemu完成 (本身作为普通进程来运行) KVM 通过 /dev/kvm 字符设备文件 API接口供qemu使用,vmx_init() |- kvm_init () | |- kvm_arch_init () | | |- kvm_mmu_module_init () | | |- kvm_init_msr_list () 保存CPU特性信息，供ioctl查询 | | |- kvm_x86_ops = xxx | |- kvm_arch_hardware_setup () | | |- k

3、vm_x86_ops- hardware_setup | | | |- 读取硬件信息 | | | |- 检查是否支持某些虚拟化特性(比如是否支持ept，是否支持vpid) | | | |- alloc_kvm_area() | | | | |- for_each_possible_cpu(cpu) | | | | |- vmcs = alloc_vmcs_cpu(cpu) | | | | |- per_cpu(vmxarea, cpu) = vmcs | | | | |- | |- register_cpu_notifier(&kvm_cpu_notifier) | |- register_r

4、eboot_notifier(&kvm_reboot_notifier) | |- sysdev_class_register(&kvm_sysdev_class) | |- sysdev_register(&kvm_sysdev) | |- kvm_vcpu_cache = kmem_cache_create(kvm_vcpu, vcpu_size) | |- misc_register(&kvm_dev) KVM API | |-,KVM内部实现(模块初始化),System KVM_GET_API_VERSION KVM_CREATE_VM VM KVM_SET_MEMORY_REGION

5、 KVM_CREATE_IRQCHIP Creates an interrupt controller model KVM_GET_CLOCK/ KVM_SET_CLOCK Gets /Sets the current timestamp of kvmclock KVM_CREATE_VCPU VCPU KVM_GET_MSRS/ KVM_SET_MSRS KVM_GET_REGS/ KVM_SET_REGS KVM_RUN ,KVM API（by 3 ioctls）,影子页表 每个Guest的每个页表都有一个对应的“影子页表”存在 MMU中导入的是影子页表，但Guest OS以为这是它的页表

6、 影子页表转换的是：GVA HPA，由KVM维护 影子页表中每项的权限都比对应的Guest OS真实的页表要低 影子页表有“刷新延迟”机制，尽量少刷新 EPT（external page table） Guest OS的页表还是放在MMU中，它转换的是GVAGPA EPT表转换的是GPA HPA，由KVM维护 所以每次内存操作理论上都要查两次页表 CPUID是它的“刷新延迟”机制 说明： GVA ：Guest Virtual Address GPA：Guest Physic Address HPA：Host Physic Address,KVM的内存管理,相当于Xen的前后端驱动机制 能显著提

7、高IO的速度 半虚拟化技术 Guest OS意识到KVM 的存在，它对磁盘和 网络的请求主动通过 host帮助来完成 Linux和windows都 支持,VirtIO技术,什么是qemu QEMU is a machine emulator(cpu，mmu，磁盘，网络等) Each device in turn can assign I/O ports or memory areas (for MMIO) to its handlers the devices use native device or advanced emulate block device formats like QCO

8、W 什么是qemu-kvm 为kvm优化后的qemu 优化了些什么东东？ Libkvm的支持 进程（ -enable-kvm ）里也可以直接调用/dev/kvm接口 Qemu进程就代表了VM？,qemu-kvm,the goal of libvirt: to provide a common and stable layer sufficient to securely manage domains on a node, possibly remote. 支持：qemu/kmv，xen，Vmware ESX，openVZ，. 语言：C，Python，C#，java，perl， 功能 VM ma

9、nagement: start, stop, pause, save, restore, and migrate disk, network, memory, cpu hotplug Remote machine support: Storage management: file image, NFS, LVM, iSCSI, raw disk Network interface management: configure (and create) interfaces, bridges, vlans, and bond devices. Virtual NAT and Route based

10、 networking: Libvirt virtual networks use firewall rules to act as a router, providing VMs transparent access to the host machines network.,Libvirt技术,Libvirt的使用,步骤 建立connection 得到class 用class的函数操作,05:/root/xua/work/python/libvirt_test.py #!/usr/bin/python import libvirt conn = libvirt.ope

11、n(qemu:/system) print The libvirt version is %d % (conn.getVersion() print - print num of domain is %d % (conn.numOfDomains() for id in conn.listDomainsID(): dom = conn.lookupByID(id) print Dom %s : State %s, OS_type %s, maxMem %d MB, maxVCPU %d % ( (), ()0,dom.OSType(),dom.maxMemory

12、()/1024,dom.maxVcpus() print - print num of network is %d % (conn.numOfNetworks() for net_name in conn.listNetworks(): net = workLookupByName(net_name) print Network %s : bridgeName %s, UUID %s % ((),net.bridgeName(),net.UUIDString() print net.XMLDesc(0) print - print System information is:

13、print conn.getSysinfo(0),代码示例,VDSM(Virtual Desktop and Server Management Daemon) High level API for managing a hypervisor node Written in Python Multithreaded, Multi-processes speaks with its guest agent via virtio-serial 功能 VM lifecycle (libvirt + guest agent). Host bootstrap/registration. Monitori

14、ng host and VMs. Network configuration. Storage management. Policy management(Scheduler, ksm, thin provisioning, storage/network/CPU QoS, ballooning) DSM should be run as a daemon on each node managed by RHEV-M. It answers XML-RPC calls from clients (mostly RHEV-M),VDSM,Component 向rhev-m注册 启动过程 Vm操作

15、/ Libvrit相关 Storage network Configuration /Log 利用python标准库：logging/ ConfigParser(get-conf-item) 安全性 利用M2Crypto （ M2Crypto = Python + OpenSSL + SWIG ） rhev-m提供CA 数字证书发放、管理、取消 rhev-h (XMLRPCServer) load自己的证书 证明自己的身份；传递出自己的公钥 rhev-m检测rhev-h的证书，向CA验证，并用证书提供的公钥来建立“安全的连接” 用“非对称加密算法”建立安全的连接 用“对称加密算法”加密通信的数

16、据 多线程，多进程 Python的多线程是废的： Python 解释器并不是线程安全的。当前线程必须持有全局解释器锁 (Global Interpretor Lock) ，才能对 Python 对象进行安全地访问。所以不能利用多CPU。 server process模型 在这个模型当中，有一个manager进程，负责管理实际的对象。真正的对象也是在manager进程的内存空间当中。所有需要访问该对象的进程都需要先连接到该管理进程，然后获取到对象的一个代理对象（Proxy object）。,VDSM,注册过程 rhev-m ssh登录到rhem-h上，执行命令。 rhem-h的print输出到l

17、og中，机制是什么呢？ 检查“redhat-lsb.x86_64” 检查“vdsm” 以前要rhn注册，修改成只要检查该包是否安装(mod by andew) 检查cpu的“virt support” 检查Linux版本rhev6.0（131） 追加内核参数“elevator=deadline，processor.max_cstate=1” 检查以下包的安装 以前要rhn注册，修改成只要检查该包是否安装(mod by andew) REQ_PACK = (SDL.x86_64, bridge-utils.x86_64, mesa-libGLU.x86_64, openssl.x86_64, m

18、2crypto.x86_64, rsync.x86_64) VDS_PACK = (qemu-kvm, qemu-kvm-tools, vdsm, vdsm-cli, libjpeg, spice-server, pixman, seabios, qemu-img, fence-agents) DEVEL_PACK = (gdb, tcpdump, strace, ltrace, sysstat, ntp, vim-common, vim-enhanced, systemtap-runtime) 修改配置文件“/etc/vdsm/vdsm.conf” 创建management bridge 生

19、成私钥和证书,注册VDSM,Start |- configure_libvirt 修改libvirtd的配置 |- 启动iscsid， multipathd， libvirtd服务 |- load_needed_modules tun，bonding， 8021q（vlan） |- /usr/share/vdsm/respawn -minlifetime 10 -daemon -masterpid /var/run/vdsm/respawn.pid /usr/share/vdsm/vdsm daemon化运行vdsm，pid保存起来 | |- run | | |- serve_clients(

20、) | | | |- 启动xmlrpcserver服务（ SecureXMLRPCServer or SimpleThreadedXMLRPCServer ） | | | |- 注册远程调用函数,启动模块,clientIF.Create() |- 参数解析 |- self.vmContainer id = libvirtvm.LibvirtVm(vmParams) libvirt初始化(建立connection) | |- self._connection = libvirtconnection.get(cif) | | |- _connection = libvirt.openAuth(qe

21、mu:/system, auth, 0) |- self.vmContainerId.run() | |- domxml = _buildCmdLine() 根据配置生成xml文件 | |- self._dom = self._connection.createXML(domxml) 利用libvirt生成VM | |- self._domDependentInit | | |- self._initInterfaces() | | |- self._initVmStats() | | | |- self._vmStats = VmStatsThread(self) | | | |- self

22、._vmStats.start() | | |- self._getUnderlyingDriveInfo() Obtain block devices info from libvirt | | |- self._getUnderlyingDisplayPort() | | |- self._dom.setSchedulerParameters(),VM操作,Fault-in path QEMU calls malloc() and allocates virtual space for the page, but no backing physical page The guest pro

23、cess touches what it thinks is a physical address, but this traps into the host since the memory is unallocated The host kernel sees a page fault, calls do_page_fault() in the area that was malloc()d, and if all goes well, allocates some memory to back it. The host kernel creates a pte_t to connect

24、the malloc()d virtual address to a host physical address, makes rmap entries, puts it on the LRU, etc. mmu_notifier change_pte()? is called, which allows KVM to create an NPT/EPT entry for the new page. (and an spte entry?) Host returns from page fault, guest execution resumes Swap-out path Now, let

25、s say the host is under memory pressure. The page from above has gone through the Linux LRU and has found itself on the inactive list. The kernel decides that it wants the page back: The host kernel uses rmap structures to find out in which VMA (vm_area_struct) the page is mapped. The host kernel lo

26、oks up the mm_struct associated with that VMA, and walks down the Linux page tables to find the host hardware page table entry (pte_t) for the page. The host kernel swaps out the page and clears out the pte_t (lets assume that this page was only used in a single place). But, before freeing the page:

27、 The host kernel calls the mmu_notifier invalidate_page(). This looks up the pages entry in the NPT/EPT structures and removes it. Now, any subsequent access to the page will trap into the host (2) in the fault-in path above),参考-1（ A day in the life of a KVM guest physical page ）,如何查看debug信息：mount -

28、t debugfs none /sys/kernel/debug TAP (as in network tap) simulates an Ethernet device and it operates with layer 2 packets such as Ethernet frames. TUN (as in network TUNnel) simulates a network layer device and it operates with layer 3 packets such as IP packets. TAP is used to create a network bri

29、dge, while TUN is used with routing. What is M2Crypto M2Crypto = Python + OpenSSL + SWIG M2Crypto is a crypto and SSL toolkit for Python featuring the following: RSA, DSA, DH, HMACs, message digests, symmetric ciphers (including AES). SSL functionality to implement clients and servers. HTTPS extensi

30、ons to Pythons httplib, urllib and xmlrpclib. Unforgeable HMACing AuthCookies for web session management. FTP/TLS client and server. S/MIME ZServerSSL: A HTTPS server for Zope. ZSmime: An S/MIME messenger for Zope. xxx,参考2,def _registerFunctions(self): self.server.register_introspection_functions() for method, name in ( (self.destroy, destroy), (self.create, create), (self.list, list), (self.pause, pause), (self.cont, cont), (self.s