容器网络架构设计指南

1、容器网络架构设计指南 （配图）技术创新，变革未来CPUNetworkMemoryStorage I/O Cgroup2 Cgroup1Linux Cgroup示意图容器与操作系统Generic RuntimedockershimremoteCRI grpcCRI shimContainerruntimedockerdkubeletDocker client APIHigh-level runtimeHigh-level runtimeKubelet的架构 CRIdockerShimCRI-containerdCRI-OfraktidockercontainerdruncruncKata-run

2、timeruncKata-runtimecontainerdCRIOCIPod: RuntimerClassPod: annotations:io.kubernetes.cri-o.TrustedSandboxPodappVMtrusteduntrustedPodappVMPodappVMappappappPodPodPod 91e54dfb1179 0 B d74508fb6632 1.895 KB c22013c84729 194.5 KB d3a1f33e8a5a 188.1 MBUbuntu: 15.04Imageroot29fi4375e9k6:/# lsBin dev home l

3、ib64 mnt proc run srv tmp varBoot etc lib media opt root sbin sys usertmpDocker镜像 91e54dfb1179 0 B d74508fb6632 1.895 KB c22013c84729 194.5 KB d3a1f33e8a5a 188.1 MBUbuntu: 15.04Container(based on Ubuntu:15.04 image)Thin R/W LayerContainer LayerImage Layers (R/O) Container（容器） 91e54dfb1179 0 B d74508

4、fb6632 1.895 KB c22013c84729 194.5 KB d3a1f33e8a5a 188.1 MBUbuntu: 15.04 ImagedockerThin R/W LayerThin R/W LayerThin R/W Layerdockerdocker容器与镜像Clientdocker builddocker pulldocker runDocker_Hostdocker daemonContainersImagesRegistryNGINXDocker仓库Docker0 Bridgevethxx vethxxeth0Container1eth0Container2et

5、h0HOST1iptablesBridge模式eth0(host interface)eth0Container1eth0Container2HOST1Host网络模式Docker0 Bridgevethxxeth0Container1eth0Container2HOST1Mapped Container模式GuestOSHypervisorPhysical ServerHost Operating SystemPhysical ServerBins/LibsApp1VM1GuestOSBins/LibsApp2VM2GuestOSBins/LibsApp3VM3Container Engin

6、eBins/LibsApp1C1Bins/LibsApp2C2Bins/LibsApp3C3虚拟机和容器架构对比KVM CPU消耗：14.6%KVM RAM消耗：185MSource: IBMDocker RAM消耗：46MDocker CPU消耗：1.6%Source: IBMDATALOG中间件APP传统单体应用架构DATALOG中间件APP传统单体应用架构DATALOG中间件APP传统单体应用架构单体应用集群APPAPPAPP中间件中间件中间件DATADATADATALOGLOGLOG容器集群容器集群容器集群容器集群微服务架构应用集群微服务特点ChrootJailBorgPhase 1:

7、 容器单机模式Phase 2: 容器集群模式1982200020042005200820092013201420152018Version 1.11容器集群管理技术的发展12018年K8S平台占比83%数据来源：云原生应用计算基金会K8S容器集群管理技术的发展2Pod1Pod2Kube-proxyHost Operating SystemPhysical or Virtual Servernode 1ETCDAPI ServerControllerManagerSchedulerRest APIKubernetes UIKubectrl CLIKubernetes Master Node(Co

8、ntrol Plane)Worker nodesPod1Pod2Kube-proxyHost Operating SystemPhysical or Virtual Servernode NKubernetes Cluster Kubernetes组件架构ContainerRuntime(Docker)ContainerRuntime(Docker)KubeletKubeletPod1:PauseContainer1Container2Container3PodPauseContainer1Container2Container3Pod1Address1Pod2Address2Pod3Addr

9、ess3Pod4Address4Pod5Address5ServiceAddressClientLabels:app: MyAppLabels:app: MyAppLabels:app: MyAppLabels:app: MyAppLabels:app: MyAppKind: ServiceapiVersion: V1metadata: name: my-servicespec: selector: app: MyApp ports: - protocol: TCP port: 80 targetPort: 9376Service与POD的逻辑关系KubernetesPODNodePortSe

10、rvicePODPODREQKubernetesPODLoadBalancerServicePODPODREQKubernetesPODIngressControllerREQPODPODservicePODPODserviceNodePortLoadBalancer Ingress资源对象apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: web-ingress namespace: defaultspec: rules: - host: http: paths: - path: / backend: servi

11、ceName: web servicePort: 80apiVersion: v1kind: Servicemetadata: name: webspec: selector: app: web ports: - protocol: TCP port: 80 targetPort: 9376apiVersion: apps/v1kind: Deploymentmetadata: name: web-deployment labels: app: webspec: replicas: 3 selector: matchLabels: app: web template: metadata: la

12、bels: app: web spec: containers: - name: web image: web:1.7.9 imagePullPolicy: IfNotPresent ports: - containerPort: 9376创建Service创建Ingress，定义相应的Ingress规则部署DeploymentRuntimeNet pluginConfigCNI commands and network configConfigures container networkK8S网络实现方式：CNI接口Macvlan方案简介eth0: eth0.1eth0ContainerIP

13、eth0: Containereth0.2macvlan1macvlan2VLAN 1: gatewayVLAN 2: gatewayeth0: eth0.1eth0Containereth0: Containereth0.2macvlan1macvlan2Host (VM/BM)Host (VM/BM)Calico简介SourceworkloadSource hostrouting, iptablesDestination hostrouting, iptablesDestinationworkloadData Center FabricFirst IP HopOne or more IP

14、HopsLast IP HopvRouter之间运行BGPKerneleth0callxxxeth0callxxxeth0callxxx0iptablesroutersFelixBGP Clienteth0etcdRouter Reflector Calico L3路由方案Podcache1 containerapp1 containerWeb App Frontend1Podcache2 containerapp2 containerWeb App Frontend2veth0/24veth1/24Docker0/24flannel.1/16flanneldEth000CoreOS Mach

15、inePodbackend1 containerbackend1 containerBackend Service1Podbackend2 containerbackend2 containerBackend Service2veth0/24veth1/24Docker0/24flannel.1/16flanneldEth000CoreOS MachinepacketMACOuterIPUDPInnerIPPayloadSource: 00Dest: 00Source: Dest: Flannel方案简介裸机容器和虚拟机容器API Servervalidating admission cont

16、rollercalico network controllercalico mastercalico etcdcalicoctlcalico kube controllerac monitormasterCE1800Vcalico nodenodePODeth0PODeth0eth0eth0user(kubectl)SchedulerController ManagerETCDKubernetes原生组件Calico原生组件华为自研组件kubeletkube-proxyKubernetes原生组件华为自研组件K8S master节点的组件K8S node节点组件VXLANSpineLeaf网络

17、分析/控制层网络服务层业务呈现层计算接入层FirewallBMBMBMKubernetes masterNetwork Overlay方案架构 K8S node (BM)SpineSpineLeafLeafLeafLeafLeafLeafVTEPPod-GWVLANVLANK8S Master nodeAC API Server WatcherVTEPDVRCE1800VL3 routing CNI plugineBGPPod K8S node (BM)SR-IOVL2 bridging CNI pluginPod K8S node (BM)CE1800VL2 bridging CNI plu

18、ginPodVTEPPod-GWAPI Server容器网络与物理网络联动的Network Overlay方案架构vbond2Container1vbond3Container2PF1HOST1CNI pluginPF2VF1SRIOV网卡VF2VF1VF2 SRIOV直通模式示意图CE1800Vcalicoxx calicoxxeth0Container1eth0Container2eth0HOST1eth1bond0CNI pluginL2桥接模式CE1800Vcalicoxx calicoxxeth0Container1eth0Container2eth0HOST1eth1bond0CN

19、I plugineBGPBIRD L3路由模式CE1800VPOD1eth0POD2eth0POD3eth0bond0bond1K8S 管理面K8S 数据面CE1800VPOD1eth0POD2eth0POD3eth0bond0K8S 管理面K8S 数据面bond0.xxK8S nodeK8S node管理面、数据面独立网卡管理面、数据面共网卡K8S NodeAS64512K8S NodeAS64512K8S NodeAS64512K8S NodeAS64512防火墙LBspineeBGPeBGPspineFabric出口PE出口PEServer leafServer leafService

20、leafBorder leafGate wayNVERRAS100L3 GWNVEL3 GWNVEL3 GWNVEL3 GWNVEL3 GWRRCE1800VK8S Node1MLAGKni0Linux vRoutereBGPeBGPPOD2eth0Loopback1vbdif5000eth0POD1eth0K8S_VRFLeaf1Loopback1vbdif5000K8S_VRFLeaf2peer linkCE1800VK8S Node2MLAGKni0Linux vRoutereBGPeBGPPOD4eth0Loopback1vbdif5000eth0POD3eth0K8S_VRFLeaf

21、3Loopback1vbdif5000K8S_VRFLeaf4peer linkSpine1Spine2VTEPVTEP宿主机路由表：28/26 Gateway： interface：*42/32 Gateway： interface：kni0 43/32 Gateway： interface：kni0 43/3242/32宿主机路由表：92/26 Gateway： interface：*/32 Gateway： interface：kni0 /32 Gateway： interface：kni0 /32/32Leaf1,2设备组K8S_VRF的路由表：28/26 nextHop： inter

22、face：vbdif5000/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan92/26 nextHop： interface：vxlanLeaf3,4设备组K8S_VRF的路由表：/32 nextHop： interface：vbdif5000/32 nextHop： interface：vbdif500092/26 nextHop： interface：vbdif500028/26 nextHop： interface：vxlanPod的路由发布过程（关闭BGP路由抑制时）CE1800VK8S Node1MLAGKni0Linu

23、x vRoutereBGPeBGPPOD2eth0Loopback1Kni-gweth0POD1eth0K8S_VRFLeaf1Loopback1Kni-gwK8S_VRFLeaf2peer linkCE1800VK8S Node2MLAGKni0Linux vRoutereBGPeBGPPOD4eth0Loopback1Kni-gweth0POD3eth0K8S_VRFLeaf3Loopback1Kni-gwK8S_VRFLeaf4peer linkSpine1Spine2K8S_VRFBL1K8S_VRFBL2peer link宿主机路由表：28/26 Gateway： interface

24、：*42/32 Gateway： interface：kni0 43/32 Gateway： interface：kni0 宿主机路由表：92/26 Gateway： interface：*/32 Gateway： interface：kni0 /32 Gateway： interface：kni0 BL1,2设备组K8S_VRF的路由表：/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan92/26 nextHop： interface：vxlan28/26 nextHop： interface：vxlan43/3242/32/32

25、/32Leaf1,2设备组K8S_VRF的路由表：28/26 nextHop： interface：vbdif5000/0 nextHop：0 interface：vxlanLeaf3,4设备组K8S_VRF的路由表：/32 nextHop： interface：vbdif5000/32 nextHop： interface：vbdif500092/26 nextHop： interface：vbdif5000/0 nextHop：0 interface：vxlanVTEPVTEPVTEP0Pod的路由发布过程（开启BGP路由抑制时CE1800VK8S Node1MLAGPOD3eth0vbd

26、if5001vbdif5000eth0POD1eth0K8S_VRFLeaf1vbdif5001vbdif5000K8S_VRFLeaf2peer linkCE1800VK8S Node2MLAGPOD6eth0vbdif5001vbdif5000eth0POD4eth0K8S_VRFLeaf3vbdif5001vbdif5000K8S_VRFLeaf4peer linkSpine1Spine2VTEPVTEP/24/24/24/32POD2eth0/24POD5eth0/241234 Pod东西向流量互访模型（Network Overlay L2桥接模式）CE1800VK8S Node1ML

27、AGKni0Linux vRoutereBGPeBGPPOD2eth0Loopback1vbdif5000eth0POD1eth0K8S_VRFLeaf1Loopback1vbdif5000K8S_VRFLeaf2peer linkCE1800VK8S Node2MLAGKni0Linux vRoutereBGPeBGPPOD4eth0Loopback1vbdif5000eth0POD3eth0K8S_VRFLeaf3Loopback1vbdif5000K8S_VRFLeaf4peer linkSpine1Spine2VTEPVTEPCE1800V的路由表：42，islocal=1 ， out

28、_ifindex：calicoxxx 43，islocal=1 ， out_ifindex：calicoxxx nextHop：，out_ifindex：eth0 43/3242/32CE1800V的路由表：，islocal=1 ， out_ifindex：calicoxxx ，islocal=1 ， out_ifindex：calicoxxx nextHop：，out_ifindex：eth0/32/32Leaf1,2设备组K8S_VRF的路由表：28/26 nextHop： interface：vbdif5000/32 nextHop： interface：vxlan/32 nextHop

29、： interface：vxlan92/26 nextHop： interface：vxlanLeaf3,4设备组K8S_VRF的路由表：/32 nextHop： interface：vbdif5000/32 nextHop： interface：vbdif500092/26 nextHop： interface：vbdif500028/26 nextHop： interface：vxlan12Pod东西向流量互访模型（Network Overlay L3路由模式）CE1800VK8S Node1MLAGPOD3eth0vbdif5001vbdif5000eth0POD1eth0K8S_VRF

30、Leaf1vbdif5001vbdif5000K8S_VRFLeaf2peer linkK8S_VRFBL1K8S_VRFBL2peer linkSpine1Spine2VTEPVTEP0/24/24POD2eth0/24Exit_VRFExit_VRFPE1PE21243外南北向流量互访模型（Network Overlay L2桥接模式）CE1800VK8S Node1MLAGKni0Linux vRoutereBGPeBGPPOD2eth0Loopback1vbdif5000eth0POD1eth0K8S_VRFLeaf1Loopback1vbdif5000K8S_VRFLeaf2peer

31、 linkK8S_VRFBL1K8S_VRFBL2peer linkSpine1Spine2VTEPVTEP0CE1800V的路由表：42，islocal=1 ， out_ifindex：calicoxxx 43，islocal=1 ， out_ifindex：calicoxxx nextHop：，out_ifindex：eth0 43/3242/32Leaf1,2设备组K8S_VRF的路由表：28/26 nextHop： interface：vbdif5000/0 nextHop：0 interface：vxlanBL1,2设备组K8S_VRF的路由表：28/26 nextHop： inte

32、rface：vxlan/0 nextHop：X.X.X.X interface：vlanifxxxPE1PE21234南北向流量互访模型（Network Overlay L3路由模式）CE1800VK8S Node1MLAGPOD3eth0vbdif5001vbdif5000eth0POD1eth0K8S_VRFLeaf1vbdif5001vbdif5000K8S_VRFLeaf2peer linkOVSMLAGeth0VM_VRFLeaf3vbdif5002VM_VRFLeaf4peer linkSpine1Spine2VTEP/24/24/24/24POD2eth0/24/24K8S_VR

33、FBL1K8S_VRFBL2peer linkVTEP0VM_VRFVM_VRFvbdif5002Leaf1,2设备组K8S_VRF的路由表：/24 nextHop： interface：vbdif5000/24 nextHop： interface：vbdif5001/24 nextHop：0 interface：vxlanVTEPLeaf3,4设备组VM_VRF的路由表：/24 nextHop： interface：vbdif5002/24 nextHop： 0 interface：vxlan/24 nextHop：0 interface：vxlanBL1,2设备组K8S_VRF的路由表：

34、/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan/24 nextHop：VM_VRFBL1,2设备组VM_VRF的路由表：/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan/24 nextHop：K8S_VRF/24 nextHop：K8S_VRF12345 VPC互通模型（Network Overlay L2桥接模式）CE1800VK8S Node1MLAG

35、Kni0Linux vRoutereBGPeBGPPOD2eth0Loopback1vbdif5000eth0POD1eth0K8S_VRFLeaf1Loopback1vbdif5000K8S_VRFLeaf2peer linkSpine1Spine2VTEPCE1800V的路由表：42，islocal=1 ， out_ifindex：calicoxxx 43，islocal=1 ， out_ifindex：calicoxxx nextHop：，out_ifindex：eth0 43/3242/32Leaf1,2设备组K8S_VRF的路由表：28/26 nextHop： interface：v

36、bdif5000/0 nextHop：0 interface：vxlanLeaf3,4设备组VM_VRF的路由表：/24 nextHop： interface：vbdif500228/26 nextHop：0 interface：vxlan123OVSMLAGeth0VM_VRFLeaf3vbdif5002VM_VRFLeaf4peer link/24/24/24vbdif5002VTEPK8S_VRFBL1K8S_VRFBL2peer linkVTEP0VM_VRFVM_VRFBL1,2设备组K8S_VRF的路由表：28/26 nextHop： interface：vxlan/0 nextH

37、op：VM_VRFBL1,2设备组VM_VRF的路由表：/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan/32 nextHop： interface：vxlan28/26 nextHop：K8S_VRF456VPC互通模型（Network Overlay L3路由模式）api servervalidating admission controllercalico network controllercalico mastercalico etcdcalicoctlcalico kube controllerac monitorma

38、sterCE1800Vcalico nodenodePODeth0PODeth-bondLeaf1Leaf2Leaf3Leaf4Spine1Spine2eth0eth-bondMLAGVTEPMLAGVTEPuser(kubectl)Network Overlay L2桥接模式Pod发放流程CE1800VCE1800V CNI pluginAC API Server WatcherAPI ServerK8S masterK8S NodeETCDkubeletdockerdrunceth0M-LAG12345689ETCDPod GW54BDPod GW54BDDVRPodeth0VTEP7Ne

39、twork Overlay L3路由模式固定IP场景Pod发放流程CE1800VCNI pluginAC API Server WatcherAPI ServerK8S masterK8S NodeETCDkubeletdockerdrunCLeaf1Leaf2eth0M-LAG12436958kni0vRouterEBGPEBGPETCDPodeth079Network Overlay L3路由模式随机IP场景Pod发放流程CE1800VCNI pluginAC API Server WatcherAPI ServerK8S masterK8S NodeETCDkubeletdockerdr

40、unCLeaf1Leaf2eth0M-LAG12436kni0vRouterEBGPEBGPETCDPodeth05Host 1vRouterpod1pod2pod3AgentKubernetes MasterPolicyAPI URLPolicy ControllerListenerPolicyHost nvRouterpod1pod2pod3AgentClient123Network Policy Kubernetes masterK8S ETCDk8s node(BM)PODCE1800Veth0K8S API server CE1800V CNIkubelet3Network policy create4AC ETCDcalico kube controller12API ServerK8S masterCalico ETCDETCD