云原生容器基础设施运维实践

1、云原生容器基础设施运维实践Manage large scale Kubernetes nodes in Cloud-Native fashionAgenda阿里巴巴节点运维的挑战KubeNode：云原生节点运维底座未来展望阿里巴巴节点运维的挑战规模大数百ASI集群 (Ali Serverless Infra, ACK + Ali addon)数十万节点 (单集群节点最多10k台)数万应用数百万容器环境复杂x86 / ARM / GPU / FPGA在线 (应用类型差异大)、混部、安全容器稳定性要求高在线业务延迟、抖动敏感宕机、夯机业务无感知KubeNode：阿里巴巴云原生节点底座What &

2、Why以云原生方式管理节点生命周期及节点组件申明式、面向终态组成：中心端：Machine Operator ：节点及组件管理Remedy Operator ： 节点故障自愈节点侧：Kube Node Agent：单机 agent配套组件：Kube Defender 统一风控NPD: 单节点故障检测kube-apiserverKube DefenderASI Control PlaneMachine OperatorRemedy OperatorKube Node Agent节点组件(kubelet / docker / npd / .)Node阿里云、AWS、Azure .KubeNode 和

3、社区项目关系/kube-node不相关，该项目2018年初已停止ClusterAPIKubeNode 可以作为 ClusterAPI 节点终态的补充功能对比：Cluster APIKubeNode集群 ProvisionYesNo节点 ProvisionYesYes节点组件终态NoYes节点故障自愈Yes (simple)Yes (full, rule based)CRDsMachine: 节点元信息MachineSet (MS): 节点集合MachineComponentSet (MCS): 节点组件集合MachineComponent (MC): 节点组件ControllersMS con

4、troller: 节 点 provisionMCS controller: 节点组件分批安装、升级Infra Provider: 对接云厂商 OpenAPIKube Node Agent单机组件安装、升级、终态维持KubeNode Machine Operatorkube-apiserverMachine ControllerMachoneComponentSet ControllerKube DefenderMachine OperatorKube Node AgentNode阿里云、AWS、Azure .MachineSet ControllerInfra ProviderMachine

5、/ Machine Component节点组件(kubelet / docker / npd / .)Use Case: 节点导入k8s 扩展 CRD 描述节点及组件MachineMachineComponentMachineComponentSet节点组件确保终态一致versionconfigstatuskube- apiserver多集群管理系统Machine ControllerMachoneComponentSet ControllerKube Node Agent1. 提交操作2. 安装 kube node agent, 3, 提交 Machine CRD4. watch Machi

6、ne CRD, 同步 label/taint etc. and update Machine phase5. watch MachineComponentSet CRD, update MachineComponents to Machine, such as kubelet, pouch, npd, etc.6. watch Machine / MachineComponent CRD, do real operation to install components (kubelet, pouch, npd, etc.), and ensure all components working

7、status is fine.Use Case: 组件升级ASIOpsASI 组件变更统一 CD 平台上百集群 Pipeline 自动流水线发布测试 - 预发 - 正式变更后自动触发健康巡检KubeNode 组件升级逐批次灰度、暂停升级单机 watch 变化触发升级，高并发高 效率健康巡检异常状态上报、暂停自动变更kube- apiserver多集群管理系统MachoneComponentSet ControllerKube Node Agent1. 提交升级操作2. (Loop) 逐批次更新 MachineComponentSet，发布后自动健康巡检3. watch MachineCompo

8、nentSet CRD, update MachineComponents to Machine, such as kubelet, pouch, npd, etc.4. watch Machine / MachineComponent CRD, do real operation to install components (kubelet, pouch, npd, etc.), and ensure all components working status is fine.CRDsNodeRemedier：节点故障修复规则RemedyOperationJob：节点自愈修复任务Contro

9、llersRemedy controller: 自愈控制RemedyJob controller: 自愈任务控制NodeRemedier Registry: 自愈规则注册中心Host Doctor: 中心故障诊断，对接主动运维事件NPD: 节点故障检测 (插件式: kernel/kubelet/docker/)Kube Node Agent单机自愈修复任务执行KubeNode Remedy Operatorkube-apiserverRemedy ControllerRemedyJob ControllerKube DefenderRemedy OperatorKube Node AgentN

10、PD(plugins)节点组件Node阿里云、AWS、Azure .NodeRemedier RegistryInfra ProviderRemedyOperationJobHost DoctorUse Case: 夯机自愈故障自愈NPD - Node Condition - Remedykube- apiserverASI CaptainRemedy ControllerRemedyJob ControllerKube Node Agent1.1. 发布、更新自愈规则1.2. 更新 NodeRemedier 规则4. 生成对应 RemedyOperationJob5. watch Remed

11、yOperationJob CRD, 执行自愈修复， 迁移置换节点上的业务容器，避免夯机影响业务6. watch RemedyOperationJob CRD,执行单机自愈修复操作NPD2. 发现内核死锁日志， 上报故障 ConditionKube Defender3. watch Node CRD, 对接风控是否允许自愈Remedy 自愈优势云原生自闭环自愈链路覆盖广：硬件、OS、组件秒级故障发现、分钟级故障自愈对接风控，防止自愈操作引发二次故障数据体系数据采集链路统一数据采集和存储数据平台应用资源利用率分析统计实时监控报警整体故障分析统计节点组件覆盖度、一致率分析节点自愈效率分析全链路诊断KubeNode 数据体