谷歌容器集群管理系统实践

1、谷歌容器集群管理系统实践Engineering in GoogleDevelopment20% time side projectwork on any project you wantdashboard for posting jobsyou can do more than you think!Be SRE for 2 or 3 months (20% more salary!)SRE (Site Reliability Engineering)50% time for developmentto automate routine tasksscale nicelySRE consists

2、 of two categories: 55% SDE, and 45% OpsSay no to development teamerror quota, e.g. 99.95% slo monthly means 4hrs impactboth SRE and Development team mange the riskAbility to change code basePostmortemNot to blame, but to find problems and fix itContainerReasonprimary goal: save money - VM is heavyh

3、igh-density and performancefast to startStart Container Journey2004 - ?Everything runs in containeruse container for decade2B container a weekLimited Isolation chrootcgroupslxclmctfy2004200620082013user namespace20142007namespaces?2015OCIContainerImage from RedhatLimited Isolation chrootcgroupslxclm

4、ctfy2004200620082013user namespace20142007namespaces?2015OCIcgroupresource isolation (cpu, memory, blockio, etc.)ContainerLimited Isolation chrootcgroupslxclmctfy2004200620082013user namespace20142007namespaces?2015OCIcgroup v1 - v2multiple hierarchysubcontainer and asymmetric isolationv2: unified h

5、ierarchyImage from Redhatsub1sub2sub3subcontainerCPUMemNetcontainer1container2container3asymmetric isolationcg1cg10cg11cg100cg101.ContainerLimited Isolation chrootcgroupslxclmctfy2004200620082013user namespace20142007namespaces?2015OCIImage from ToptalNamespacesisolate processs view of the operating

6、 environmentMount, UTS, IPC, PID, Network, UserSupport added fair recentlyPrimarily Linux cgroupsnamespace is done at user-space policiesContainerlxc? lmctfy?internal version of Imctfy exists long before lxclxc: no strong abstractionnamespace abstraction + raw cgrouplxc: no programmable APImust be b

7、uilt to work with other toolslmctfy: more abstraction and enhancemente.g. subcontainer and asymmetric isolatione.g. quality of serviceOCIContainer: runtime + imageBased on DockerLimited Isolation chrootcgroupslxclmctfy2004200620082013user namespace20142007namespaces?2015OCIContainer?Container manage

8、mentclustering is the hard parthundreds of engineerings building cluster management systemBorg: conceptsConfigurationBorg configuration languageMost hated languageJob and TaskJob: unit of deployment, including resources requirement, number of tasks, etcTask: unit of running entityResource container:

9、 the container to run taskJobTask1Task2Borg: conceptsAllocSet and Allocco-scheduling: e.g. logsaver and applicationshare resources for all taskspersist data when tasks exitJobTask1Task2host bind mount/cg11/cg12Borg: conceptsSchedulingpriorityresources: resource has priorityquota: quota has priorityp

10、ackages: vs docker imagemachine constraints%ports% (BNS, chubby)options:repeated failuresquorum requirementetcRuntimeapplication classresource estimation:decrease resource reservation (gradual decay function)borg config fileborgcfg1RPC/Protocol BufferPrimary BorgMasterschedulerservertask1pkg12Paxos4

11、pkg2resource containerBorglet5task_Nresource containerBorglet7polling6 installing packageBorg Cell8 run + args + resource93AuthN/ZBorg: Request Lifecyclelink shardBorg: The Good, The Bad, The Ugly高稳定性、高自动化、高智能极其复杂的配置文件 最流行语言排行榜？Go, Python, Java, C+, Javascript, 最令人畏惧语言排行榜？Python, Borgcfg, Borgmon, S

12、hell, Borgmaster becomes borg monsterPeople step on each others footOmega: 早年寄予厚望From Monolithic to Shared StatePersistent Storage as the ground truthLook to the Future!Collaboration across the globeA bottom up driven projectOmega: 中年四面楚歌Engineering mayhemTesting cellAPI rewrites, all Borg callers h

13、ave to change (along with behavioral change)!Borgmaster is hard to rewrite向现实妥协：项目进度与Borg的关系Omega: 晚年戛然而止工程与理论的差距精确度 vs 吞吐量预测的困难Borg的进化性能软件工程终究下架Big shuffleCloud: The Ursquake经济上的巨大回报技术上的巨大领先产品化上的差异PaaS与IaaS的矛盾GAE vs GCEManaged VMKubernetes此处为何地?Google Mountain ViewGoogle PittsburghKubernetes birth

14、placekubernetes乳名?Kubernetesyaml config filekubectl1HTTPSingle MasterschedulerAPI servercon12etcd4KubeletKubernetes Cluster3 AuthN/Zcontroller managercon2infra76installing package5 watchpostcon1Kubeletcon2infra一个Cluster里有一个或多个Borg cellBorg容器之外的其他任务Borg自身谁来管理？机器层面谁来管理？网络层面谁来管理？安全谁来控制？镜像如何管理？Cluster m

15、anagementBorgHareware Ops流程工 具自动系统 安装工具在线硬盘 修复系统生产系统 维护管理 系统节点医生网络 医生耗竭系 统集群工作流 管理系统日历系统Capacity安 全把控系统分布式存储集群数据 库用户和组管 理系统SDN网络自动节能系统负载均衡资源分配扩 容系统集群监测 系统安全扫描系统分布式构 建系统分布式测 试系统调试系统 和工具调试系统 和工具发布管理 工具The entire familyAnd . The RobotDisk EraseDecommissionInventory ManagementAn example workflow: Pre-borg集群工作流管理系统自动系统安装工具集群数据库节点管理agentBorgdriverChubby driver节点医生节点医生agentAn example workflow: with b