28Kubernetes资源pod健康检查机制

Kubernetes资源Pod健康检查机制目录readiness健康就绪01任务liveness在线健康检查02任务startup启动性探测03任务学习目标【知识目标】掌握Kubernetes资源对象pod的基本使用。【技能目标】能够掌握Kubernetes资源对象pod健康检查机制。Kubernetes提供健康检查机制，当发现应用异常时会自动重启容器，将应用从service服务中剔除，保障应用的高可用性。Kubernetes定义了三种探针：readiness：用于判断应用是否已经就绪完毕，即是否可以接受外部转发的流量，健康检查正常则将pod加入到service的endpoints中，异常则从endpoints中删除。探测类型

liveness在线检查机制，检查应用是否可用，如死锁，无法响应，异常时会自动重启容器。探测类型

startup启动检查机制会探测容器中的应用是否已经启动。如果提供了启动探测startupprobe，则禁用所有其他探测，直到它成功为止。如果启动探测失败，kubelet将杀死容器，容器服从其重启策略进行重启。如果容器没有提供启动探测，则默认状态为成功Success。探测类型每种探测机制支持三种健康检查方法，exec提供命令或shell的检测，在容器中执⾏命令检查，返回码为0健康，非0异常httpGethttp协议探测，在容器中发送http请求，根据http返回码判断业务健康情况，返回码⼩于400即表示健康tcpSockettcp协议探测，向容器发送tcp建立连接，能建立则说明正常其中exec通用性最强，适用与大部分场景，tcpSocket适用于TCP业务，httpGet适用于web业务。检查机制每次探测都将获得以下三种结果之一：Success：表示通过检测。Failure：表示未通过检测。Unknown：表示检测没有正常进行。探测结果探针(Probe)有许多可选字段，可以用来更加精确的控制Liveness和Readiness两种探针的行为：initialDelaySeconds：容器启动后要等待多少秒后探针开始工作，单位“秒”，默认是0秒，最小值是0periodSeconds：执行探测的时间间隔（单位是秒），默认为10s，单位“秒”，最小值是1。Pod探针相关的属性timeoutSeconds：探针执行检测请求后，等待响应的超时时间，默认为1，单位“秒”。successThreshold：连续探测几次成功，才认为探测成功，默认为1，在Liveness探针中必须为1，最小值为1。failureThreshold：探测失败的重试次数，重试一定次数后将认为失败，在readiness探针中，Pod会被标记为未就绪，默认为3，最小值为1Pod探针相关的属性1liveness在线健康检查1定义一个容器，启动时创建一个文件，健康检查时ls-l/opt/liveness.log，返回码为0，健康检查正常，30秒后将其删除，返回码为非0，健康检查异常。[root@node-1~]#viliveness.yamlspec:containers:livenessProbe:exec:command:-ls--l-/opt/liveness.loginitialDelaySeconds:5#容器初始化时间为5秒periodSeconds:5#多久进行一次健康探测5秒timeoutSeconds:1#多久定义为超时1秒successThreshold:1#执行多少次是成功的1次liveness在线健康检查exec命令行健康检查1应用配置生成容器[root@node-1~]#kubectlapply-fliveness.yamlpod/liveness-exec-democreatedliveness在线健康检查exec命令行健康检查1查看容器的event日志，容器启动后，23秒以内容器状态正常，23秒开始执行liveness健康检查，检查异常，触发容器重启。[root@node-1~]#kubectldescribepodsliveness-exec-demoEvents:TypeReasonAgeFromMessage-------------------------NormalScheduled114sdefault-schedulerSuccessfullyassigneddefault/liveness-exec-demotonode-2NormalPulled33s(x3over113s)kubelet,node-2Containerimage"busybox:latest"alreadypresentonmachineNormalCreated33s(x3over113s)kubelet,node-2Createdcontainerliveness-exec-demoNormalStarted33s(x3over113s)kubelet,node-2Startedcontainerliveness-exec-demoWarningUnhealthy23s(x9over107s)kubelet,node-2Livenessprobefailed:/opt/liveness.logls:l:Nosuchfileordirectory#23秒后检测时找不到文件NormalKilling23s(x3over103s)kubelet,node-2Containerliveness-exec-demofailedlivenessprobe,willberestartedliveness在线健康检查exec命令行健康检查1如下定义⼀个nginx应⽤，通过探测http://:port/index.html的⽅式判断健康状态[root@node-1~]#viliveness-http.yamlspec:containers:livenessProbe:httpGet:port:80scheme:HTTPpath:/index.htmlinitialDelaySeconds:3periodSeconds:5timeoutSeconds:1liveness在线健康检查httpGet健康检查1在node-1节点创建pod/livess-http[root@node-1~]#kubectlapply-fliveness-http.yamlpod/livess-httpcreated[root@node-1~]#kubectlgetpodslivess-http1/1Running033sliveness在线健康检查httpGet健康检查1查看Pod设置[root@node-1~]#kubectlgetpodslivess-http-oyamlspec:containers:-image:nginx:1.7.9imagePullPolicy:IfNotPresentlivenessProbe:failureThreshold:3httpGet:path:/index.htmlport:80scheme:HTTPinitialDelaySeconds:3periodSeconds:5successThreshold:1timeoutSeconds:1name:nginxports:-containerPort:80name:http-80protocol:TCPliveness在线健康检查httpGet健康检查1手动删除index.html文件模拟故障，此时发送http请求时会健康检查异常，会触发容器⾃动重启。[root@node-1~]#kubectlexec-itlivess-http/bin/bashkubectlexec[POD][COMMAND]isDEPRECATEDandwillberemovedinafutureversion.Usekubectlkubectlexec[POD]--[COMMAND]instead.root@livess-http:/#cd/usr/share/nginx/html/root@livess-http:/usr/share/nginx/html#ls50x.htmlindex.htmlroot@livess-http:/usr/share/nginx/html#rm-rfindex.htmlroot@livess-http:/usr/share/nginx/html#exitexitliveness在线健康检查httpGet健康检查1查看pod的列表，此时RESTART的次数会增加1，表示重启过⼀次查看pod的详情，观察容器重启的情况，通过Liveness检查容器出现404错误，触发重启。liveness在线健康检查httpGet健康检查1以nignx为例使用tcp健康检查机制，探测80端口的连通性。[root@node-1~]#viliveness-tcp.yamlspec:containers:-name:nginximage:nginx:1.7.9imagePullPolicy:IfNotPresentports:-name:http-80containerPort:80protocol:TCPlivenessProbe:tcpSocket:port:80initialDelaySeconds:3periodSeconds:5timeoutSeconds:1liveness在线健康检查tcpSocket健康检查1应用配置创建容器[root@node-1~]#kubectlapply-fliveness-tcp.yamlpod/livess-tcpcreated[root@node-1~]#kubectlgetpodsNAMEREADYSTATUSRESTARTSAGElivess-tcp1/1Running05s[root@node-1~]#kubectldescribepodlivess-tcpLiveness:tcp-socket:80delay=3stimeout=1speriod=5s#success=1#failure=3liveness在线健康检查tcpSocket健康检查1模拟故障，动态获取pod所属节点，登录到pod中，安装查看进程工具http[root@node-1~]#kubectlgetpodslivess-tcp-wNAMEREADYSTATUSRESTARTSAGElivess-tcp1/1Running016s查看容器详情，发现容器有重启的记录。[root@node-1~]#kubectldescribepodlivess-tcpEvents:TypeReasonAgeFromMessage-------------------------NormalStarted4mkubelet,node-2StartedcontainernginxWarningBackOff0skubelet,node-2Back-offrestartingfailedcontainerliveness在线健康检查tcpSocket健康检查2readiness健康就绪2创建一个pod，使用httpGet的健康检查机制，定义readiness就绪检查探针检查路径/health。[root@node-1~]#viliveness-http.yamlspec:containers:livenessProbe:httpGet:port:80scheme:HTTPpath:/index.htmlinitialDelaySeconds:3periodSeconds:5timeoutSeconds:1readinessProbe:httpGet:port:80path:/healthscheme:HTTPreadiness健康就绪2pod已经启动起来了，但是状态并没有ready[root@node-1~]#kubectldelete-fliveness-http.yamlpod"livess-http"deleted[root@node-1~]#kubectlapply-fliveness-http.yamlpod/livess-httpcreated[root@node-1~]#kubectlgetpodsNAMEREADYSTATUSRESTARTSAGElivess-http0/1Running032sreadiness健康就绪2查看健康机制和事件[root@node-1~]#kubectldescribepodlivess-httpLiveness:http-gethttp://:80/index.htmldelay=3stimeout=1speriod=5sReadiness:http-gethttp://:80/healthdelay=5stimeout=1speriod=10sEvents:TypeReasonAgeFromMessage-------------------------NormalScheduled2m53sdefault-schedulerSuccessfullyassigneddefault/livess-httptonode-2NormalPulled2m52skubelet,node-2Containerimage"nginx:1.7.9"alreadypresentonmachineNormalCreated2m52skubelet,node-2CreatedcontainernginxNormalStarted2m52skubelet,node-2StartedcontainernginxWarningUnhealthy10s(x16over2m40s)kubelet,node-2Readinessprobefailed:HTTPprobefailedwithstatuscode:404readiness健康就绪2检查应用[root@master1~]#kubectlgetpodslivess-http-wNAMEREADYSTATUSRESTARTSAGElivess-http0/1Running057s4[root@node-1~]#curl-I3/healthHTTP/1.1404NotFoundServer:nginx/1.7.9Date:Tue,28Jul202007:31:06GMTContent-Type:text/htmlContent-Length:168Connection:keep-alivereadiness健康就绪2服务发现[root@node-1~]#kubectlexposepodlivess-httpservice/livess-httpexposed[root@node-1~]#kubectlgetservicesNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGEapp-demoClusterIP30<none>80/TCP12dapp-nginx-demoClusterIP29<none>80/TCP12dKubernetesClusterIP<none>443/TCP13dlivess-httpClusterIP1<none>80/TCP19m[root@node-1~]#curl1curl:(7)Failedconnectto1:80;Connectionrefusedreadiness健康就绪2[root@node-1~]#kubectldescribeservicelivess-httpName:livess-httpNamespace:defaultLabels:run=livess-httpAnnotations:<none>Selector:run=livess-httpType:ClusterIPIP:1Port:<unset>80/TCPTargetPort:80/TCPEndpoints:SessionAffinity:NoneEvents:<none>未获取到endpoints,无法访问nginx服务器readiness健康就绪2手动创建health文件，让它能够探测到这个文件。[root@node-1~]#kubectlexec-itlivess-http/bin/bashkubectlexec[POD][COMMAND]isDEPRECATEDandwillberemovedinafutureversion.Usekubectlkubectlexec[POD]--[COMMAND]instead.root@livess-http:/#cd/usr/share/nginx/html/root@livess-http:/usr/share/nginx/html#ls50x.htmlindex.htmlroot@livess-http:/usr/share/nginx/html#echook>healthroot@livess-http:/usr/share/nginx/html#exitexitreadiness健康就绪2查看service的详情，获得endpoint[root@node-1~]#kubectldescribeservicelivess-httpName:livess-httpNamespace:defaultLabels:run=livess-httpAnnotations:<none>Selector:run=livess-httpType:ClusterIPIP:1Port:<unset>80/TCPTargetPort:80/TCPEndpoints:3:80SessionAffinity:NoneEvents:<none>readiness健康就绪2查看service的集群IP地址。[root@node-1~]#kubectlgetservicesNAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGElivess-httpClusterIP1<none>80/TCP32m[root@node-1~]#curl1<!DOCTYPEhtml><html><head><title>Welcometonginx!</title>[root@node-1~]#curl1/healthokreadiness健康就绪3startup启动性探测3创建一个启动性探测案例：[root@node-1~]#viliveness-http.yamlspec:containers:-name:nginximage:nginx:1.7.9imagePullPolicy:IfNotPresentports:-name:http-80containerPort:80protocol:TCPstartupProbe:httpGet:port:80path:/startup.logperiodSeconds:10failureThreshold:30

startup启动性探测3startup启动性探测启动pod[root@node-1~]#kubectldelete-fliveness-http.yamlpod"livess-http"deleted[root@node-1~]#kubectlapply-fliveness-http.yamlpod/livess-httpcreated[root@node-1~]#[root@node-1~]#kubectlgetpodsNAMEREADYSTATUSRESTARTSAGElivess-http0/1Running073s3startup启动性探测启动失败[root@node-1~]#kubectlde