1、在大規模Kubernetes集群上實現SLOMethods to achieve high SLOs on a large scale Kubernetes clusterWhy SLO?LatencySLIAvailabilityQPSCorrectnessSLOPunishmentSLASLI defines an indicator, which can represent user experience.SLO is the object that try to meets all SLIs in a period of time.SLA = SLO + Punishment.SLO

2、(Service-Level Objective). Within service-level agreements (SLAs), SLOs are the objectives that must be achieved for each service activity, function and process to provide the best opportunity for service recipient success. GartnerSLA/SLO/SLIWhat we concern about Large k8s ClusterIs the cluster heal

3、thy1 Are all software components working fine 2 How many failures occurred on the clusterWhat happened about the cluster1 Is there something unexpected happened in the cluster 2 What end users did in the clusterHow to locate failureWhich component is going wrongWhich component that leads delivery of

4、 the pod to failureSLIs on Large k8s ClusterCluster health stateA combination value indicates the risk in the cluster. Currently Healthy, Warning and Fatal are the possible value.Success rateA rate value indicates the rate of success about creating/upgrading pod.Number of Terminating PodA number val

5、ue indicates the count of pods that can not be deleted in a certain period.Centralized Components AvailabilityA ratio value indicates the time in which the cluster is available. It is used to evaluate the master components.Nodes AvailabilityA number value indicates the number of unhealthy node in th

6、e cluster. Pods scheduled to unhealthy nodes may not be delivered in time, success rate would decrease consequently.The success standard and reason classificationThe success standard:PodFeatureTime limitSuccess conditionPodRestartPolicy=Always1min(example value)the status of .Status.Conditions. “typ

7、e=Ready” is True”Job PodRestartPolicy=Never/OnFailure1min.Status.Phase= Running/Succeeded/FailedTerminating Pod1minpod is removed from etcdUnhealthy NodeTaint/Degrade1minNode has taints or is degradedProcessingBase on the failure reasonUnhealth node is healed or removed.Reason classification:SourceF

8、eatureExampleSystemFailure caused by cluster itselfRuntimeError, ImageFailed, Unscheduled, KubeletDelay.End UsersFailure caused by end usersContainerCrashLoopBackOff, FailedPostStartHook, UnhealthyTrace systemIncrease of SLOData CollectAudit logEventThe unhealthy nodeMonitoringIsolationRecoverDegrad

9、eData AnalysisFailures/MachineFailures/ReasonReportLifecycle of PodFailure ReasonTargetKubeletApiserverSchedulerOperatorRuntimeDaemonsetAlertGray ScaleBug FixSuccess RateSLOThe unhealthy nodeCluster HealthyTerminating Pod NumberDaily ReportValidationHousekeepi ngHigh AvailableFast RecoveryDisplay Bo

10、ardAlertAnalysis PlatformWeekly ReportSLO：Indicate the cluster is healthy or thereis something unexpected happened.Trace system：Collect and analyze logs in cluster. Sowe can known what happened about the cluster.Increase of SLO：Get the weakness of the cluster byanalyzing the failure reasons and effe

11、ctive actions should be taken to increase the success rate.The unhealthy node：Detect the unhealthy machines timely and fix problems automatically。The infrastructureLogEventEnd UserStorageAnalysis PlatformTraceReportWeaknessThe trace systemData Collect:Collect Audit log for the whole cluster.Data ana

12、lysis:Analyze failure reason if pod is failed.Reason analysis:Analyze the failure reasons. Try to find somethingabnormal in the cluster.Trace Result:We can get:It is failed to deliver the pod，and the fail reason isFailedMount.Pod LifecycleFailureReasonTrace System:Node Metricsnodemetricskubelet metr

13、icsdaemonset metricsnode loadslo metricscsi metricsdirty dataWith huge amount of metrics data collected, statistical methods can be used to check whether the node is healthy or not.Besides, node delivery capacity can also be evaluated via historical data.With dirty data metrics which consists ofesca

14、ped/zombie/uninterruptible processorphaned containersorphaned pod directories/volumesorphaned cgroupsorphaned net deviceand so on, node recovery system can cleanup those dirty data or alert cluster admins to process dirty data manually.Unhealthy nodeSLOTraceNPDMetricsEvent SourceruntimeErrorCo ntoll

15、erfailedPodContr ollerDetectorStrategyUnhealthy node listFast TaintWeight AdjustRecoveryManualHandlingImproveAuto Human experienceImprove of strategyCollect data from metrics NPD, Trace system and Log.Analyze the problem of the node, such as DiskRO, critical Daemonset is not ready.Processing unhealt

16、hy node: Heal, Degrade or Isolate. With scoring mechanism and historical operation records, unhealthy nodes can be recovered automatically. Otherwise manual intervention is required.Generate daily report to show what happens, and programmers can improve the system with the reportscontinuously.Daily ReportTips on increasing SLOCase 1: Image DownloadImage lazyload technology provides the ability to run a container without downloading image.Case 2: RetryPod should be recreate when the previous pod is failed. The previous node sho