kubernetes cluster in large scale Patrickxie ( 谢谆志） Background Cloud has been the general trend. How to manage so many clusters ,resources and businesses How to ensure load balancing of cluster nodes resources and businesses Cluster1 Cluster2 Cluster2 … Region N … Cluster1 Cluster2 … Cluster2 Business 1 Business 2 Business 3 Business N … How to ensure load balancing of cluster nodes ？ Dynamic-Scheduler Controller of native K8S use only one goroutine to handle the scaling of all businesses in the cluster, and personalization configuration is not supported. Worker HPA1 HPA2 … HPAn HPA1 HPAPlus-controller

Wanted: Container Orchestrator! Kubernetes in 5 min https://youtu.be/PH-2FfFD2PU Kubernetes Cluster ​Docker ​Run One Container at a Time What is Kubernetes? 12 OS App 1 Bins/Libs App 2 Bins/Libs ContainerImage2 Replicas: 2 Kubernetes 101 at the Highest Level • Container Cluster = “Desired State Management” – Kubernetes Cluster Services (w/API) • Node = Container Host w/agent called “Kubelet” • Application Leader based clustering • Can be clustered across Master Nodes • Contains all state known about cluster • Kubernetes Front-end Control Plane • Provides RESTful interface • Returns state objects as JSON

Methods to achieve high SLOs on a large scale Kubernetes cluster Kang FAN, Jinghua YAO Why SLO? SLO (Service-Level Objective). Within service-level agreements (SLAs), SLOs are the objectives that must What we concern about Large k8s Cluster What happened about the cluster 1 Is there something unexpected happened in the cluster 2 What end users did in the cluster How to locate failure 1 Which component of the pod to failure Is the cluster healthy 1 Are all software components working fine 2 How many failures occurred on the cluster SLIs on Large k8s Cluster 1. Cluster health state A combination value

Background: Cluster Scale • Production environment: • Dozens of Cluster • 5k+ Nodes / Cluster • 10k+ Nodes / largest Cluster • Testing environment • Hundreds of Cluster for CI/CD • 500+ Nodes / Cluster Background: Background: Operation Requirements • Setup & Teardown Cluster fast and convenient • Add & delete Node at any time • Upgrade Master & Node Components reliably • Canary Rollout • Master & Node Component Kube-on-Kube-Operator: Overview • Biz-Cluster: used to deploy our application. • Meta-Cluster: used to set up Biz-Cluster master components. We add Biz-Cluster master nodes to Meta-Cluster. • User: SREs who can

aliyuncs.com/google_containers controlPlaneEndpoint: "k8s-master:6443" networking: dnsDomain: cluster.local podSubnet: 10.98.0.0/16 serviceSubnet: 10.96.0.0/16 --- apiVersion: kubeproxy.config -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' Step2: k8s node 上执行加入到 k8s cluster kubeadm join k8s-master:6443 --token xvxx9v.ugbbvrdncqv061hk \ --discovery-token-ca-cert-hash sessionAffinity: None type: ExternalName externalName: kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local kubectl apply –f service.yaml kubectl apply –f service2default.yaml Step7: 创建 SSL

proxy/#!/overview? namespace=default 参考： https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/ 02-安装单机版Kubernetes 9 使⽤Kubespray部署⽣产可⽤的Kubernetes集群 （1.11.2） 前提：科学上⽹，或⾃⾏将gcr 部署⽅案 优点 缺点 Kubeadm 官⽅出品 部署较麻烦、不够透明 Kubespray 官⽅出品、部署较简单、懂Ansible就能上⼿ 不够透明 RKE 部署较简单、需要花⼀些时间了解RKE的cluster.yml配置 ⽂件 不够透明 ⼿动部署 第三⽅操作⽂ 档 完全透明、可配置、便于理解K8s各组件之间的关系 部署⾮常麻烦，容易出 错 其他诸如Kops之类的⽅案，由于⽆法跨平台，或者其他因素，被我pass了。 contrib/inventory_builder/inventory.py ${IPS[@]} 此时，会看到 inventory/mycluster/host.ini ⽂件内容类似如下： [k8s-cluster:children] 03-使⽤Kubespray部署⽣产可⽤的Kubernetes集群（1.11.2） 12 kube-master kube-node [all]

server地址 --pod-network-cidr=10.244.0.0/16 \ # pod容器网段 --service-cidr=10.7.0.0/16 \ # service网段，即cluster ip网段 --ignore-preflight-errors=Swap \ #忽略swap未关闭而导致的检查错误 --image-repository="cof- lee.com:5443/k8s" cof-lee.com:5443/k8s kind: ClusterConfigura�on kubernetesVersion: v1.19.4 networking: dnsDomain: cluster.local serviceSubnet: 10.7.0.0/16 podSubnet: 10.244.0.0/16 scheduler: {} --- apiVersion: kubelet server地址 --pod-network-cidr=10.244.0.0/16 \ # pod容器网段 --service-cidr=10.7.0.0/16 \ # service网段，即cluster ip网段 --ignore-preflight-errors=Swap \ #忽略swap未关闭而导致的检查错误 --image-repository="cof- lee.com:5443/k8s"

負載平衡 整合無流量上 限的Google Cloud Load Balancer Demo: Create Your First Service in 10 Mins Run your cluster the way Google does GKE On-Prem ● Turn-key, production-grade, conformant Kubernetes with best-practice Solutions ALPHA IN FALL Run your cluster the way Google does ● The same tools are used to install, configure, and manage clusters in GKE and GKE On-Prem ● Cluster environments are consistent (k8s Installation and Configuration $ gke-on-prem create cluster --dry-run Welcome! This command will take you through the installation of a cluster. --dry-run saves your configuration to a YAML file. Please

Extensibility/Integration • CI/CD examples • Future plan Overall Architecture Kubernetes Cluster Kubernetes Cluster Node Node Node Node Job Job Job Job Pod Pod Pod Pod ElasticSearch ElasticSearch Logging Consistent data Node group of build nodes Node group of user applications Scheduling customization Cluster Resource Auto Scaling kubelet can do image GC DevOps Service DevOps Operator DevOps Operator • Alertmanager to invoke various alert and related actions docker registry Kubernetes Cluster Kubernetes Cluster CRD and Operator Design BuildJob DevOps Operator Job Job Job Job BuildJob BuildJob

Head of distributed system Self driving infrastructure Topics ● Cluster management systems ● Today’s problems with operating cluster management systems ● A self-driving approach Motivation: microservices a lot of components ○ dynamic dependencies ○ fast deployment iteration ● Solution: automation Cluster management system ● Automation ○ Scheduling ○ Deployment ○ Healing ○ Discovery/load balancing ○ lot of manual/semi-manual work No standard way to approach all the problems do it wrong, lose the cluster! gcc // gcc source code #include int main() { compile_c(argv[1]); } gcc Self hosting