been the general trend. How to manage so many clusters ,resources and businesses How to ensure load balancing of cluster nodes 1 2 Improper resource requests 3 Multi-tenant resource preemption How Business N … How to ensure load balancing of cluster nodes ？ Dynamic-Scheduler Node1 Node2 Kube-scheduler Pod Request Load Level Request Load Level Real Load Level Real Load Level Assigned to Node2 but high load, while some nodes have high resource requests but low load. Dynamic-Scheduler Node1 Node2 Kube-scheduler Pod Request Load Level Request Load Level Real Load Level Real Load Level

#设置时区 # systemctl restart chronyd # chronyc sources -v ⑦设置网络参数 #首先加载模块 # cat > /etc/modules-load.d/br_ne�ilter.conf < /etc/modules-load.d/k8s-ipvs.conf <load导入；或者使 用内部registry仓库（内部registry镜像仓库里要有以上7个镜像） ★直接使用命令行方式初始化集群 （以下是非HA模式的master初始化，如果要部署高可用集群，则参考第4章）

annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb"） 内部及外部Load Balancer 支持（internal LB支持方式 -> annotation: service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0） © 2019, Amazon Service load balancer: Network Load Balancer apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx annotations: service.beta.kubernetes.io/aws-load-balancer-type:

Services etcd Pods Endpoints Controller Manager Node Node … Load Balancer Endpoints Controller KubeProxy Pod Node Load Balancer KubeProxy Pod Pod Pod Pod Pod Kubernetes�Service�� Iptables��Service���� 16K LOCs • 200+ �� • ��1.8 Alpha��, Owner @Huawei���1.9�beta • ��ClusterIP�NodePort�External IP�Load Balancer�OnlyLocalNode… • ��iptables�SNAT����� Kubernetes�Service�� Iptables��Service���� ��iptables�������

dhcpOptions/{{ ClusterName }} {{ end }} # --------------------------------------------------------------- # Load Balancer - Masters # # Attach each master ASG to the ELB # ----------------------------------- dhcpOptions/{{ ClusterName }} {{ end }} # --------------------------------------------------------------- # Load Balancer - Masters # # Attach each master ASG to the ELB # -----------------------------------

automation Cluster management system ● Automation ○ Scheduling ○ Deployment ○ Healing ○ Discovery/load balancing ○ Scaling Scheduling Scheduler Scheduler Scheduling Scheduler Scheduling color=yellow Discovery Select color = yellow color=yellow Discovery yellow.mycluster Select color = yellow Load balancing Controller manager 5 Healing Controller manager 5 Healing Controller manager 5 Healing

Flexible and dynamic resource management Dynamic Scheduler is to solve the problem of unbalanced node load in the cluster. Ø Base on current and history node resource usage. Ø Extend Predicate & Priority Ø Reduce container OOM from bursted traffic. Ø Improve cluster resource utilization. Ø If a node load is high, VWA can quickly adjust offline containers‘ cgroup to ensure the SLA of online tasks without

… • 找出集群不正常工作的原因 : ( Kubernetes 典型异常 组件异常 • API Server Load Balancer 异常 • API Server Pod 异常 影响 • 通过 API Server 访问集群概率失败 • 升级集群失败 Load Balancer Service Master API Server Pod Master API Server

Resource Management with vSphere DRS What is DRS? The vSphere Distributed Resource Scheduler (DRS) is a load balancer for VMs deployed on a hypervisor cluster. It has advanced features that can provider actual independent of DRS, but the combination of the two in a cluster is recommended. This will enable load balancing and application of affinity/anti-affinity rules Deploying HA Hosts in an HA cluster are

API 存储 Kube-APIServer Webhook ETCD 数据构建 压测场景 压测环境 压测报告 压测平台 监控&大盘• APIServer & ETCD & Webhook Load balance ETCD Webhook Client Kubelets APIServer Http2 -> http1.1 Upgrade Etcd client v3.3.15 cpu分配策略 2. 应用/单机打散策略 3. 应用互斥/亲和策略 4. …… Pod 1. CPU精细化分配 2. 应用AZ/Node打散 3. CPU敏感Pod打散 4. 节点CPU/Load感知 5. Pod近期最大cpu利用率感知• 节点负载均衡 规模化容器调度 Agent Pod Pod Pod Node 离线数据统计 应用预估峰 CPU值CR Agent Pod