Schedule Ceres Job Queue Manager Spark-Operator OfflineJobs Scheduler Kubeflow Hybrid Deploy StatefulSetPlus-Operator Tencent Cloud Mesh MultiCluster-Route-Manager Application & Route Management Management VWA Controller (Vertical Workload Autoscaler) HPAPlus Controller HNA Controller Auto Scale CronHPA Controller CLB-Service/Ingress-Controller Efficient and reliable container release Ø Why Support HPA, CronHPA, VWA (Vertical Workload Autoscaler) Ø Keep share memory during Pod upgrade Ø Scaled Up with LGV (Last Good Version) Ø Per Pod Per PV Ø Per Workload Per PV Ø Pod Auto Migrate when

The Kubernetes Master Node Basic Components Master Node ETCD kube-apiserver kube-controller-manager kube-scheduler • Key/Value Store • Leader based clustering • Can be clustered across Master Nodes Watches shared state through apiserver • Makes changes from current to desired • Policy-based workload scheduler • Topology aware • Assists with availability, performance and capacity • Affinity/Anti-Affinity The Kubernetes Worker Node Basic Components Master Node ETCD kube-apiserver kube-controller-manager kube-Scheduler Worker Node CRI-containerd Kubeproxy Kubelet • Container Runtime Interface

image - replicas - port 抽象 Deployment - image - replicas Service - port 原始 k8s API 资源 Workload - image - replicas Rollout - canary ArgoRollout - image - replicas - rollout Deployment 做抽象容易形成“谷仓” • 一个抽象满足不了所有场景，所以… 有状态应用 PaaS 无状态应用 PaaS Serverless PaaS 用户 Kubernetes Cert Manager Ingress Let’s Encrypt Flagger Virtual Service Manual Scaling App CRD HPA Knative Service interoperability Application Application Application Platform foo Platform bar Serverless baz Common Workload Types Manual Scaler K8s Operators Kubernetes + OAM K8s Plugin HPA Deployment scale-to-0 Function

Cloud Native batch system (Volcano) development • IBM spectrum computing - Cluster resource and workload scheduling platform development l Gaps for Spark • Agenda l Why Spark on Kubernetes l Volcano com/apache-spark-on-k8s/spark l The goal is to bring native support for Spark to use Kubernetes as a cluster manager like YARN, or Mesos. l Spark 2.3 added native support for Kubernetes. l Spark 2.4 added support proportion/namespace fair-share, job fair-share to share resource l Use task-topology to improve the spark workload efficiency. Summary p Queue priority p Queue reclaim p Queue plugin p Hierarchical queue p

project was to enable GPU on Kubernetes with vSphere. Also actively contributing to kubelet, device manager, device plugin area. GitHub: @figo Steve Wong Hui Luo Presenter Bios 3 Abstract ​Kubernetes NUMA? Memory intensive workloads Nearly all database servers (e.g. Oracle, MongoDB), present a workload which will attempt to detect and consume as much of the system’s memory as possible. Where does 14 Using a NUMA aware hypervisor to solve issues now VM composition guidelines • Assuming you workload fits with the footprint of a single node, compose worker node VMs as “walled gardens” corresponding

yellow.mycluster Select color = yellow Load balancing Controller manager 5 Healing Controller manager 5 Healing Controller manager 5 Healing People love automation! I hate Kubernetes! I hate to - Systemctl start kubelet Installation - master - SSH - Install scheduler - Install controller manager - Install API server - Config them correctly - Start them Installation - etcd - SSH - Install etcd -f kube-apiserver.yaml $ kubectl apply -f kube-scheduler.yaml $ kubectl apply -f kube-controller-manager.yaml $ kubectl apply -f kube-proxy.yaml Simplify k8s lifecycle management Manage your cluster with

多种资源，例如Deployment，Job等，都是构建 在Pod的基础概念之上的。 § 用户可以通过kubectl配合描述资源的yaml文件 创建这些资源 Helm • The package manager for Kubernetes • Easy to create, version, share, and publish — so start using Helm and stop the – CouchDB – Redis – Zookeeper – Nginx StatefulSet/DaemonSet • StatefulSet is the workload API object used to manage stateful applications. Manages the deployment and scaling of a set

思考题： 有状态的复杂应用如何管理？ 基础设施能力还如何演进和透出？ 研发自己的诉求如何传达给运维和基础设施？ K8s 扩展能力的真实情况 我的 Zookeeper 该用 哪种K8s Workload 接入？ 你恐怕得写个 Operator…… Operator是啥？ CRD Controller Informer Reflector Event Handler Loop … 我们业务压力大 2. A list of overwritable parameters (schemas) 1.Description of the application Component 核心workload 可访问 可复制 长久运行 Server √ √ √ Singleton Server √ × √ Worker × √ √ Singleton Worker × × √ Task × as value Operational hint from developers to operators Component 从 CRD 到 扩展 Workload OpenFaaS CRD OpenFaaS 扩展Workload 可发现、可管理的运维能力：OAM Traits System 发现运维能力 查看能力用法 绑定能力给应用 提前暴露冲突 kubectl get traits

dev/v1alpha2 kind: Component metadata: name: frontend annotations: description: Container workload spec: workload: apiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: web get deployment NAME REVISION AGE frontend-c8bb659c5 1 2d15h $ kubectl get components NAME WORKLOAD frontend deployment.apps.k8s.io Component：应用中的一个组成部分，例如容器、 Function或者云服务等 应用组件 运维能力 扩容策略 io services.k8s.io route route.core.oam.dev apps.k8s.io tls tls.core.oam.dev apps.k8s.io Workload 与 Trait 注册与发现机制 # 示例：将 Istio VirtualService 注册为平台 的流量管理能力 示例: 使用 OAM 模型管理应用 1. 创建应用组件 2. 绑定运维特征

Automation; kubebuilder + controller-runtime + helm Operator Capability Levels Installation of the workload • Operator deploys an Operand or configures off- cluster resources • Operator waits for managed resources to the user leveraging the status block of the Custom Resource Configuration of the workload • Operator provides configuration via the spec section of the Custom Resource • Operator reconciles configuration and updates to it with the status of the managed resources Upgrade of the managed workload • Operand can be upgraded in the process of upgrading the Operator, or • Operand can be upgraded