Model and Operate Datacenter by Kubernetes at eBay 辛肖刚, Cloud Engineering Manager, ebay 梅岑恺, Senior Operation Manager, ebay Agenda About ebay Our fleet Kubernetes makes magic at ebay Model + Controller Controller How we model our datacenter Operation in large scale Q&A About ebay 177M Active buyers worldwide $22.7B Amount of eBay Inc. GMV $2.6B Reported revenue 62% International revenue 1.1B b e Converge & Reconcile WIRI: What it really is Kubernetes Core concept of Kubernetes - Declarative magic What is an application looks like? Replica Config LoadBalancer Rolling Update Quota

Kubernetes Introduction § K8s is a production- grade container orchestration platform § Declarative management of objects using configuration files. § More introductions, go to • K8s official Open Tech Mini Academy @ IBM http://ibm.biz/opentech-ma Kubernetes Resource Model A common resource model can satisfy any deployment requirements § Config Maps § Daemon Sets § Deployments openwhisk-deploy-kube Technical details Deployment • A Deployment controller provides declarative updates for Pods and ReplicaSets. • Stands for a long running task, can be exposed as K8s

orchestration • Automating deployment, scaling, and management of containerized applications • Declarative • Can be a mix of GPU or CPU nodes • Massive Scale • OpenAI dedicates up to 10k cores for a PyTorch, MXNet, Chainer, and more • JupyterHub to create and manage interactive Jupyter notebooks • Model serving – serve exported models with TF Serving or Seldon • Additional components for storage, workflow Demo: Run TensorFlow Training with Containers Demo: Serving the Model with TF Serving • Options for serving • Wrap model in a web framework (eg – Flask) • Tensorflow Serving • Seldon Demo:

contributors can get involved in the SIG. Kubernetes is in the process of moving to a new “out of tree” model, this effort spans all the touching points with the underlying infrastructure: compute, storage, have independent feature and patch release cycles, learn how SIG VMware is working to meet this new model on VMware platforms. Agenda 4 What is the VMware SIG Purpose, Projects managed, How to join Foundry. Cluster API provider for vSphere • The Cluster API is a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management. It provides optional

利用率並最終降低總體成本的工具。 Click to edit Master title style 13 GitOps 痛苦x甜密 • 有能力記錄叢集環境上的一切變化 • 使用宣告式(Declarative)的文件格式 來描述或是設定環境上要用到的所有 資源 • 所有的環境變化都可支援審核機制， 要通過審核才會往下運作 • 權限控管，控制誰有能力去對環境資 源進行更改 • 有辦法針對期望的狀態與運行的狀態 的好朋友 – xxxOperator • Operator 的目標是將 operation 知識 放入軟件中 • Operator 運行在 Kubernetes 集群內 並根據宣告式 (Declarative) 的 CRD 文件來自動化常見的 Day 1和 Day2 的活動。 15 Click to edit Master title style 16 Kube-Prometheus-stack

映射： "annotations": { "key1" : "value1", "key2" : "value2" } 类似以下信息可记录到Annotation中： 由declarative configuration layer管理的字段。将这些字段附加为Annotation，可将它们与客户端或服务器设置的默 认值、⾃动⽣成的字段或以及auto-sizing或auto-scaling的系统所设置的字段区分开。 另起⼀个终端，输⼊： curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/de fault/pods/cpu-demo/metrics/cpu/usage_rate 即可看到监控信息。 在本例中，尽管容器启动时，尝试使⽤2个CPU单位，但由 kubectl proxy curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/de fault/pods/memory-demo/metrics/memory/usage 可看到如下结果： { "timestamp": "2017-06-20T18:54:00Z"

desired and actual config • Action: manage resource to desired config Operator: Advantages • Declarative system • Manage resource to final state continually • kube-apiserver oriented programming •

• Adapt to various internal systems like Route System, CMDB, CI, Security Platform, etc. • Declarative application lifecycle management. • Support big data and AI jobs. • Optimize the isolation of

PaaS 层 UI (e.g. dashboard, cli) 用户 CUE schema/模板 “客户端”抽象 标准化的“服务端”抽象 – 应用模型 Open Application Model (OAM) • 通过 OAM spec 定义“以应用为中心”的原语 • 打破“谷仓”! Common Traits Function Deployment K8s Operator Manual Scaler K8s Operators Kubernetes + OAM K8s Plugin HPA Deployment scale-to-0 Function Unified Model Layer Platform Capability Pool 统一的模型层 平台统一“能力池” 模块化的交付系统 - GitOps “应用”配置 Git (as source of truth) Controller 持续交付 KubeVela “The Extensible Application Platform Based on Kubernetes and Open Application Model (OAM)” KubeVela = OAM Kubernetes Runtime + Capability Center + UI (Cli + Dashboard) KubeVela Ø

environment ○ Volume of data ○ Re-keying method ○ Number of key copies ○ Personnel turnover ○ Threat model ○ New and disruptive technologies, e.g., quantum computers Key rotation: compliance PCI DSS v3 {DEK3}KEKv3 Nov 12-Dec 12 Dec 12 - Jan 11 Jan 11 - Feb 10 KEKv1 KEKv2 KEKv3 KMS plugin: threat model and concerns ● KMS server is compromised ● KMS plugin is compromised ● Auth token for KMS - offline In external secret store Kubernetes secrets: summary ● Use encryption based on your threat model, e.g., two layers, like full-disk + application-layer ● Rotate keys regularly to limit the impact