Model and Operate Datacenter by Kubernetes at eBay 辛肖刚, Cloud Engineering Manager, ebay 梅岑恺, Senior Operation Manager, ebay Agenda About ebay Our fleet Kubernetes makes magic at ebay Model + Controller Controller How we model our datacenter Operation in large scale Q&A About ebay 177M Active buyers worldwide $22.7B Amount of eBay Inc. GMV $2.6B Reported revenue 62% International revenue 1.1B 1B Live listings 81% Sold are new 88% Fixed price $11B Mobile Our fleet 15 3 US Data Centers POPs 200K+ Managed Vms 4K 100K Managed BMs Applications 4.5PB Managed Storage All of us know

A Day in the Life of a Data Scientist Conquer Machine Learning Lifecycle on Kubernetes Brian Redmond • Cloud Architect @ Microsoft (18 years) • Azure Global Black Belt Team • Live in Pittsburgh, PA Repeatable/consistent • CI/CD • This has worked well for App Dev. Now time for AI/ML • But, must ensure data scientist are not hindered by structure Why Containers, Kubernetes & Helm? • Container • Contains PyTorch, MXNet, Chainer, and more • JupyterHub to create and manage interactive Jupyter notebooks • Model serving – serve exported models with TF Serving or Seldon • Additional components for storage, workflow

intensive cryptanalytic attacks ● A cryptoperiod is the time during which a key is used to encrypt data Key rotation: cryptoperiod There are lots of factors that influence the choice of cryptoperiod ○ Implementation ○ Operating environment ○ Volume of data ○ Re-keying method ○ Number of key copies ○ Personnel turnover ○ Threat model ○ New and disruptive technologies, e.g., quantum computers cardholder data against disclosure and misuse. 3.6 Fully document and implement all key-management processes and procedures for cryptographic keys used for encryption of cardholder data, including

缺乏交互、复用、可移植能 力。不同重复造轮子只是适 配不同 API 如何基于 K8s ，构建出一个既用户友好，又高可扩展，还 统一、标准化的应用管理平台？ 简单的“客户端”抽象： DCL (Data Configuration Language) 对 K8s 资源进行抽象实际上就是在操纵 YAML 数据，通过 DCL 来完成相比于 CRD + controller 更简单 CUE • 功能强大：专注于操纵数据，而不是写 PaaS 层 UI (e.g. dashboard, cli) 用户 CUE schema/模板 “客户端”抽象 标准化的“服务端”抽象 – 应用模型 Open Application Model (OAM) • 通过 OAM spec 定义“以应用为中心”的原语 • 打破“谷仓”! Common Traits Function Deployment K8s Operator Manual Scaler K8s Operators Kubernetes + OAM K8s Plugin HPA Deployment scale-to-0 Function Unified Model Layer Platform Capability Pool 统一的模型层 平台统一“能力池” 模块化的交付系统 - GitOps “应用”配置 Git (as source of truth)

capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: "/mnt/data" --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pv-claim spec: storageClassName: 发布了5个“example.com/foo”资源。 curl --header "Content-Type: application/json-patch+json" \ --request PATCH \ --data '[{"op": "add", "path": "/status/capacity/example.com~1foo", "value": "5"}]' \ http://k8s-master 另起⼀个终端，输⼊： curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/de fault/pods/cpu-demo/metrics/cpu/usage_rate 即可看到监控信息。 在本例中，尽管容器启动时，尝试使⽤2个CPU单位，但由

containerizing helps us to replicate our MVP launch strategy for different geographies. Cookie Cutter Model • Scaling based on business growth is very easy. Scalable • In the case of traffic spike, for instance Efficiency • Automatic service discovery allows engineers to not maintain any configuration for multi-data center deployments. Easy Configuration • Save 60% infrastructure cost per year per country. Cost-effective Sumit Gupta – Go-Jek International Expansion Team Willem Pienaar – Go-Jek Data Science Team Arief Hermansyah – Go-Jek Data Science Team Shani Pribadi – Go-Jek Business Intelligence Team Sourabh Gupta

Apache Beam TensorFlow Service Communication Management Container Orchestration Data Processing Pipelines Data Flow Graphs for Machine Intelligence Kubernetes Contributors opensource.google.com configuration) Orchestrate and manage on-prem containers just like GKE in the cloud Consistent operating model with access to GCP services across hybrid environments Single-pane-of-glass for multiple Kubernetes

On demand pricing • 降低服务运行成本：无需再为闲置的计算资源付费（No Cost when Idle） • 灵活选择容器资源规格（Fine-grained cost model） • 提高资源利用率 CPU (vCPU) Memory (GB) 1 Min. 2 and Max. 8GB, in 1GB increments 2 Min. 4 and • HTTP REST APIs and web applicated • Mobile backends • Continuous Integration Pipeline • Data Analytics • … Asynchronous Event Driven Stateless High dynamic Short Duration 14 -

Kubernetes支援Ceph、NFS、雲端儲存、NetApp、vSphere、 FlexVolume及PureStorage. • Red Hat Openshift支援Ceph/Rook、Red Hat OpenShift Data Foundation、 GlusterFS、NFS、Cinder及Flexvolume • SUSERancher支援GlusterFS、NFS、vSphere及Longhorn 12. 監控及作業管理 因素。 Canonical Kubernetes利用Juju協助企業導覽多雲佈建、安裝及設定的複雜度。 Juju Charmed Operators(以下簡稱「Charm」)利用模型導向作業(Model- Driven Operations)的概念，協助部署及管理Kubernetes，涵蓋各種不同的雲端供 應商及執行個體。Juju模型可讓低階儲存、運算、網路及軟體元件合理作為單一實 體，並於適當

Deployment Function 应用层 能力管理 用户体验层 Kubernetes Open Application Model（OAM） 一个用来构建云原生应用管理平台的标准规范与核心框架 OAM + OAM Platform UI Open Application Model Platform Kubernetes GitOps/持续集成 标准化定义应用组件 标准化配置应用运维能力 alibaba.com path: / service_port: 8001 # 2nd component - componentName: redis Open Application Model Platform 部署 应用配置 （Application Configuration） 面向应用维度配置运维能力与组件 apiVersion: core.oam.dev/v1alpha2