Kubernetes VM Solutions for Multi-Tenant Applications Guangxu Li, Senior Software Engineer, ZTE li.guangxu@zte.com.cn Container and VM Ecosystem Kubernetes Docker Swarm Marathon Nomad Container

full blown distro (Ubuntu, AL) vs. minimal environment (container- optimized distribution) • multi-tenancy requirements • gotchas: Linux packages/CVEs, leaks, GDPR (in Europe) • runtime/standards (OCI)

services. • Support Service Mesh. • Large-scale and high-performance autoscaling capabilities. • Multi-tenant and quota management. • etc. TKEx Architecture EKS (Elastic Kubernetes Service) TKE (Tencent StatefulSetPlus Batch Gray Release Key Features: Ø Manual/Auto Batch Gray Release Ø Multi-Batch Rollback Ø Multi-Batch InPlaceUpdate Ø Support HPA, CronHPA, VWA (Vertical Workload Autoscaler) Ø Keep Kube-Api VPA InPlace Offline Biz Quota (30%) Online Biz Quota (70%) Cluster Resource Pool Ø Multi-Level quota mechanism. Ø Dynamic adjustment of offline jobs quota. Ø Offline job queue manager

use a Maven plugin Download and install Docker Order of layers to optimize for cache hits Use of multi-stage builds github.com/GoogleContainerTools/jib What did we do? 1. Write first Dockerfile 2 use a Maven plugin Download and install Docker Order of layers to optimize for cache hits Use of multi-stage builds Understanding Docker cache mechanism and quirks github.com/GoogleContainerTools/jib use a Maven plugin Download and install Docker Order of layers to optimize for cache hits Use of multi-stage builds Understanding Docker cache mechanism and quirks Have elevated privileges to run Docker

1. Third-party plugins or integrations 集成第三方插件 2. Feature parity with kubectl 功能与kubectl保持一致 3. Multi-cluster management 多集群管理 4. Improved security 提高安全性 Top requested changes 1. Third-party plugins Cluster Operator, running Kubernetes on-prem and in the cloud 2. Feature parity with kubectl 3. Multi-cluster management 多集群管理 Most survey takers said that it is very or extremely important to see

and businesses How to ensure load balancing of cluster nodes 1 2 Improper resource requests 3 Multi-tenant resource preemption How to expand horizontally more quickly and flexibly 4 Region1 How Node Allocatable Resource Allocated Resource Real Resource Load Not Oversold How to solve the multi-tenant resource preemption problem DynamicQuota How to prevent resource preemption between different

• 主负责写和事件分发 • 从负责读 • 底层对接分布式强一致性存储 • CAS 事务写 • 快照读 实现架构图 存储层 存储层 – 分布式 KV Store ByteKV • Multi Raft Goup • 全局有序 Range 分区 • 强一致性 • 支持多 key 事务 • 支持 CAS • 支持快照读 • 高性能 存储层 - 数据格式 etcd KubeBrain 设计思路 • 性能优化 • 落地效果 • 未来演进 性能优化 写优化 - 1 降低锁粒度 存储引擎替换 表锁 -> 行锁，增大了写的并发 写优化 - 2 单点写 -> 多点写 multi raft range 分片，增大写并发 Brain 层无磁盘 io，只有网络 io 写优化 - 3 事务优化 精心设计 key 格式 一个 k8s 对象的索引和数据在同一分区内 跨分区分布式事务

reservations, not just shares. It also incorporates health monitoring and IO awareness Secure multi-tenant (multi-department) Kubernetes deployments • with ability to have true guaranteed resource reservations

reservations, not just shares. It also incorporates health monitoring and IO awareness Secure multi-tenant (multi-department) Kubernetes deployments • with ability to have true guaranteed resource reservations

Efficiency • Automatic service discovery allows engineers to not maintain any configuration for multi-data center deployments. Easy Configuration • Save 60% infrastructure cost per year per country