Kubernetes VM Solutions for Multi-Tenant Applications Guangxu Li, Senior Software Engineer, ZTE li.guangxu@zte.com.cn Container and VM Ecosystem Kubernetes Docker Swarm Marathon Nomad Container

10-Annotation 11-K8s架构及基本概念 12-Master与Node的通信 13-Node 14-Pod 15-Replica Set 16-Deployment 17-StatefulSet 18-Daemon Set 19-配置最佳实践 20-管理容器的计算资源 21-Kubernetes资源分配 22-将Pod分配到Node 23-容忍与污点 24-Secret 1ad8，即：将 Calico降级到3.1.3。 参考⽂档： Kubespray – 10 Simple Steps for Installing a Production-Ready, Multi-Master HA Kubernetes Cluster：https://dzone.com/articles/kubespray-10-simple-steps-for-installing-a-product pace-name-here> get pods 设置Namespace⾸选项 可在上下⽂中永久保存所有后续 kubectl 命令的Namespace。 $ kubectl config set-context $(kubectl config current-context) --namespace= # Validate it

manages desired policy. ​Enforcement passes Pod -> container runtime -> Linux OS ​Cgroups are used to map Pod CPU and Memory Resources • Note: Two Cgroups Drivers exist (cgroupfs [default], systemd) 20 reservations, not just shares. It also incorporates health monitoring and IO awareness Secure multi-tenant (multi-department) Kubernetes deployments • with ability to have true guaranteed resource reservations

manages desired policy. Enforcement passes Pod -> container runtime -> Linux OS Cgroups are used to map Pod CPU and Memory Resources • Note: Two Cgroups Drivers exist (cgroupfs [default], systemd) 20 reservations, not just shares. It also incorporates health monitoring and IO awareness Secure multi-tenant (multi-department) Kubernetes deployments • with ability to have true guaranteed resource reservations

ppa:webupd8team/java RUN echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 boolean true" | debconf-set-selections RUN apt-get update && apt-get install -y oracle-java8-installer maven ADD . /usr/local/petclinic use a Maven plugin Download and install Docker Order of layers to optimize for cache hits Use of multi-stage builds github.com/GoogleContainerTools/jib What did we do? 1. Write first Dockerfile 2 use a Maven plugin Download and install Docker Order of layers to optimize for cache hits Use of multi-stage builds Understanding Docker cache mechanism and quirks github.com/GoogleContainerTools/jib

Container Service for Kubernetes 主机 容器在哪里运行 Amazon EC2 AWS Fargate 服务注册发现 云端服务的黄页 AWS Cloud Map 服务网格 服务间通信的基础设施层 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon • full blown distro (Ubuntu, AL) vs. minimal environment (container- optimized distribution) • multi-tenancy requirements • gotchas: Linux packages/CVEs, leaks, GDPR (in Europe) • runtime/standards

Performance measurement 03 04 Future work 05 06 Lessons from eBPF What is K8s Service • It exposes a set of pods via VIP using a load balancer • Two types • ClusterIP provides in-cluster access • ingress Tc egress Hit eBPF map? Does SNAT nic nic Y N • How IPVS talks with eBPF program? • eBPF map id is passed to IPVS module • Ip_vs_new_conn() inserts eBPF map • Key: (protocol, cip:cport Value: (protocol, lip:lport, rsip:rsport) • Ip_vs_conn_unlink() deletes entries in eBPF map How eBPF does SNAT eBPF map IPVS eBPF SNAT add del refer • V.S service in pure eBPF • Reuse the IPVS’s functionality

Replica Service Replica Service User Config Map Common Config Map Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Altinity ClickHouse operator – использование

网络存储等。 批量处理执行：除服务型应用，还支持批处理作业CI（持续集成），如有需要，一样可以实现容器故障后修复。 Kubernetes特点： 可移植: 支持公有云，私有云，混合云，多重云（multi-cloud） 可扩展: 模块化, 插件化, 可挂载, 可组合 自动化: 自动部署，自动重启，自动复制，自动伸缩/扩展 6 www.h3c.com Confidential 秘密 66 Kubernetes基础结构介绍 17 www.h3c.com Confidential 秘密 17 17 K8s基本概念和术语介绍（RC） RC Replication Controller（副本管理器）和RS（Replica Set）： RC定义了一个期望的场景，即声明某种Pod的副本数量在任意时刻都符合某个预期值。RC包含如下几部分： Pod期待的副本数（replicas） 用于筛选目标Pod的Label Selector 并大大减少了很多运维工作。 Replication Controller在k8s 1.2版本之后升级成了新的概念，Replica Set（下一代RC），Replicas Set支持基于集 合的标签选择器，而RC只支持基于等式的标签选择器。 Replicas Set的一些作用和特性： 1. 大多数情况下，我们通过定义一个RC实现Pod的创建过程及副本数量的自动控制 2. RC里面包含完整的Pod定义模板

尽量将大部分代码放在eBPF中，方便升级和维护。 • eBPF loader 创建eBPF map时，将map的id 传给IPVS内核模块 • 在ip_vs_new_conn 时，插入eBPF map • (protocol, clientip:cport -> targetip:dport) • 在ip_vs_conn_unlink时，删除eBPF map • 由于eBPF中没有timer机制 IPVS 如何做SNAT？ 和业界方法比较 V.S. 纯粹的eBPF service 和其他的优化方法对比 V.S. Taobao IPVS SNAT patch • 复用了IPVS timer来回收eBPF map。避开了eBPF map没有timer的问题 • 继承了IPVS丰富的功能，稳定性。例如调度算法丰富。 • 优势 • 完全绕过了conntrack/iptables • 对内核修改更小 04 性能测试