access • NodePort provides out-of-cluster access • Major modes • Iptables • IPVS Iptables mode • How it works • DNAT at PREROUTING chain • SNAT at POSTROUTING chain • Pros • Iptables is widely / O(N) in data plane • Poor in scheduling algorithm • Iptables rules are difficult to debug IPVS mode • Services are organized in hash table • IPVS DNAT • conntrack/iptables SNAT • Pros • O(1) time Post-route Iptables snat Conntrack Post-route Pre-route IPVS entry BPF SNAT IPVS mode data path IPVS-eBPF mode data path How eBPF does SNAT • Why does SNAT with eBPF • eBPF program is easy to deploy

应用通过固定的VIP访问一组pod，应用对Pod ip变化 无感知 • 本质是一个负载均衡器 • ClusterIP提供集群内的访问 • NodePort 提供集群外部的访问 iptables mode • 在netfilter pre-routing阶段做DNAT • 在netfilter post-routing阶段做SNAT • 每个service 添加一条或多条rules。使用数组管理rules。 在linux 已经广泛部署 优势 iptables mode 不足之处 • 控制平面的时间复杂度是O(N^2), 当service达到上千时，修改rule耗时超过半小时。 • 数据平面的时间复杂度是O(N) • 调度算法比较少，仅仅支持random的 • iptables rule 不容易调试 IPVS mode • 使用hashtable 管理service • iptables+conntrack 做SNAT • 控制面和数据面算法复杂度都是O(1) • 经历了二十多年的运行，比较稳定成熟 • 支持多种调度算法 优势 IPVS mode 不足之处 • 没有绕过conntrack，由此带来了性能开销 • 在k8s的实际使用中还有一些Bug 02 优化的方法 指导思路 • 用尽量少的cpu指令处理每一个报文 • 不能独占cpu

config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: app.kubernetes.io/managed-by: {{ .Release.Service }} data: config.ini: | [Common] Mode = product CacheMode = redis ReportStorageMode = redis AttrStorageMode = mongodb mongodb://mongodb-replicaset:27017 Direct = true Timeout = 3 PoolLimit = 10 Mode = monotonic Database = admin User = Password = [InfluxDB] Address

驱动提供者：⽂件NAS存储选择 cds/nas server-ip: 系统会⾃动读取NAS盘的挂载点IP path：NAS盘对应的远端挂在⽬录，默认为 /nfsshare mode: pv使⽤⽂件夹的mode，⼀般选择 755 或 777 存储类名称：配置该存储类的名称，当创建pvc时，需要引⽤该名称 存储卷回收策略：Retain，PVC被删除后，保留对应的PV；Delete，PVC被删除后，删除对应 存储驱动：请选择CSI（Flexvolume今后会停⽌⽀持） server-ip: 系统会⾃动读取NAS盘的挂载点IP path：NAS盘对应的远端挂在⽬录，默认为 /nfsshare mode: pv使⽤⽂件夹的mode，⼀般选择 755 或 777 存储卷名称：配置该存储类的名称，当创建pvc时，需要引⽤该名称 容量：所要创建存储卷的⼤⼩，单位GB 访问模式：ReadWirteMany：可以 替换为集群配置的存储NAS云盘ip地址 9 path: "/nfsshare/wordpress" # ⽬录可⾃⾏修改为 /nfssahre/ 10 vers: "4.0" 11 mode: "777" 12 reclaimPolicy: "Delete" 创建Namespace 创建命令 1 $ kubectl create -f wordpress-storageclass-pvc

edit cm kube-proxy -n kube-system #编辑configMap 找到kind: KubeProxyConfigura�on这行往下第2行的mode: ""，在""里写入ipvs， 保存退出 （要求在做准备工作时已加载ipvs内核模块） 再删除kube-proxy的所有pod（即重启这些pod） # kubectl get pod -n kube-system 51~53）使用haproxy做反向代理 frontend k8s_api_tcp_6443 bind *:6443 mode tcp default_backend my_k8s_cluster_6443 backend my_k8s_cluster_6443 mode tcp balance roundrobin server s1 10.99.1.51:6443 规则进行流量 的转发 ★kube-proxy的代理规则模式有： 代理模式 k8s版本要求 User Space proxy mode v1.0 + iptables proxy mode v1.1 + ipvs proxy mode v1.8 + 需要在所有k8s服务器上加载ipvs内核 模块 效率：ipvs > iptables > UserSpace

• TEE as an option • Solution • Get kubeconfig • Relay server mode • Non-relay server mode • Keep kubeconfig in memory • proxy mode • kubectl ó http/uds ó proxy ó https ó apiserver • X.509 or OIDC

or Mesos. l Spark 2.3 added native support for Kubernetes. l Spark 2.4 added support for client mode, R, python etc. l Spark 3.0 will add support for dynamic resource allocation, external shuffle service Solution: Pod delay creation spark-submit \ --master k8s://https://192.168.45.93:5443 \ --deploy-mode cluster \ --name query \ --class com.databricks.spark.sql.perf.BenchmarkQuery \ --conf spark.kubernetes

Listener HTTPS Listener Rule: /cheeses Rule: /charcuterie TargetGroup: Green (IP Mode) TargetGroup: Blue (Instance Mode) NodePort NodePort © 2019, Amazon Web Services, Inc. or its Affiliates. All rights

Practice — 王磊磊 @TenxCloud Agenda • Our DevOps Expectations • Kubernetes Capabilities/Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline / Stage/ Task availability • Extensibility / Integration • CI/CD examples • Future plan Our DevOps Expectations • Build a platform and easy to integrate with other DevOps/third-party tools • Easy to be customized as Kubernetes Capabilities/Advantages to Build DevOps Solution Pod Job CronJob • k8s itself is NOT a PaaS or DevOps platform，but … • k8s resources that can be used to build DevOps solution Volumes ConfigMap

github.com/GoogleContainerTools/jib Build containers faster with Jib A container image builder for Java applications Our Team Cloud Tools for Java Appu Goundan @coollog @loosebazooka Qingyang ndencies to target/dependencies/ Some more searching github.com/GoogleContainerTools/jib ... <build> com.spotify dockerfile-maven itory> ${project.version} build> ... github.com/GoogleContainerTools/jib What did we do? 1. Write first Dockerfile 2. Reduce