Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several complexity Scalability Availability Observability Security Reliability Messaging Analytics Multi-tenancy caveats ● Single underlying infrastructure ● Reduce operational complexity ○ Infrastructure Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster DNS Aware NetworkPolicy Increased Istio security External Services TLS visibility Performance

have gone through CPU ID Time Stamp User defined tracing data (with Lua script) … Use case • Multi tenant HV networking using SRv6 + VRF • Contributed to find the bug in SRv6 GSO handling • Upstream SRv6 TSO/GSO issue (jp) • https://engineering.linecorp.com/ja/blog/tso-problems-srv6- based-multi-tenancy-environment/ • ipftrace source • https://github.com/YutaroHayakawa/ipftrace2 And more… • SRv6

Platform ● App Platform: push-to-deploy PaaS offering by DigitalOcean ● Built on top of DOKS with multi-tenancy ○ various measures applied to guarantee isolation between tenants ● Makes extensive use of

stomization-with-docker Container customization with Docker 3 rock-images/multi-node-configuration-with-docker-compose Multi-node configuration with Docker-Compose 2 Software 3 package-management Package s390x IBM LPAR autoinstall on s390x ROCK Images Introduction Container customization with Docker Multi-node configuration with Docker-Compose Software Package management Upgrade Third party APT repositories /dev/loop3: Can't open blockdev ¬ 595.610434| systemd¬1|: multi-user.target: Job getty.target/start deleted to break ordering cycle starting with multi-user.target/start ¬ ¬0;1;31m SKIP ¬0m| Ordering cycle

Component Overview Terminology Networking Network Security eBPF Datapath Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Security Bugs Operations System Requirements This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts 1/1 Running 0 68s host-to-b-multi-node-clusterip-585db65b4d-x74nz 1/1 Running 0 68s host-to-b-multi-node-headless-77c64bc7d8-kgf8p 1/1 Running

Server DPUDirect HPCRunner Efficient concurrency and ultimate performance A-Tune etMem Gazelle Multi-kernel architecture Diversified computing power Cloud-native/Edge KubeOS Rubik Unified compose Simplified O&M and development A-Ops SysCare CPDS CPU GPU Optimal performance for a single scenario Multi-scenario capability collaboration and sharing Fundamental innovation Lego-style architecture addresses to establish multiple TCP/RDMA links for different IP addresses, achieving functions such as multi-path link setup, fault recovery and failover, and load balancing. eNFS: Building an E2E High-Performance

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts allows to significantly reduce transmission tail latencies for applications and to avoid locking under multi-queue NICs compared to traditional approaches such as HTB (Hierarchy Token Bucket) or TBF (Token Bucket

Operations Istio Other Orchestrators Concepts Component Overview Terminology Address Management Multi Host Networking Security Datapath Failure Behavior Architecture Datapath Scale Kubernetes Integration This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts 1/1 Running 0 4m50s host-to-b-multi-node-clusterip-78f9869d75-l8cf8 1/1 Running 0 4m50s host-to-b-multi-node-headless-798949bd5f-vvfff 1/1 Running

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts allows to significantly reduce transmission tail latencies for applications and to avoid locking under multi-queue NICs compared to traditional approaches such as HTB (Hierarchy Token Bucket) or TBF (Token Bucket

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts allows to significantly reduce transmission tail latencies for applications and to avoid locking under multi-queue NICs compared to traditional approaches such as HTB (Hierarchy Token Bucket) or TBF (Token Bucket