Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several complexity Scalability Availability Observability Security Reliability Messaging Analytics Multi-tenancy caveats ● Single underlying infrastructure ● Reduce operational complexity ○ Infrastructure Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster DNS Aware NetworkPolicy Increased Istio security External Services TLS visibility Performance

have gone through CPU ID Time Stamp User defined tracing data (with Lua script) … Use case • Multi tenant HV networking using SRv6 + VRF • Contributed to find the bug in SRv6 GSO handling • Upstream SRv6 TSO/GSO issue (jp) • https://engineering.linecorp.com/ja/blog/tso-problems-srv6- based-multi-tenancy-environment/ • ipftrace source • https://github.com/YutaroHayakawa/ipftrace2 And more… • SRv6

Platform ● App Platform: push-to-deploy PaaS offering by DigitalOcean ● Built on top of DOKS with multi-tenancy ○ various measures applied to guarantee isolation between tenants ● Makes extensive use of

Component Overview Terminology Networking Network Security eBPF Datapath Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Security Bugs Operations System Requirements This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts 1/1 Running 0 68s host-to-b-multi-node-clusterip-585db65b4d-x74nz 1/1 Running 0 68s host-to-b-multi-node-headless-77c64bc7d8-kgf8p 1/1 Running

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts allows to significantly reduce transmission tail latencies for applications and to avoid locking under multi-queue NICs compared to traditional approaches such as HTB (Hierarchy Token Bucket) or TBF (Token Bucket

Operations Istio Other Orchestrators Concepts Component Overview Terminology Address Management Multi Host Networking Security Datapath Failure Behavior Architecture Datapath Scale Kubernetes Integration This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts 1/1 Running 0 4m50s host-to-b-multi-node-clusterip-78f9869d75-l8cf8 1/1 Running 0 4m50s host-to-b-multi-node-headless-798949bd5f-vvfff 1/1 Running

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts allows to significantly reduce transmission tail latencies for applications and to avoid locking under multi-queue NICs compared to traditional approaches such as HTB (Hierarchy Token Bucket) or TBF (Token Bucket

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts allows to significantly reduce transmission tail latencies for applications and to avoid locking under multi-queue NICs compared to traditional approaches such as HTB (Hierarchy Token Bucket) or TBF (Token Bucket

Operations Istio Other Orchestrators Concepts Component Overview Terminology Address Management Multi Host Networking Security Datapath Failure Behavior Architecture Datapath Scale Kubernetes Integration This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts understanding required to run a full Cilium deployment. Component Overview Terminology Address Management Multi Host Networking Security Datapath Failure Behavior Component Overview A deployment of Cilium consists

on Arm64; • Multi-arch support for cilium-related images • CI/CD • Travis • Unit test Travis CI • Arm64 Full VM • arm64-graviton2 • Refer to: • https://docs.travis-ci.com/user/multi-cpu-architectures