Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several complexity Scalability Availability Observability Security Reliability Messaging Analytics Multi-tenancy caveats ● Single underlying infrastructure ● Reduce operational complexity ○ Infrastructure Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster DNS Aware NetworkPolicy Increased Istio security External Services TLS visibility Performance

Component Overview Terminology Networking Network Security eBPF Datapath Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Security Bugs Operations System Requirements Automatically run unit tests on code changes BPF and XDP Reference Guide BPF Architecture Instruction Set Helper Functions Maps Object Pinning Tail Calls BPF to BPF Calls JIT Hardening Offloads Toolchain these new BPF powers. Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs Automatically run unit tests on code changes BPF and XDP Reference Guide BPF Architecture Instruction Set Helper Functions Maps Object Pinning Tail Calls BPF to BPF Calls JIT Hardening Offloads Toolchain these new eBPF powers. Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency

Operations Istio Other Orchestrators Concepts Component Overview Terminology Address Management Multi Host Networking Security Datapath Failure Behavior Architecture Datapath Scale Kubernetes Integration Automatically run unit tests on code changes BPF and XDP Reference Guide BPF Architecture Instruction Set Helper Functions Maps Object Pinning Tail Calls BPF to BPF Calls JIT Hardening Offloads Toolchain This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs Automatically run unit tests on code changes BPF and XDP Reference Guide BPF Architecture Instruction Set Helper Functions Maps Object Pinning Tail Calls BPF to BPF Calls JIT Hardening Offloads Toolchain these new eBPF powers. Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency

Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs Automatically run unit tests on code changes BPF and XDP Reference Guide BPF Architecture Instruction Set Helper Functions Maps Object Pinning Tail Calls BPF to BPF Calls JIT Hardening Offloads Toolchain these new eBPF powers. Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency

Operations Istio Other Orchestrators Concepts Component Overview Terminology Address Management Multi Host Networking Security Datapath Failure Behavior Architecture Datapath Scale Kubernetes Integration This means that each host can allocate IPs without any coordination between hosts. The following multi node networking models are supported: Overlay: Encapsulation-based virtual network spanning all hosts or for environments which want to leverage the clustermesh functionality, a kvstore set up is required which can be set up using an Installation with external etcd or using the Installation with managed

kubeadm/#pod-network]. Standard Installation This guides takes you through the steps required to set up Cilium on Kubernetes using the cilium-etcd-operator. The cilium-etcd-operator replaces the requirement for the TLS cer�ficates between etcd peers to work correctly, a DNS reverse lookup on a pod IP must map back to pod name. If you are using CoreDNS, check the CoreDNS ConfigMap and validate that in-addr more details. Installation with external etcd This guide walks you through the steps required to set up Cilium on Kubernetes using an external etcd. Use of an external etcd provides be�er performance

and don’t want to pay for it ● TCP and UDP is enough Solution: ● Make task use specified IP by a set of BPF_PROG_TYPE_CGROUP_SOCK_ADDR and BPF_CGROUP_SOCK_OPS programs Move TCP/UDP servers to task BPF_CGROUP_SOCK_OPS programs → ● In proxy on accept(2) learn orig_dst by connection’s src IP and port from BPF map. ● Encrypt, see [0] for details on proxy itself. [0] https://atscaleconference.com/videos/scale- orig_dst.ip = ctx->user_ip6 ● orig_dst.port = ctx->user_port ● Save in a map ● ctx->user_ip6 = proxy.ip ● ctx->user_port = proxy.port BPF_SOCK_OPS_TCP_CONNECT_CB: ● src.ip

TCP packets with destination 10.0.0.10 # iptables -t raw -A OUTPUT -p tcp -d 10.0.0.10 -j MARK --set-mark 0xdeadbeef # ipft -m 0xdeadbeef • Network domain specific function call tracer • Trace “which have gone through CPU ID Time Stamp User defined tracing data (with Lua script) … Use case • Multi tenant HV networking using SRv6 + VRF • Contributed to find the bug in SRv6 GSO handling • Upstream investigation of SRv6 TSO/GSO issue (jp) • https://engineering.linecorp.com/ja/blog/tso-problems-srv6- based-multi-tenancy-environment/ • ipftrace source • https://github.com/YutaroHayakawa/ipftrace2 And more…