Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several engineers ● Reduce costs ● Security issues ● Scalability issues Namespaces +400 Pods +10k Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster

pods. kind Install kind >= v0.7.0 per kind documentation: Installation and Usage [https://kind.sigs.k8s.io/#installation-and-usage] curl -LO https://raw.githubusercontent.com/cilium/cilium/v1.10/Documentatio Cilium: Install Cilium into your newly created RKE cluster: cilium install k3s To install Cilium on k3s [https://rancher.com/docs/k3s/latest/en/quick-start/], perform the following steps: Default Configuration: Configuration: Datapath IPAM Datastore Encapsulation Cluster Pool Kubernetes CRD Requirements: Install your k3s cluster as you normally would but making sure to disable support for the default CNI plugin and the

Sandbox environment Getting Started Using Minikube Getting Started Using MicroK8s Getting Started Using K3s Getting Started Using Kind Getting Started Using Minikube This guide uses minikube [https://kubernetes load-balancing issues, then set [https://minikube.sigs.k8s.io/docs/commands/config/] any other driver from the supported list [https://minikube.sigs.k8s.io/docs/drivers/]. minikube start --cni=cilium --memory=4096 Cassandra Database Securing a Kafka cluster Getting Started Using K3s This guide walks you through installation of Cilium on K3s [https://k3s.io/], a highly available, certified Kubernetes distribution designed

Sandbox environment Getting Started Using Minikube Getting Started Using MicroK8s Getting Started Using K3s Getting Started Using Kind Getting Started Using Minikube This guide uses minikube [https://kubernetes Cassandra Database Securing a Kafka cluster Getting Started Using K3s This guide walks you through installation of Cilium on K3s [https://k3s.io/], a highly available, certified Kubernetes distribution designed Master Node The first step is to install a K3s master node making sure to disable support for the default CNI plugin: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --no-flannel'

pods. kind Install kind >= v0.7.0 per kind documentation: Installation and Usage [https://kind.sigs.k8s.io/#installation-and-usage] curl -LO https://raw.githubusercontent.com/cilium/cilium/v1.11/Documentatio into your newly created RKE cluster: cilium install --version 1.11.20 k3s To install Cilium on k3s [https://rancher.com/docs/k3s/latest/en/quick-start/], perform the following steps: Default Configuration: Configuration: Datapath IPAM Datastore Encapsulation Cluster Pool Kubernetes CRD Requirements: Install your k3s cluster as you normally would but making sure to disable support for the default CNI plugin and the

use the Cluster Mesh functionality. If you are running in an environment with more than 250 nodes, 5k pods, or if you observe a high overhead in state propagation caused by Kubernetes events. If you do deleted pod "kube-dns-autoscaler-76fcd5f658-22r72" deleted pod "kube-state-metrics-7d9774bbd5-n6m5k" deleted pod "l7-default-backend-6f8697844f-d2rq2" deleted pod "metrics-server-v0.3.1-54699c9cc8-7l5w2" 3, …) for HA. For simplicity, you can just use 1 region. The cluster NAME variable should end with k8s.local to use the gossip protocol. If creating multiple clusters using the same kops user, then make

Sandbox environment Getting Started Using Minikube Getting Started Using MicroK8s Getting Started Using K3s Getting Started Using Kind Getting Started Using Minikube This guide uses minikube [https://kubernetes Cassandra Database Securing a Kafka cluster Getting Started Using K3s This guide walks you through installation of Cilium on K3s [http://k3s.io], a highly available, certified Kubernetes distribution designed Master Node The first step is to install a K3s master node making sure to disable support for the default CNI plugin: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --no-flannel'

The DaemonSet will automa�cally install itself as Kubernetes CNI plugin. K8s 1.15 K8s 1.14 K8s 1.13 K8s 1.12 K8s 1.11 K8s 1.10 kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1 The DaemonSet will automa�cally install itself as Kubernetes CNI plugin. K8s 1.15 K8s 1.14 K8s 1.13 K8s 1.12 K8s 1.11 K8s 1.10 kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1 during the bootstrapping phase of Cilium. For Docker as container runtime: K8s 1.15 K8s 1.14 K8s 1.13 K8s 1.12 K8s 1.11 K8s 1.10 kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1

from source to deploy in one click, on any language or stack ● Every developer gets to their own k8s namespace About Okteto A few months ago... GCP was not thrilled Open Source to the rescue! ● automatically reload falco on rule changes ● Rules: monitor well known IPs, binary names, forbidden k8s actions ● Action: Notify to slack for human banhammer ● Use Ubuntu instead of ContainerOS Current

Balancer Service • Part of our private cloud service since 2017 • 5100 private, 760 public VIPs • k8s CCM integration (Type: LoadBalancer) L4LB Node L4LB Architecture XDP DPlane L3DSR with IPIP, Magrev Advertise VIP with eBGP Configure with RPC Health check daemon etc… Service Discovery Per-flow ECMP k8s CCM Frontend (dash board) To Backends User For More Information • Our motivation, detailed architecture