Accelerate Istio-CNI with ebpf Xu Yizhou & Guo Ruijing #IstioCon Agenda ● Istio-CNI ● tcp/ip stack overhead between sidecar and service ● Background knowledge of ebpf ● Acceleration for Inbound/Outbound/Envoy functionality provided by the istio-init container. #IstioCon Tcp/ip stack overhead between sidecar and service Overhead sidecar traffic from 3 scopes ● Inbound ● Outbound ● Envoy to Envoy(same

● TCP/IP stack overhead in service mesh ● Background knowledge of eBPF ● Independent solution to bypass TCP/IP stack ● Performance Comparision Istio Meetup China TCP/IP stack overhead ● All the application

clusters, including enabling alpha features and multicluster - Local resource utilization - Some overhead of Kubernetes and docker images - Attaching a debugger is not trivial #IstioCon Fully Local

Current Istio functionality meets user needs ○ Measured feature introduction ● Reducing operational overhead ○ Maintenance ○ Upgrades ○ Debugging https://istio.io/latest/blog/2020/tradewinds-2020/ #IstioCon

multi-containers pods Stabilizing Istio CPU: 1 Memory: 100MB Pod App container Container requests 23 Define HPA target for multi-containers pods Stabilizing Istio CPU: 1 Pod App container Container Container requests HPA configuration (70% CPU) metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 Will trigger when the container more than 700m CPU 24 Define HPA target for multi-containers pods Stabilizing Istio CPU: 1 Pod App container Sidecar container CPU: 100m Container requests HPA configuration (70% CPU) metrics:

pilot resolved this issue. • Tune CPU/MEM to ensure enough capacity Leveraged Metrics to monitor Istio & Knative components’ CPU and MEM under workload to avoid CPU throttling and OOM and ensure enough Istio 1.5.4: Istio scalability optimization during Knative Service provisioning Project Component CPU MEM HorizontalPodAutoscaler (HPA) request limit request limit Istio (1.7.3) istio- ingressgateway accelerate the whole debug and fix process: https://github.com/knative-sandbox/kperf ● Get Istio CPU/MEM stats: https://github.com/istio/istio/wiki/Analyzing-Istio- Performance ● Debugging Envoy and

L4层网络读取及数据发送为全异步读写模式，采用网络事件触发机制完成响应数据的接收和发送。 • 由于Router部分请求处理方向需要进行更多路由选择计算及负载均衡计算工作，因此通常outbound方向处理较复杂，CPU消耗比inbound更高。 Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 20 生产环境问题分析及解决方法（1） listener3 backend listener3 dst conn1 解析发送 fd2 默认选链接策略：接收线程即为后续连接数据处理线程， 导致连接分配完全凭运气，无法有效发挥所有worker CPU处理 能力。 默认连接处理策略 • virtualoutbound监听器监听在相同的监听端口， 由内核随机挑选一个线程用于处理新连接。 • 当Envoy配置的线程数比较多时（越多越明显）， 常观察到新连接被分配到某些线程。 script -i perf.data > out.perf; stackcollapse-perf.pl out.perf > out.folded; flamegraph.pl out.folded > cpu.svg • 镜像修改 • 编译pilot-agent, envoy二进制后替换现有envoy镜像并配置到自定义deployment的image中， • Dockerfile: • From

support eBay scale ■ Proxy config convergence time (CDS, EDS, LDS, RDS push times) ■ Resource usage (CPU, memory, etc.) ○ Secondary Goal ■ Fine-tune configuration params - debounce interval, push concurrency Main Takeaways ○ P99.9 time from single Pilot instance to 0 - 3,000 sidecars < 1 second ○ Pilot CPU & memory within acceptable limits: < 10 cores, 25 GB memory ○ Pilot can scale horizontally ● Need

command: - operator - server imagePullPolicy: IfNotPresent resources: limits: cpu: 200m memory: 256Mi requests: cpu: 50m memory: 128Mi env: - name: WATCH_NAMESPACE value: istio-system - name: L

的包括日志、网络数据在内的所 有信息。 宏观下的监控需求 链路总体展示 展示整个服务调用过程中链路上 每一个节点的服务状况，包括延 时、吞吐量等基本信息。 服务器总体展示 展示当前所有服务器的运行状况， 包括CPU、内存、网络、I/O读写 等信息 业务总体展示 展示当前业务相关数据的 从宏观上快速定位问题，在微观上找到问题根因的 监控方案问题二：现有的系统能否完全满足需求 现有系统如何满足运维需求Istio现有的监控体系