support eBay scale ■ Proxy config convergence time (CDS, EDS, LDS, RDS push times) ■ Resource usage (CPU, memory, etc.) ○ Secondary Goal ■ Fine-tune configuration params - debounce interval, push concurrency Testing: Results ● Default wide-open egress sidecar configuration does not scale ○ Results in high memory usage & convergence times since each sidecar knows about all services in the cluster ○ Disabled egress from single Pilot instance to 0 - 3,000 sidecars < 1 second ○ Pilot CPU & memory within acceptable limits: < 10 cores, 25 GB memory ○ Pilot can scale horizontally ● Need to tune PILOT_DEBOUNCE_AFTER, PILOT_DEBOUNCE_MAX

Mercari? ● Service start: July 2013 ● OS: Android, iOS *Can also be accessed by web browsers ● Usage fee: Free *Commission fee for sold items: 10% of the sales price ● Regions/languages supported: the HPA calculation 22 Define HPA target for multi-containers pods Stabilizing Istio CPU: 1 Memory: 100MB Pod App container Container requests 23 Define HPA target for multi-containers pods the Istio adoption… The other big problem is estimating what is the Istio sidecar container CPU usage, which we’ll talk about in the second part of the presentation. 28 Are you prepared to handle Istio

which is: “The first request on an h2c connection is read entirely into memory before the Handler is called. To limit the memory consumed by this request, wrap the result of NewHandler in an http.MaxBytesHandler project from memory-unsafe implementation issues such as buffer overflow and use-a�er-free issues. Envoy - which plays a core role in the Istio service mesh - is implemented in C++ and memory-corruption out-of-bounds, race conditions, resource exhaustion issues and other issues stemming from improper usage of the language. Istio consists of two components: The controlplane and the dataplane. The data plane

for inline components & workflows ○ Trust model augmentation ■ Impersonating ■ Secret clear in memory ■ Secret persistence ● Key protection ○ Private key for TLS ○ Signing key ○ … #IstioCon Performance SW-HW co-designs #IstioCon Latency Analysis ● ~3ms P90 latency added ○ Istio v1.6 ○ More for VM usage ● Hotspots ○ 1  2 ○ 3  4: 30%~50% ● Others ○ Latency between Pods ○ Latency introduced by ● CapEx, OpEx #IstioCon RDMA (Remote Direct Memory Access) ● Advance transport protocol (same layer as TCP and UDP) ● Main features ○ Remote memory r/w semantics in addition to send/receive ○ Kernel

Ingress/Egress: Networking controls allowing inbound and outbound access of Istio services. • Istio Envoy Usage: The configuration and implemen- tation of Envoy within Istio (NOTE: Envoy itself was not part of /main.go Impact Trace profiling risks providing attackers with information about the processes, memory, and potentially sensitive information about Istio. An attacker with network access to the control operator - server imagePullPolicy: IfNotPresent resources: limits: cpu: 200m memory: 256Mi requests: cpu: 50m memory: 128Mi env: - name: WATCH_NAMESPACE value: istio-system - name: LEADER_ELECTION_NAMESPACE

Derive different types of tests Mocks for External APIs Istio enables learning tests from API usage Learnt by Mesh API Studio Third-party apps Manual QA trace: r trace: r trace: r trace: Service Testing by Devs Component, E2E Tests Service Tests Learning from usage of application and services Dev Usage Staging/UAT Env API catalog | CONFIDENTIAL #Rollbacks MTTR #Bugs

Release and Upgrade Notes ○ Release date slip ○ Release with known issues ○ Performance and resource usage ● Istio community didn’t have a process #IstioCon Led To ● Upgrade Working Group ● Release Note Where to post announcements ● What to look for when examining releases ○ Performance ○ Resource usage ○ Open issues ○ Features being promoted ○ Release notes and upgrade notes #IstioCon Continuous

0+ resolved this issue. o Istiod MEM bumped with large numbers of Knative Services (#25532) Mem usage optimization of pilot resolved this issue. • Tune CPU/MEM to ensure enough capacity Leveraged Metrics

• Accesslog：格式 https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage • 调试日志：pilot-agent request POST /logging?connection=trace #Cxxx • 抓包 • 进入pod容器网络空间执行 tcpdump -i