nais.yaml cluster kubectl apply -f nais.yaml application deployment service virtualservice autoscaler networkpolicy servicerole servicerolebinding serviceentry apiVersion: "nais.io/v1alpha1" kind: accessPolicy: inbound: - name: consumer-a deployment service autoscaler deployment service virtualservice autoscaler networkpolicy servicerole servicerolebinding serviceentry apiVersion: nav.local secrets: true accessPolicy: inbound: - name: consumer-a init-container pod apiVersion: "nais.io/v1alpha1" kind: "Application" metadata: name: app labels: team: pension

Istio Pod App container Sidecar container All incoming traffic must flow through the sidecar first when entering the pod All outgoing traffic must flow through the sidecar before leaving the pod 12 not ready? Stabilizing Istio Pod App container Sidecar container (not running) The incoming traffic is sank into the void The outgoing traffic cannot leave the pod 13 What happens when the sidecar During pod creation ○ During pod deletion ● To prevent it, we need to make sure that: 1. Envoy is started before any other container in a pod 2. Envoy is stopped after any other container in a pod 14

新、目标健康检查、 完整的可观测性等。 • 目前常见数据面主要有三种：Envoy、Linkerd、Traefic。Envoy由于高性能和扩展能力前在数据面遥 遥领先。 • Iptables使Pod间出入应用的流量均由Envoy代理，对应用来说完全透明。支持主要常用网路协议 Http1/Http2/Tls/gRPC/Tcp等。 Copyright © Huawei Technologies backend:8123 127.0.0.1:8123 zipkin Pod1 Pod2 业务容器 业务容器 Istio-proxy容器 Istio-proxy容器 Istio-init 容器 Istio-init 容器 Pod内共享网络 Pod内共享网络 Virtual inbound -15006 kubelet 拦截指定命名空间 Pod创建请求 xDS Iptables 规则 ./etc/istio/proxy/SDS /etc/istio/proxy/SDS 证书 获取 配置 文件 可以修改全局注入参数 作用于所有目标空间的 pod 证书更新 Envoy启动流程 Envoy控制面流量 Envoy数据面流量 ./etc/istio/proxy/XDS SDS xDS CSR Prometheus configmaps Copyright © Huawei Technologies Co., Ltd

Kubernetes Deployment POD POD POD POD S E R V I C E (Load balancer) www.my-application.com External Traffic 75% 25% Deployment Canary Releases Using Kubernetes Deployment POD POD POD S E R V I C E (Load com External Traffic POD 50% 50% Deployment Canary Releases Using Kubernetes Deployment POD S E R V I C E (Load balancer) www.my-application.com External Traffic POD POD 0% 100% Deployment Deployment application Layers Deployment POD POD S E R V I C E (Load balancer) www.my-application.com External Traffic 75% 25% POD POD POD POD S E R V I C E (ClusterIP) 75% 25% POD POD Cross-version Traffic

svce svce.ns svcd.ns svcd.ns Kube-proxy Kube-APIServer ServiceIp Backend Pod1 Labels:app=svcb Port:9379 Backend Pod2 Labels:app=svcb Port:9379 svca 基础设施(Kubernetes)看Istio: 能力增强 服务部署运 维 服务治理 Kube-APIServer Etcd istioctl / kubectl Pilot Envoy SVC Pod Node Envoy SVC Pod Node Envoy SVC Pod list/watch (Service, Endpoints, Pod) 用户 Istio & Kubernetes：统一服务发现 Pilot ServiceController( 灰度发布：蓝绿 灰度发布：A/B Testing 灰度发布：Canary releases 灰度发布：基于Kubernetes RC Version2 SVC SVC Pod1 Pod2 Pod3 SVC Pod1 Pod2 Version1(canary) 40% svcB svcA KubeAPIServer 60% Scheduler Controller- Managerr

svce svce.n s svcd.n s svcd.n s Kube-proxy Kube-APIServer ServiceIp Backend Pod1 Labels:app=svcb Port:9379 Backend Pod2 Labels:app=svcb Port:9379 svca8 基础设施(Kubernetes)看Istio: 能力增强 服务部署运 维 服务治理 Kube-APIServer Etcd istioctl / kubectl Pilot Envoy SVC Pod Node Envoy SVC Pod Node Envoy SVC Pod list/watch (Service, Endpoints, Pod) 用户13 Istio & Kubernetes：统一服务发现 Pilot ServiceController( 灰度发布：A/B Testing19 灰度发布：Canary releases20 灰度发布：基于Kubernetes RC Version2 SVC SVC Pod1 Pod2 Pod3 SVC Pod1 Pod2 Version1(canary) 40% svcB svcA KubeAPIServer 60% Scheduler Controller- Managerr21

cluster svcA svcB envoy envoy Pod1：10.244.0.20 Pod2：10.244.0.25 Dest: 127.0.0.1 Src：127.0.0.1 #IstioCon Istio Traffic Flow - ingress svcB envoy envoy Pod1：10.244.0.19 Pod2：10.244.0.25 Dest: 127.0.0 Address Preserve #IstioCon Preserve TCP Original Src Addr - inner svcA svcB envoy envoy Pod1：10.244.0.20 Pod2：10.244.0.25 Dest: 127.0.0.1 Src：10.244.0.20 ① Setting annotation sidecar.istio.io/interceptionMode: 0/route_localnet #IstioCon Preserve TCP Original Src Addr - ingress svcB envoy envoy Pod1：10.244.0.19 Pod2：10.244.0.25 Dest: 127.0.0.1 Src：100.10.10.10 Ingress gateway ELB ingress EIP: 192.168

to be a way to restrict a Pod’s access to them. Attempts to modify the settings to “controlPlaneAuth Policy: MUTUAL_TLS” did not appear to have any effect on preventing a Pod not managed by Istio from --set values.global.controlPlaneSecurityEnabled=true • Deploy the customized default policy • Start a Pod in a namespace that is not managed by Istio 1https://istio.io/latest/news/releases/1.5.x/announcing-1 used repeatedly. Reproduction Steps With the reference cluster setup and replacing PODNAME with a Pod that has curl installed, the following displays the debug interface for Pilot kubectl exec -it PODNAME

manage iptable IPTables: Responsible for translating service IP addresses (which are static) into Pod IP addresses CNI plugins: allocate ip addresses for workloads exist in nodes CNI interface Calico routing rules Networking lifecycle (Istio CNI) Kubelet Start a pausing pod Kubelet invoke CNI plugins CNI plugins setup ip for pod Istio CNI install isidecar network routing rule to workload iptable Benefits （need validation instead) Issue in Istio CNI Kubelet Start a pausing pod Kubelet invoke CNI plugins CNI plugins setup ip for pod Pod could get started in here and bypassing istio sidecar proxy(race condition)

with Istio mesh/mTLS #IstioCon o Init-container added which cost ~5 seconds for Knative application pod code start. o Every sidecar needs full mesh information by default. Not a scalability solution. o push them to the sidecar. o Istio-proxy (envoy) sidecar costs ~2 seconds for Knative application pod cold start. Unleash maximum scalability by fully leveraging Istio features in Knative with service daemonset pod of istio CNI plugin is up and running before knative pods scheduling on the node. o Crontab job could help to detect whether pod was configured correctly and restart pod Unleash maximum