Sebastian Toader & Zsolt Varga 2021-Feb-26 Apache Kafka with Istio on K8s 2 • Scalability • Resiliency • Security • Observability • Disaster recovery Production grade Apache Kafka on Kubernetes certificate attached automatically by Istio Proxy sidecar container • Client certificate includes the K8s service account of the Kafka client application • SPIFE:///ns//sa/

0 码力 | 14 页 | 875.99 KB | 1 年前

3

External IPs #IstioCon V1.1 ServiceEntry #IstioCon V1.6-1.8 Better VM Workload Abstraction A K8s Service and Pods Two separate object with distinct lifecycles Before Workload Entry, a single Istio Machine Basic schedule unit Pod WorkloadEntry Component Deployment WorkloadGroup Service registry and discovery Service ServiceEntry K8s Pods labels: app: foo class: pod ServiceEntry selector: app: ■ based on a platform-specific identity ■ w/o a platform-specific identity ● using a short-lived K8s service account token ● Automatic certificate rotation ● Validation of the proxy’s status for VM-based

svce svce.ns svcd.ns svcd.ns Kube-proxy Kube-APIServer ServiceIp Backend Pod1 Labels:app=svcb Port:9379 Backend Pod2 Labels:app=svcb Port:9379 svca 基础设施(Kubernetes)看Istio: 能力增强 服务部署运 维 服务治理 Kube-APIServer Etcd istioctl / kubectl Pilot Envoy SVC Pod Node Envoy SVC Pod Node Envoy SVC Pod list/watch (Service, Endpoints, Pod) 用户 Istio & Kubernetes：统一服务发现 Pilot ServiceController( 灰度发布：蓝绿 灰度发布：A/B Testing 灰度发布：Canary releases 灰度发布：基于Kubernetes RC Version2 SVC SVC Pod1 Pod2 Pod3 SVC Pod1 Pod2 Version1(canary) 40% svcB svcA KubeAPIServer 60% Scheduler Controller- Managerr

svce svce.n s svcd.n s svcd.n s Kube-proxy Kube-APIServer ServiceIp Backend Pod1 Labels:app=svcb Port:9379 Backend Pod2 Labels:app=svcb Port:9379 svca8 基础设施(Kubernetes)看Istio: 能力增强 服务部署运 维 服务治理 Kube-APIServer Etcd istioctl / kubectl Pilot Envoy SVC Pod Node Envoy SVC Pod Node Envoy SVC Pod list/watch (Service, Endpoints, Pod) 用户13 Istio & Kubernetes：统一服务发现 Pilot ServiceController( 灰度发布：A/B Testing19 灰度发布：Canary releases20 灰度发布：基于Kubernetes RC Version2 SVC SVC Pod1 Pod2 Pod3 SVC Pod1 Pod2 Version1(canary) 40% svcB svcA KubeAPIServer 60% Scheduler Controller- Managerr21

Istio Pod App container Sidecar container All incoming traffic must flow through the sidecar first when entering the pod All outgoing traffic must flow through the sidecar before leaving the pod 12 not ready? Stabilizing Istio Pod App container Sidecar container (not running) The incoming traffic is sank into the void The outgoing traffic cannot leave the pod 13 What happens when the sidecar During pod creation ○ During pod deletion ● To prevent it, we need to make sure that: 1. Envoy is started before any other container in a pod 2. Envoy is stopped after any other container in a pod 14

FreeWheel的Istio实践 • 未来工作 • FreeWheel的痛点 Istio架构 • Istio Proxy: 劫持Pod的所有通信， 是Mesh的基础 • Pilot: 为Proxy提供动态配置管理 • Citadel: 自动维护mTLS密钥 • Mixer: 在k8s中部署了两组Mixer • Policy提供授权、Quota等能力 • Telemetry提供监控数据收集能力 基本原理 Proxy: Mesh的基础 • 网络安全：兼容Spiffe标准实现 • 配置管理：为C++实现的Proxy接 入k8s的动态配置管理 • Attribute Machine: 授权，Quota ，Tracing，监控的基础 Istio管理下的微服务 • 右图是部署mock1.v1 Pod之后发生的事 情： • Sidecar Injection: 注入initContainer, Sidecar Sidecar, istio-certs volume • Citadel: 自动刷新secrets, k8s自动加 载istio-secrets volume • Pilot: 和Sidecar建立连接，管理动态配 置 • Mixer: 和Sidecar建立连接，管理授权 、Quota和审计数据 • Istio的架构和基本原理 • FreeWheel的Istio实践 • 未来工作

platform has multiple shard k8s clusters, each cluster should support 1000 sequential (interval 5s) Knative service provisionings with route ready time <= 30s. Type Info K8s Cluster Capacity 12 nodes with Istio mesh/mTLS #IstioCon o Init-container added which cost ~5 seconds for Knative application pod code start. o Every sidecar needs full mesh information by default. Not a scalability solution. o push them to the sidecar. o Istio-proxy (envoy) sidecar costs ~2 seconds for Knative application pod cold start. Unleash maximum scalability by fully leveraging Istio features in Knative with service

Tetrate Service Bridge: Management Plane Envoy: Data Plane Workload (Service) POD Workload (Service) POD Workload (Service) POD Workload (Service) VM Workload (Service) VM Workload (Service) VM API Gateway Tetrate Service Bridge: Management Plane Envoy: Data Plane Workload (Service) POD Workload (Service) POD Workload (Service) POD Workload (Service) VM Workload (Service) VM Workload (Service) VM API Gateway DMZ zone Solving the OSS Istio pain with TSB: ● Managing multi cluster at scale ● They can’t have k8s cluster in the DMZ zone ● Simpler and better VM onboarding expereince ● Better zero trust architecture

服务身份认证、访问鉴权、通信加密 Proxy Application Layer Service 1 Istio 流量管理 – 概览 • 控制面下发流量规则： Pilot • 数据面标准协议：xDS • 集群内Pod流量出入： Sidecar Proxy • 集群外部流量入口：Ingress Gateway • 集群外部流量出口：Egress Gateway（可选,在一个集中点对外部访问进行控制） • Service v 通过CRD定义的服务数据 q 自定义流量规则（如何将请求路由到这些服务？） v 通过CRD定义的流量规则 服务数据 流量规则 3 Istio 流量管理 – 控制面 – 服务发现 • K8s Service ： Pilot 直接支持 • ServiceEntry： 手动添加 Service 到 Pilot 内部注册表中 • WorkloadEntry：单独添加 Workload，对于虚机支持更友好 调用的流量拦截及处理流程： 1. Productpage 发起对 reviews 服务的调用：http://reviews:9080/reviews/0 。 2. 请求被 productpage Pod 的 iptables 出向流量规则拦截，处理后重定向到本地 15001 端口。 3. Envoy 在 15001 端口上监听的 VirtualOutbound listener 收到了该请求。 4

现 Kubernetes Cluster MIxer 全链路关联 平台 Cloud Native App POD Agent logfile Proxy Transaction ID Transaction ID Cloud Native App POD Agent logfile Proxy Transaction ID Transaction ID …Commit Adapter作为grpc服务端，运行在独立的pod当中， Mixer通过通过rpc调用，将属性与日志发送给Adapter。基于Mixer的二次开发的流程 • 编写grpc服务端程序，接收来自mixer的数据，并实现自身业务逻辑 • 编写handler、instance、rule配置文件 • 编译打包adapter，上传至docker仓库 • 编写k8s的deployment和service配置文件