Sebastian Toader & Zsolt Varga 2021-Feb-26 Apache Kafka with Istio on K8s 2 • Scalability • Resiliency • Security • Observability • Disaster recovery Production grade Apache Kafka on Kubernetes certificate attached automatically by Istio Proxy sidecar container • Client certificate includes the K8s service account of the Kafka client application • SPIFE:///ns//sa/

0 码力 | 14 页 | 875.99 KB | 1 年前

3

Application Deployment: Cloud Layout ● Multiple K8s Clusters in an AZ ○ Each K8s cluster ~ 200 - 5,000 nodes ○ Upto 100,000 Pods in a cluster ○ 10,000+ K8s services - including prod, pre-prod, staging worst-case scenario Region R1 AZ 1 AZ 2 AZ n Data Center DC1 K8s Cluster K8s Cluster K8s Cluster K8s Cluster K8s Cluster K8s Cluster Region Rn #IstioCon Application Specs Region - Global IPAM, Access-control Policy store, etc. ● AZ Control Plane ○ Syncs specs to workload K8s clusters in the AZ ○ Shared-Nothing Architecture ■ Hosts services catering to the AZ, e.g., AZ

no additional cost) worked on the project in tight partnership with Google’s Istio subject matter experts. Scope NCC Group’s evaluation of Istio included: • Istio Architecture: The overall design and security: Istio allows a variety of customizations to fit it into different envi- ronments, but it’s difficult to say which is a hardened, production-ready approach. Having a secured profile with an opinionated account the severity of the risk, application’s exposure and user population, technical difficulty of exploitation, and other factors. For an explanation of NCC Group’s risk rating and finding categorization

• Pilot-Agent只是负责启动S，其他步骤由envoy完成。 • 1. 启动另外一个S进程（Secondary process） • 2. S通知P（Primary process）关闭其管理的端口，由S接管 • 3. S加载配置，开始绑定listen sockets，在这期间使用UDS从P获取合适的listen sockets • 4. S初始化成功，通知P停止监听新的链接并优雅关闭未完成的工作 关闭未完成的工作 • 5. 在P优雅关闭过程中，S会从共享内存中获取stats • 5. 到了时间S通知P自行关闭 • 6. S升级为P • 官方博客：Envoy hot restart什么时候会进行热重启？监控envoy ü获取非正常退出状态 ü抢救机制触发 ü抢救令牌减少一个（总共10个） ü在2(n-1) * 200毫秒后执行（为什么不立即执行） ü失败再次触发抢救机制 ü10 ü10个令牌用完，没有抢救成功，放弃退出优雅关闭envoy ü K8s发送SIGTERM信号让容器优雅关闭 ü Pilot-Agent接收信号通过context关闭子服务，发送SIGKILL关闭envoy ü Envoy不支持优雅关闭，需要通过金丝雀或蓝绿部署方式实现 Envoy优雅关闭实现方式讨论：#3307 #2920Pilot-Discovery——配置中心（PD） uv1版本和v2版本之间的区别

issues. 3. Review the fixes for the issues found in an audit from 2020. 4. Review and improve Istio's fuzzing suite. 5. Perform a SLSA review of Istio. The audit was started with a kickoff meeting, and team time to triage and assess criticality. Results summarised 6 fuzzers written and added to Istio's OSS-Fuzz integration 1 CVE found in Golang 1 vulnerability found that affected Googles managed Istio findings Issue 10 - “H2c handlers are uncapped” - was an interesting finding, in that it affected Googleʼs managed Istio offering, and it led to further investigation that revealed a vulnerability in Golang

platform has multiple shard k8s clusters, each cluster should support 1000 sequential (interval 5s) Knative service provisionings with route ready time <= 30s. Type Info K8s Cluster Capacity 12 nodes in thinks the configuration works. o [Istio 1.5.4] Istio is picking up new VirtualService slowly 30s #IstioCon Istio scalability optimization during Knative Service provisioning [Istio 1.6.5&1.7.0] 0ms and PILOT_DEBOUNCE_MAX=10s are the env vars on pilot that can be tuned. o Set PILOT_DEBOUNCE_AFTER=1s helps under our workload. (we tested with 100ms, 1s, 2s, 5s, 10s) o With 800 Knative Services

onprem register mysql 1.2.3.4 3306 #IstioCon V1.1 Introducing Service Entry Service Entry v.s. Service v.s. Endpoints ● Service Entry ○ An entry that Istio maintains internally ○ Describing the properties External IPs #IstioCon V1.1 ServiceEntry #IstioCon V1.6-1.8 Better VM Workload Abstraction A K8s Service and Pods Two separate object with distinct lifecycles Before Workload Entry, a single Istio WorkloadEntry Component Deployment WorkloadGroup Service registry and discovery Service ServiceEntry K8s Pods labels: app: foo class: pod ServiceEntry selector: app: foo Istio Workload Entries labels:

主线程 初始化 日 志 线 程 读 取 配 置 x D S 监 听 网络事件 启 动 工 作 线 程 定时器事件 a d m i n 请 求 X D S 更 新 合 并 s t a t 刷 新 D N S 调度器 工作线程 网络事件 定时器事件 监 听 器 监 听 过 滤 器 释 放 内 存 记 录 s t a t 状 态 更 新 调度器 L 4 网 络 过 滤 Technologies Co., Ltd. All rights reserved. Page 20 生产环境问题分析及解决方法（1） 503 UF问题分析 现象 日志报错503 UF，等待8S后建立连接失败。 日志如下： [2021-02-09T06:29:10.489Z] "GET /v1/xx/xx/xx/xx HTTP/1.1" 503 UF "-" "-" 0 91 288 - "100 接数增加快速恶化 端到端平均时延 降低23%左右 • Envoy: 4线程，4core，默认内存 • fortio –q 0 –c 2~1024连接，http1长 连接模式，每组测试三次，每次30s 测试结果 测试条件 Copyright © Huawei Technologies Co., Ltd. All rights reserved. Page 24 针对Envoy做的一些优化及效果

Deployment POD POD POD POD S E R V I C E (Load balancer) www.my-application.com External Traffic 75% 25% Deployment Canary Releases Using Kubernetes Deployment POD POD POD S E R V I C E (Load balancer) com External Traffic POD 50% 50% Deployment Canary Releases Using Kubernetes Deployment POD S E R V I C E (Load balancer) www.my-application.com External Traffic POD POD 0% 100% Deployment – Across application Layers Deployment POD POD S E R V I C E (Load balancer) www.my-application.com External Traffic 75% 25% POD POD POD POD S E R V I C E (ClusterIP) 75% 25% POD POD Cross-version

Prevention Certificate Authority K8s Network Policy K8s RBAC Audit Logging Image Verification Admission Control Workload Identity K8s RBAC K8s CNI AuthZ Policy Peer AuthN Policy access control Service 2 Service 1 1. Ensure traffic is natively encrypted, such as HTTPS 3. use k8s network policies to limit traffic bypassing sidecars Cluster security best practices: safely handle authz policies Namespace bar 2. Enforce k8s RBAC policies: roles bound to namespace, only mesh admins are allowed to have ClusterRole. 1. Use k8s network policies to limit the traffic in & out