Stream Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 4/09: Flow control and load shedding ??? Vasiliki Kalavri | Boston University 2020 Keeping up with the producers queue: what if the queue grows larger than available memory? • block the producer (back-pressure, flow control) 2 ??? Vasiliki Kalavri | Boston University 2020 Load management approaches 3 ! Load shedder applications with strict latency constraints that can tolerate approximate results. Slow down the flow of data: • The system buffers excess data for later processing, once input rates stabilize.

State Upstream&C onnpool Pilot SOFARegistry Consul Nacos Consul Others Control Plane MCP Data flow Control flow xDS Discovery Client Metrics Admin REST API Request Request MOSN 做业务扩展 Discovery 1 2 4 1 2 Envoy MOSN Data flow Control flow GoLang L7 extension SDK GoLang L7 extension filter 5 Other http filter(via GoLang) CGO 1 2 xDS Control Plane Data Plane MoE 2 4 3 Request Others Cluster Manager/xDS Pilot SOFARegistry Antvip Nacos Antvip Others Control Plane MCP Data flow Control flow xDS Discovery Client Admin API REST API Request REST API Callback Callback

with NIST ● Author SP 800-204 series on microservice security ● R&D on Next Generation Access Control (NGAC) ● Exclusively co-host annual zero trust multi-cloud conference Best in Class Team ● Creators operating an application Why is Istio? TSB: The Application-Aware Networking Platform Istio: Control Plane Tetrate Service Bridge: Management Plane Envoy: Data Plane Workload (Service) POD Workload availability & resiliency enabling active-active deployments ● Cross cluster security policies & access control ● Unified telemetry and availability reporting ● Service discovery across multiple clusters ●

[REJECTREQUEST]system busy, start flow control for a while  too many requests and system thread pool busy, RejectedExecutionExc eption  [PC_SYNCHRONIZED]broker busy, start flow control for a while  [PCBUSY_CLEAN_QUEUE]broker [PCBUSY_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: %sms, size of queue: %d 二、原理解读 在进行消息中间件的选型时，如果待选中间件在功能上、性能上都能满足业务的情况下， 我各个建议把中间件的实现语言这个因素也考虑进去，毕竟选择一门用自己擅长的语言实现 的中间件会更具掌 offerFirst(byteBuffer); } 其调用栈如下： 从上面的分析看来，并不会随着消息的不断写入而导致内存溢出。 三、现象解答 1. [REJECTREQUEST]system busy, start flow control for a while 其抛出的源码入口点：NettyRemotingAbstract#processRequestCommand，上 面的原理分析部分已经详细介绍其实现原理，总结如下。

#IstioCon V0.2 Mesh Expansion ● Prerequisites ○ IP connectivity to the endpoints in the mesh ○ Istio control plane services (Pilot, Mixer, CA) accessible from the VMs ○ (optional) Kubernetes DNS server accessible Istio and manually register the services running #IstioCon V0.2 Mesh Expansion (cont.) ● Traffic flow (VM -> Container) 1. Dnsmasq accepts DNS queries 2. Access the built-in Kube DNS (exposed by ILB) intercepted by the sidecar proxy 5. xDS ■ Traffic forwarded to ingress in the mesh ● Traffic flow (Container -> VM) 1. Manual registration istioctl -n onprem register mysql 1.2.3.4 3306 #IstioCon

Security • Intellectual property stays in organization • Access Control 13 Enterprise Oriented Features • User management & access control • RBAC: admin, developer, guest • AD/LDAP integration • Policy y,z) • Concurrency ( asynchronousness ) • Shared the same address space • Non blocking for main flow • Ideal for background replication Channel for Communication between Threads • Syntax • No buffering: (logging) • Error handling • Arbitrary exit • Graceful retry • Auto recovery •Really messy in control flow State Machine Comes in to Rescue • Simple is beautiful! • A divide-and-conquer mindset to simplify

automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new row in the automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new row in the automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new row in the

application traffic end to end in production • Allow platform to use Istio authorization policy to control the access to each Knative service based on Istio service roles. How Istio is leveraged in a Knative overload issue still exits 800 Knative Services #IstioCon o 1400 total with dev release with flow control fix looks great, ingress_ready p100 < 30s o [Istio 1.9.x] Support for backpressure on XDS pushes configuration churn. This is disabled by default and can be enabled by setting the PILOT_ENABLE_FLOW_CONTROL environment variable in Istiod. o Final solution is envoy delta-XDS push in future Istio release

推荐的主机实 实践 践 1.1. 推荐的节点主机实践 1.2. 创建 KUBELETCONFIG CRD 来编辑 KUBELET 参数 1.3. 修改不可用 WORKER 节点的数量 1.4. CONTROL PLANE 节点大小 1.5. 推荐的 ETCD 实践 1.6. 将 ETCD 移动到不同的磁盘 1.7. 分离 ETCD 数据 1.8. OPENSHIFT CONTAINER PLATFORM 第 第 2 章 章 IBM Z 和 和 LINUXONE 环 环境的推荐主机 境的推荐主机实 实践 践 2.1. 管理 CPU 过量使用 2.2. 禁用透明巨页 2.3. 使用 RECEIVE FLOW STEERING（RFS）提高网络性能 2.4. 选择您的网络设置 2.5. 确保 Z/VM 上使用 HYPERPAV 的高磁盘性能 2.6. IBM Z 主机上的 RHEL KVM 建议 第 机器配置池： 2. 将 maxUnavailable 设置为您需要的值： 重要 重要 当设置该值时，请考虑无法使用的 worker 节点数量，而不影响在集群中运行的应 用程序。 1.4. CONTROL PLANE 节点大小 控制平面节点资源要求取决于集群中的节点和对象的数量和类型。以下控制平面节点大小是基于控制平面 密度测试的结果，或 Clusterdensity。此测试会在给定很多命名空间中创建以下对象：

App container Sidecar container All incoming traffic must flow through the sidecar first when entering the pod All outgoing traffic must flow through the sidecar before leaving the pod 12 What happens Shortcoming 1: Controlling the running order for containers Stabilizing Istio Kubernetes lacks good control APIs to customize the containers lifecycle in a pod. There is no official way to instruct a pod said 32 A full mesh is utopian, know what you need only Stabilizing Istio The reality: ● The control plane is burning down when pushing your thousand services updates to the hundreds of proxies running