semantics, and guarantees Vasiliki Kalavri | Boston University 2020 Today’s topics • High-availability and fault-tolerance in distributed stream processing • Recovery semantics and guarantees • Exactly-once of the primary and O’ the output produced by the secondary after recovery. • Precise recovery guarantees Of + O’ = Oe • Rollback recovery allows duplicate tuples downstream: • repeating: duplicate Kalavri | Boston University 2020 Processing guarantees and result semantics 11 sum 4 3 2 1 0 … Vasiliki Kalavri | Boston University 2020 Processing guarantees and result semantics 11 sum 4 3 2 1 …

and design Scheduling and load management Scalability and elasticity Fault-tolerance and guarantees State management Operator semantics Window optimizations Filtering, counting, sampling stream processing vs other technology • be able to comprehensively compare features and processing guarantees of streaming systems • be proficient in using Apache Flink and Kafka to build end-to-end, scalable | Boston University 2020 Some hard problems in stream processing 32 Time Order Processing guarantees Progress Retractions & results amendment Reconfiguration & updates Debugging Fault-tolerance

ins_u(P)) else ins_u(P). Insert-Replace: If the stream has a key, the reconstitution function ins_r guarantees that only the most recent item with a given key is included: • ins_r([]) = Ø • ins_r(P:i) = results are typically exact • Challenges: computation progress, fault-tolerance and result guarantees, automatic scaling and state migration, out-of-order processing 37 Vasiliki Kalavri | Boston

manage stateful applications. Manages the deployment and scaling of a set of pods, and provides guarantees about the ordering and uniqueness of these Pods • A DaemonSet ensures that all (or some)

stored on the queue until they are processed and deleted • transactional, timing, and ordering guarantees • Each message is processed only once, by a single consumer • Event retrieval is not defined

will be broadcast to all the consumer processes. Vasiliki Kalavri | Boston University 2020 25 Guarantees • Messages sent by a producer to a particular topic partition will be appended in the order they

Kafka ● Apache Kafkaに入門した 原文链接：Apache RocketMQ 介绍 ● 解决KafKa数据存储与顺序一致性保证 ● Kafka : Ordering Guarantees ● Apache Kafka 从 0.7 到 1.0：那些年我们踩过的坑 ● Apache Kafkaの概要とアーキテクチャ ● Apache RocketMQ ● 专访Rocke

Key Drivers [1] #IstioCon What Do We Need Else to Augment Istio? ● Strong security and privacy guarantees ○ Confidentiality, integrity and privacy protection for sensitive data ○ Strong isolation for

correctness of downstream window operators. • This approach preserves window integrity and guarantees that the results under shedding will not be approximations but a subset of the exact answers.

commonly applied on windows • Union operators combine two or more streams without ordering guarantees • elements have to be of the same type • Difference operators take two streams and output elements