downstream. ??? Vasiliki Kalavri | Boston University 2020 28 Credit-based flow control • This classic networking technique turns out to be very useful for load management in modern, highly-parallel stream processors

needs to be aware of message loss, producers and consumers always online 5 Message queues • Asynchronous point-to-point communication • Lightweight buffer for temporary storage • Messages stored on Effective failure handling, crashes or disconnects • Broker responsible for message durability • Asynchronous communication, i.e. producer only needs to receive ack from broker 9 Communication patterns Space Decoupling Time Decoupling Synchronization Decoupling Message-passing RPC/RMI Asynchronous RPC Futures Message Queues Pub/Sub Yes Yes Yes Can you fill this in? 19 Pub/Sub vs. other

the barriers, Flink knows that all records before the barriers will never be needed again. ▶ Asynchronous barrier snapshotting for globally consistent checkpoints. 74 / 79 Fault Tolerance (2/2) ▶ Acks the barriers, Flink knows that all records before the barriers will never be needed again. ▶ Asynchronous barrier snapshotting for globally consistent checkpoints. 74 / 79 Summary 75 / 79 Summary Structured streaming ▶ Flink • Unified batch and stream • Different windowing semantics • Asynchronous barriers 76 / 79 Summary 77 / 79 References ▶ M. Zaharia et al., “Spark: The Definitive Guide”

Traditional DW vs. SDW Traditional DW SDW Update Frequency low high Update propagation synchronized asynchronous Data historical recent and historical ETL process complex fast and light-weight ETL: Extract-Transform-Load


snapshot algorithm that is used in distributed systems for recording a consistent global state of an asynchronous system ??? Vasiliki Kalavri | Boston University 2020 Requirements: • Taking a snapshot does Committed The Epoch Commit Protocol Output Logs 38 ??? Vasiliki Kalavri | Boston University 2020 Asynchronous checkpoints in Apache Flink 39 ??? Vasiliki Kalavri | Boston University 2020 40 • A source need to checkpoint the complete application state in every checkpoint? • RocksDB supports both asynchronous and incremental checkpoints: • take a local snapshot and use a background thread to copy the