process events. • What happens if consumers cannot keep up with the event rate? • drop messages • buffer messages in a queue: what if the queue grows larger than available memory? 2 ??? Vasiliki Kalavri process events. • What happens if consumers cannot keep up with the event rate? • drop messages • buffer messages in a queue: what if the queue grows larger than available memory? • block the producer Vasiliki Kalavri | Boston University 2020 22 ??? Vasiliki Kalavri | Boston University 2020 22 Durably buffer events in a channel or source Adjust processing rate of all operators to that of the slowest

block channels and upstream operators buffer incoming records ??? Vasiliki Kalavri | Boston University 2020 Pause-and-restart state migration 32 buffer incoming records block channels and Vasiliki Kalavri | Boston University 2020 Pause-and-restart state migration 32 re-configure buffer incoming records block channels and upstream operators • State is scoped to a single task Boston University 2020 Pause-and-restart state migration 32 re-configure state load state load buffer incoming records block channels and upstream operators • State is scoped to a single task

store in local buffer and possibly update state 3. produce output 5 mi mo Vasiliki Kalavri | Boston University 2020 What is a failure? op 1. receive an event 2. store in local buffer and possibly Kalavri | Boston University 2020 What is a failure? op 1. receive an event 2. store in local buffer and possibly update state 3. produce output 5 mi mo Was mi fully processed? Was mo delivered Kalavri | Boston University 2020 What is a failure? op 1. receive an event 2. store in local buffer and possibly update state 3. produce output What can go wrong: • lost events • duplicate or

are processed by Flink (see previous section). 3. Some operators in a streaming topology need to buffer events for some time (e.g. in a time window) for functional reasons. 4. Each computation in your the JVM for the direct buffer pool (in bytes). Status.JVM.Memory.Mapped.Me moryUsed job-/ taskmana ger The amount of memory used by the JVM for the mapped buffer pool (in bytes). Status

threads of control • avoid communication cost without losing pipeline parallelism • use a shared buffer for communication • Fused filters / projections at the source can significantly reduce I/O and process, the sender task serializes the outgoing records into a byte buffer. • A TaskManager needs one dedicated network buffer for each receiving task that any of its tasks need to send data to

consumers always online 5 Message queues • Asynchronous point-to-point communication • Lightweight buffer for temporary storage • Messages stored on the queue until they are processed and deleted • transactional

KeyedProcessFunction { /* we'll use a PriorityQueue to buffer not-yet-fully-sorted events */ private ValueState> queueState

pyx", line 244, in pyflink.fn_execution.coder_ ˓→impl_fast.FlattenRowCoderImpl._encode_one_row_to_buffer File "pyflink/fn_execution/coder_impl_fast.pyx", line 550, in pyflink.fn_execution.coder_ ˓→impl_fast

pyx", line 244, in pyflink.fn_execution.coder_ ˓→impl_fast.FlattenRowCoderImpl._encode_one_row_to_buffer File "pyflink/fn_execution/coder_impl_fast.pyx", line 550, in pyflink.fn_execution.coder_ ˓→impl_fast