a coordinator or generated periodically • We want to snapshot stream process graphs after the complete computation of an epoch. Epoch Snapshotting 35 ??? Vasiliki Kalavri | Boston University 2020 LWv2vbdeatzXafRQEfoGJ0iG12gDrpFXeQginL0jF7Rm/FkvBjvxsesdcmoZw7QHxifP4y cle0= A epoch-complete consistent cut that includes events that Epoch cuts p4

0 码力 | 81 页 | 13.18 MB | 1 年前

3

block channels and upstream operators All affected operators block until the reconfiguration is complete • State is scoped to a single task • Each stateful task is responsible for processing and operators can check the frontier (watermark) at the output of the stateful operator to ensure only complete state is migrated Live state migration ??? Vasiliki Kalavri | Boston University 2020 36 control operators can check the frontier (watermark) at the output of the stateful operator to ensure only complete state is migrated Helpers buffer data that cannot yet be safely routed and configuration commands

appended to the result table since the last trigger will be written to the external storage. 2. Complete: the entire updated result table will be written to external storage. 3. Update: only the rows appended to the result table since the last trigger will be written to the external storage. 2. Complete: the entire updated result table will be written to external storage. 3. Update: only the rows appended to the result table since the last trigger will be written to the external storage. 2. Complete: the entire updated result table will be written to external storage. 3. Update: only the rows

Reconfiguring Flink applications ??? Vasiliki Kalavri | Boston University 2020 • A consistent and complete snapshot of an application’s state • Checkpoints are automatically created and removed by Flink the new set of parallel tasks • For exactly-once results, we need to prevent a checkpoint to complete after the savepoint! • Use the integrated savepoint-and-cancel command 15 Scaling from a Savepoint

arrival rate or order f’ ∞ ? Continuously arriving, possibly unbounded data f read write Complete data accessible in persistent storage 30 Vasiliki Kalavri | Boston University 2020 Consider

add/sort this event into the queue */ /* set an event-time timer for when the stream is complete up to the event-time of this event */ } @Override public void onTimer(long timestamp, OnTimerContext

recovery node may need to re-process many tuples • all tuples that contributed to lost state • a complete queue-trimming interval worth of tuples, if level-0 and level-1 acks are periodically transmitted

is appended to the stream table? Synopses: Maintain summaries of streaming data instead of the complete history. 29 Vasiliki Kalavri | Boston University 2020 Stream synopses requirements • Single-pass:

data items • Batching hurts latency as events can only be processed once the entire batch is complete Batching Profitability A A’ Spark Streaming • Treat streaming computation as a series of deterministic

∪τ S = Lτ ∪ Sτ Languages supporting union operators and non-blocking UDAs on data streams are complete, in the sense that they can express every monotonic function on their input. 49 Vasiliki Kalavri