Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 4/02: Elasticity policies and state migration ??? Vasiliki Kalavri | Boston University 2020 Streaming applications are long-running • Workload requires state migration with correctness guarantees. ??? Vasiliki Kalavri | Boston University 2020 State migration 29 ??? Vasiliki Kalavri | Boston University 2020 State migration strategies • Stop-and-restart during migration if the state is large • Progressive • move state to be migrated in smaller pieces, e.g. key-by-key • can be used to interleave state transfer with processing • migration duration

reconfiguration mechanism often relies on fault-tolerance mechanism • State re-partitioning and migration • minimize communication • keep duration short • minimize performance disruption, e.g. latency • Partitioning function performance • space required to implement routing • lookup cost • Migration performance • re-assignment computation cost • state movement cost 25 State redistribution • Evenly distributes keys across parallel tasks • Fast to compute, no routing state • High migration cost • When a new node is added, state is shuffled across existing and new nodes • Random I/O

Challenges: computation progress, fault-tolerance and result guarantees, automatic scaling and state migration, out-of-order processing 37 Vasiliki Kalavri | Boston University 2020 • No particular basic stream

• Ensure security constraints: what are the trusted hosts for each operator? • Ensure state migration: if placement is dynamic and the operator is stateful, its state must be moved in a consistent