stream ingestion 12 ??? Vasiliki Kalavri | Boston University 2020 –Leslie Lamport The distributed snapshot algorithm described here came about when I visited Chandy, who was then at the University of (liveness): Obtain a valid system configuration A full system configuration is eventually captured A snapshot algorithm attempts to capture a coherent global state of a distributed system ??? Vasiliki Kalavri p2xD8OZfXiT+ae2q5t2dV+vXR RolOIQjOAEPLqAOt9AHxhweIZXeHOk8+K8Ox+z1iWnmDmAP3A+fwCD9I4G Snapshot of C {s1 1, s1 2, s1 3} {m0} ACXicbZC9TsMwFIWd8lfCX4CRJ

2020 State migration strategies • Stop-and-restart • halt the whole computation, take a state snapshot of all operators, restart • unnecessary stalls if only one or few operators need to be rescaled single task • Each stateful task is responsible for processing and state management 31 snapshot snapshot block channels and upstream operators buffer incoming records ??? Vasiliki Kalavri

Container 5. Start Task Manager 6. Schedule Task YARN RM K8S RM 增量 Checkpoint 时间 全量状态 增量状态 增量 snapshot 基于 credit 的流控机制 Streaming SQL ------------------------- | USER_SCORES | -------------------------

efficient search while MBs only offer topic-based subscription. • DB query results depend on a snapshot and clients are not notified if their query result changes later. 13 Message delivery and ordering

Flink applications ??? Vasiliki Kalavri | Boston University 2020 • A consistent and complete snapshot of an application’s state • Checkpoints are automatically created and removed by Flink. • Savepoints

processing systems, such as Apache Graph, GraphX, Pregel, operate offline. They are built to analyze a snapshot of the real graph: • the Facebook social network on January 30 2016 • user web logs gathered

Apache Iceberg asic Data Metadata Database Table Partition Spec Manifest File TableMetadata Snapshot Current Table Version Pointer Apac2e Ice-er1 Bas3c Part3t354- f f3 Part3t354-2 f4 f5 Part3t354-3