2, 4), (1, 5, 3)) inputStream.keyBy(0).sum(1).print() 16 keyBy what’s the output for each key? Vasiliki Kalavri | Boston University 2020 coMap / coFlatMap val factors: DataStream[(String, Double)] options conf/flink-conf.yaml contains the configuration options as a collection of key-value pairs with format key:value Common options you might need to adjust: jobmanager.heap.size: JVM heap size A topic identifies a category of stream records stored in a Kafka cluster. 
 Records consist of a key, a value, and a timestamp. A producer publishes a stream of records to a Kafka topic and a consumer

resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations to ensure result correctness resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations to ensure result correctness resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations to ensure result correctness

• A log can be partitioned, so that each partition can be read and written independently of others • a topic is a set of partitions • Within each partition, every message carries an offset, a monotonically monotonically increasing sequence number • Within a partition, all messages are totally ordered but there is no ordering guarantee across partitions 28 29 Failure handling • The broker does not delays: If a message is slow to process, this delays processing of subsequent messages, as each partition is read by a single thread What would you use when priority is: - latency but not ordering?

...................................................................................... 14 4.12.1 Key Metrics .......................................................................................... .................................................................................... 17 4.13.1.1 Key Metrics .......................................................................................... .......................................... 21 caolei – 监控Apache Flink应用程序(入门) – 3 4.13.2.1 Key Metrics ..........................................................................................

replicates a stream, commonly to be used as input to multiple downstream operators. • Group by / Partition Operators split a stream into sub-streams according to a function or the event contents. • one Pattern-Matching: a simpler approach SELECT ‘modified-pattern123’, X.CustomerId FROM webevents PARTITION BY CustomerId AS PATTERN (X Y Z) WHERE X.Event = ‘order’ AND Y.Event = ‘rebate’ Z.Event = ‘cancel’ AND Z.ItemID = Y.ItemID Partitions the stream into substreams according to a key A sequence of events that immediately follow one another AS PATTERN (X V* Y W* Z) • Match zero

Keyed state is scoped to a key defined in the operator’s input records • Flink maintains one state instance per key value and partitions all records with the same key to the operator task that maintains maintains the state for this key • State access is automatically scoped to the key of the current record so that all records with the same key access the same state State management in Apache Flink Vasiliki Kalavri | Boston University 2020 RocksDB 10 RocksDB is an LSM-tree storage engine with key/value interface, where keys and values are arbitrary byte streams. https://rocksdb.org/ https://www

pairs where the values for each key are aggregated using the given reduce function. ▶ countByValue • Returns a new DStream of (K, Long) pairs where the value of each key is its frequency in each RDD of pairs where the values for each key are aggregated using the given reduce function. ▶ countByValue • Returns a new DStream of (K, Long) pairs where the value of each key is its frequency in each RDD of pairs where the values for each key are aggregated using the given reduce function. ▶ countByValue • Returns a new DStream of (K, Long) pairs where the value of each key is its frequency in each RDD of

并DMerge-On-Rea- Mkh取 支持I量P取便于进一 步数RTransform I量h Apache Iceberg asic Data Metadata Database Table Partition Spec Manifest File TableMetadata Snapshot Current Table Version Pointer Apac2e Ice-er1 Bas3c cHFck_ePenON :ET =.4UE: (... TX6.3: /E4ETE 1R75 cHFck_ePenON W2ERE LMFIAMy_key - XX TX6.4: U8/.TE cHFck_ePenON :ET =.4UE:(... W2ERE LMFIAMy_key - XX :nALNEoO 5AnFDeNO /AOA//eHeOe 1FHeN 36:ERT 134E: /E4ETE

resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations to ensure result correctness migration if the state is large • Progressive • move state to be migrated in smaller pieces, e.g. key-by-key • can be used to interleave state transfer with processing • migration duration might increase

1. Load: read the graph from disk and partition it in memory 10 ??? Vasiliki Kalavri | Boston University 2020 1. Load: read the graph from disk and partition it in memory 2. Compute: read and mutate from disk and partition it in memory 2. Compute: read and mutate the graph state 11 ??? Vasiliki Kalavri | Boston University 2020 1. Load: read the graph from disk and partition it in memory 2 2020 Distributed Stream Connected Components 36 1. partition the edge stream, e.g. by source Id 2. maintain a disjoint set in each partition 3. periodically merge the partial disjoint sets into