. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.1 QuickStart: Table API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.2 QuickStart: DataStream 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.1 QuickStart: Table API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.2 QuickStart: DataStream 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . . . . .

[consumer group id], [number of partitions]) 15 / 79 Input Operations - Custom Sources (1/3) ▶ To create a custom source: extend the Receiver class. ▶ Implement onStart() and onStop(). ▶ Call store(data) close() } } 33 / 79 Output Operations (4/4) ▶ A better solution is to use rdd.foreachPartition ▶ Create a single connection object and send all the records in a RDD partition using that connection. dstream 79 Word Count in Spark Streaming (1/6) ▶ First we create a StreamingContex import org.apache.spark._ import org.apache.spark.streaming._ // Create a local StreamingContext with two working threads and

• stream-to-relation: define tables by selecting portions of a stream. • relation-to-stream: create streams through querying tables Declarative language: CQL 4 Vasiliki Kalavri | Boston University }); t (t, l1) (t, (l1, l2)) Streaming Iteration Example Terminate after 100 iterations Create the feedback loop 13 Vasiliki Kalavri | Boston University 2020 Blocking vs. Non-Blocking operators equivalent to having an append-only table to which new tuples are continuously added. 34 Vasiliki Kalavri | Boston University 2020 Example: CREATE STREAM CREATE STREAM OpenAuction( itemID INT, sellerID

Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain a hash table ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain a hash table Convert the stream into a multi-set of uniformly distributed random numbers using a hash function Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain a hash table The more different elements we encounter in the stream, the more different hash values we shall

Streaming Dataflow DataStream API Stream Processing DataSet API Batch Processing Table API & SQL Relational Table API & SQL Relational Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN Physical 统一 Operator 抽象 Pull-based operator Push-based operator 算子可自定义读取顺序 Table API & SQL 1.9 新特性 全新的 SQL 类型系统 DDL 初步支持 Table API 增强 统一的 Catalog API Blink Planner What’s new in Blink Planner 数据结构

步数RTransform I量h Apache Iceberg asic Data Metadata Database Table Partition Spec Manifest File TableMetadata Snapshot Current Table Version Pointer Apac2e Ice-er1 Bas3c Part3t354- f f3 Part3t354-2

w2 w1 w3 round-robin hash-based • Items are perfectly balanced among workers • No routing table required • Key semantics are not preserved: values of the same key might be routed to different different workers • Workers are responsible for roughly the same amount of keys • No routing table is required • Key semantics preserved: values of the same key are always processed by the same worker

是⼀一个分布式⼤大数据处理理引擎 * 可对有限数据流和⽆无限数据流进⾏行行有状态计算 * 可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch

14 ??? Vasiliki Kalavri | Boston University 2020 Load Shedding Road Map (LSRM) • A pre-computed table that contains materialized load shedding plans ordered by how much load shedding they will cause