Summer School 2019 Apache Flink Based on https://training.ververica.com Maximilian Michels Software Engineer / Consultant
 Committer @ Apache Beam / Apache Flink @stadtlegende Dr Paris Apache Flink @SenorCarbone Contents • DataSet API • DataStream API • Concepts • Set up an environment to develop Flink programs • Implement streaming data processing pipelines • Flink managed managed state • Event time Streaming in Apache Flink • Streams are natural • Events of any type like sensors, click streams, logs • Batch processing as a subset of stream processing Processing Data

Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems micro-batch processing ▶ Record-at-a-Time vs. declarative APIs 3 / 79 Outline ▶ Spark streaming ▶ Flink 4 / 79 Spark Streaming 5 / 79 Contribution ▶ Design issues • Continuous vs. micro-batch processing groupBy(window(col("timestamp"), "10 minutes", "5 minutes"), col("word")).count() 67 / 79 Flink 68 / 79 Flink ▶ Distributed data flow processing system ▶ Unified real-time stream and batch processing

vkalavri@bu.edu Spring 2020 1/30: Introduction to Apache Flink and Apache Kafka Vasiliki Kalavri | Boston University 2020 Apache Flink • An open-source, distributed data analysis framework • True Source Data Stream Operator Data Stream Sink Source Data Set Operator Data Set Sink Writing a Flink Program 1.Bootstrap Sources 2.Apply Operators 3.Output to Sinks 3 Vasiliki Kalavri | Boston University (let,1) (live,2) 4 Vasiliki Kalavri | Boston University 2020 Distributed architecture client Flink program JobManager web dashboard TaskManager TaskManager TaskManager 5 Vasiliki Kalavri |

Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 3/24: Exactly-once fault-tolerance in Apache Flink ??? Vasiliki Kalavri | Boston University 2020 Some slides in this lecture have been generously Output Logs 38 ??? Vasiliki Kalavri | Boston University 2020 Asynchronous checkpoints in Apache Flink 39 ??? Vasiliki Kalavri | Boston University 2020 40 • A source of increasing numbers partitioned Exactly-once state consistency (in Apache Flink) can be achieved only if all streaming sources are re-settable ??? Vasiliki Kalavri | Boston University 2020 44 • Flink checkpoints are initiated by the JobManager

Soon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.2.2 PyFlink + Flink ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.2.2.1 Coming . . 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . .

Soon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.2.2 PyFlink + Flink ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.2.2.1 Coming . . 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . .

various kinds of modern computing frameworks, e.g., Apache Spark [https://spark.apache.org/], Flink [https://flink.apache.org/], Doris [https://doris.apache.org/], Hive [https://hive.apache.org/], and Trino SQL or REST + code. For most scenarios, the superpower of corresponding engines, such as Spark, and Flink, is no longer necessary. That is, most work related to deployment, runtime optimization, etc., should Engine Side Extensions Connectors Connectors Connectors for Spark SQL Query Engine Connectors For Flink SQL Query Engine Connectors for Hive SQL Query Engine Connectors For Trino SQL Engine Kyuubi Insider

various kinds of modern computing frameworks, e.g., Apache Spark [https://spark.apache.org/], Flink [https://flink.apache.org/], Doris [https://doris.apache.org/], Hive [https://hive.apache.org/], and Trino SQL or REST + code. For most scenarios, the superpower of corresponding engines, such as Spark, and Flink, is no longer necessary. That is, most work related to deployment, runtime optimization, etc., should Engine Side Extensions Connectors Connectors Connectors for Spark SQL Query Engine Connectors For Flink SQL Query Engine Connectors for Hive SQL Query Engine Connectors For Trino SQL Engine Kyuubi Insider

various kinds of modern computing frameworks, e.g., Apache Spark [https://spark.apache.org/], Flink [https://flink.apache.org/], Doris [https://doris.apache.org/], Hive [https://hive.apache.org/], and Trino SQL or REST + code. For most scenarios, the superpower of corresponding engines, such as Spark, and Flink, is no longer necessary. That is, most work related to deployment, runtime optimization, etc., should Engine Side Extensions Connectors Connectors Connectors for Spark SQL Query Engine Connectors For Flink SQL Query Engine Connectors for Hive SQL Query Engine Connectors For Trino SQL Engine Kyuubi Insider