EIT Summer School 2019 Apache Flink Based on https://training.ververica.com Maximilian Michels apache.org> Software Engineer / Consultant
 Committer @ Apache Beam / Apache Flink @stadtlegende @stadtlegende Dr Paris Carbone Senior Researcher @ RISE Committer @ Apache Flink @SenorCarbone Contents • DataSet API • DataStream API • Concepts • Set up an environment to develop Implement streaming data processing pipelines • Flink managed state • Event time Streaming in Apache Flink • Streams are natural • Events of any type like sensors, click streams, logs • Batch processing

Kalavri
 vkalavri@bu.edu Spring 2020 1/30: Introduction to Apache Flink and Apache Kafka Vasiliki Kalavri | Boston University 2020 Apache Flink • An open-source, distributed data analysis framework file:///home/user/wordcount_out Run with a class entry point and arguments:
 ./bin/flink run -c org.apache.flink.examples.java.wordcount.WordCount \
 ./examples/batch/WordCount.jar Kalavri | Boston University 2020 Resources • Documentation • https://flink.apache.org/ • Community • https://flink.apache.org/community.html#mailing-lists • Conference • http://flink-forward.org/

Apache Flink的过去、现在和未来 杨克特（鲁尼） 阿里巴巴高级技术专家 过去 一切从2014年开始 2009 - 2014 2014 • 柏林工业大学博士生项目 • 基于流式 runtime 的批处理引擎 • 2014 年 8 月份 发布 Flink 0.6.0 Flink 0.7 Runtime Distributed Streaming Dataflow DataStream Processing & Streaming Analytics Event-driven Applications ✔ ✔ ✔ 扫码加入社群 与志同道合的码友一起 Code Up 阿里云开发者社区 Apache Flink China 2群 粘贴二维码 谢谢！

监控Apache Flink应用程序(入门) caolei Exported on 01/10/2020 caolei – 监控Apache Flink应用程序(入门) – 2 Table of Contents 1 Flink指标体系 ...................................................................... ............................................................................... 21 caolei – 监控Apache Flink应用程序(入门) – 3 4.13.2.1 Key Metrics ..................................................... caolei – 监控Apache Flink应用程序(入门) – 4 原文地址：https://www.ververica.com/blog/monitoring-apache-flink-applications-101 这篇博文介绍了Apache Flink内置的监控和度量系统，通过该系统，开发人员可以有效地监控他们的Flink作 业。通常，对于一个刚刚开始使用Apache Flink进行

Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 3/24: Exactly-once fault-tolerance in Apache Flink ??? Vasiliki Kalavri | Boston University 2020 Some slides in this lecture have been generously Protocol Output Logs 38 ??? Vasiliki Kalavri | Boston University 2020 Asynchronous checkpoints in Apache Flink 39 ??? Vasiliki Kalavri | Boston University 2020 40 • A source of increasing numbers partitioned the position up to which they were consumed when the checkpoint was taken. • Event logs like Apache Kafka can provide records from a previous offset of the stream. 43 ??? Vasiliki Kalavri | Boston

. . . . . . 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues large . . . . . . . . . . . . . . . . . . . . 30 1.3.5.2 Q2: An error occurred while calling z:org.apache.flink.client.python.PythonEnvUtils.resetCallbackClient 30 1.3.6 Data type issues . . . . . . .

. . . . . . 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues large . . . . . . . . . . . . . . . . . . . . 30 1.3.5.2 Q2: An error occurred while calling z:org.apache.flink.client.python.PythonEnvUtils.resetCallbackClient 30 1.3.6 Data type issues . . . . . . .

Spring 2020 2/11: Windows and Triggers Vasiliki Kalavri | Boston University 2020 • Practical way to perform operations on unbounded input • e.g. joins, holistic aggregates • Compute on most recent Boston University 2020 • “Introducing Stream Windows in Apache Flink”: https://flink.apache.org/ news/2015/12/04/Introducing-windows.html • “How Apache Flink™ Enables New Streaming Applications, Part 3”:

highly-parallel stream processors and is implemented in Apache Flink. • Each task informs its senders of its buffer availability via credit messages. • This way, senders always know whether receivers have the (ACM SGCOMM’94). • https://www.ververica.com/blog/how-flink-handles-backpressure • https://flink.apache.org/2019/06/05/flink-network-stack.html 30

Distributed execution in Flink ??? Vasiliki Kalavri | Boston University 2020 9 Identify the most efficient way to execute a query • There may exist several ways to execute a computation • query plans, e.g. order network buffer for each receiving task that any of its tasks need to send data to. Batching in Apache Flink • The TaskManagers ship data from sending tasks to receiving tasks. • The network component computation at scale (SOSP ’13). • Fabian Hueske, and Vasiliki Kalavri. Stream Processing with Apache Flink. (O’Reilly Media ’19). Lecture references ??? Vasiliki Kalavri | Boston University 2020