\ ln -s /usr/local/bin/python3 /usr/local/bin/python && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # install PyFlink RUN pip3 install apache-flink==1.15.2 Execute PyFlink jobs in application mode Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not Getting Started 19 pyflink-docs, Release release-1.15 DataStream Creation DataStream is a core component of the Python DataStream API. A DataStream object describes a pipeline of data transformations.

\ ln -s /usr/local/bin/python3 /usr/local/bin/python && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # install PyFlink RUN pip3 install apache-flink==1.15.2 Execute PyFlink jobs in application mode Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not Getting Started 19 pyflink-docs, Release release-1.16 DataStream Creation DataStream is a core component of the Python DataStream API. A DataStream object describes a pipeline of data transformations.

4 5 3 . . . 1 3, 4 2 1, 4 5 3 . . . ??? Vasiliki Kalavri | Boston University 2020 15 A component is a subgraph in which every vertex is reachable from all other vertices in the subgraph. Connected State: the graph and a component ID per vertex • initially equal to vertex ID • Iterative step: For each vertex • choose the min of neighbors’ component IDs and own component ID as the new ID • • if the component ID changed since the last iteration, notify neighbors 16 ??? Vasiliki Kalavri | Boston University 2020 1 4 3 2 5 i=0 Batch Connected Components 17 6 7 8 ??? Vasiliki Kalavri

Documentation • https://flink.apache.org/ • Community • https://flink.apache.org/community.html#mailing-lists • Conference • http://flink-forward.org/ 20 Vasiliki Kalavri | Boston University 2020 A distributed

key-groups • The metadata of key-group-to-subtask assignments are small. No need to maintain explicit lists of key-groups, only range boundaries. • The maximum parallelism parameter of an operator defines

Operator state Keyed state State types 6 Vasiliki Kalavri | Boston University 2020 A pluggable component that determines how state is stored, accessed, and maintained. State backends are responsible

. trigger evictor evaluation function result stream Custom windows 20 • Describe each component Vasiliki Kalavri | Boston University 2020 32 4 2 5 7 44 8 18 Window max over 5 last elements

Boston University 2020 Implementation • Load shedding is commonly implemented by a standalone component integrated with the stream processor • The load shedder continuously monitors input rates or

Apache Flink • The TaskManagers ship data from sending tasks to receiving tasks. • The network component of a TaskManager collects records in buffers before they are shipped, i.e., records are not shipped

2 Synopsis: a lossy, compact summary of the input stream input stream synopsis maintenance component user queries approximate results ??? Vasiliki Kalavri | Boston University 2020 A simple and