Python 3.6 to 3.9 PyFlink 1.15 Python 3.6 to 3.8 PyFlink 1.14 Python 3.6 to 3.8 You could check your Python version as following: 3 pyflink-docs, Release release-1.15 python3 --version Create a Python production when there are massive Python dependencies. It’s supported to use Python virtual environment in your PyFlink jobs, see PyFlink Dependency Management for more details. Create a virtual environment using environment, run: source venv/bin/activate That is, execute the activate script under the bin directory of your virtual environment. Create a virtual environment using conda To create a virtual environment using

Python 3.6 to 3.9 PyFlink 1.15 Python 3.6 to 3.8 PyFlink 1.14 Python 3.6 to 3.8 You could check your Python version as following: 3 pyflink-docs, Release release-1.16 python3 --version Create a Python production when there are massive Python dependencies. It’s supported to use Python virtual environment in your PyFlink jobs, see PyFlink Dependency Management for more details. Create a virtual environment using environment, run: source venv/bin/activate That is, execute the activate script under the bin directory of your virtual environment. Create a virtual environment using conda To create a virtual environment using

and reliable streaming applications • have a solid understanding of how stream processing systems work and what factors affect their performance • be aware of the challenges and trade-offs one needs Flink and Kafka to build a real-time monitoring and anomaly detection framework for datacenters. Your framework will: • Detect “suspicious” event patterns • Raise alerts for abnormal system metrics transaction analysis • Fraud detection, online risk calculation Example: Someone steals your phone and sings in your banking app. The app allows transfers of up to €1000 and so the thief makes transfers

Directory for checkpoints filesystem # # state.checkpoints.dir: path/to/checkpoint/folder/ In your Flink program: val env = StreamExecutionEnvironment.getExecutionEnvironment val checkpointPath: checkpointId, long timestamp) void restoreState(List state) Operator state 22 • A function can work with operator list state by implementing the ListCheckpointed interface • snapshotState() is invoked

terms of the number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers. millisBehindLatest buffer events for some time (e.g. in a time window) for functional reasons. 4. Each computation in your Flink topology (framework or user code), as well as each network shuffle, takes time and adds to checkpointing interval for each record. In practice, it has proven invaluable to add timestamps to your events at multiple stages (at least at creation, persistence, ingestion by Flink, publication by

Kalavri | Boston University 2020 14 Combining estimates • Average won’t work: The expected value of 2R is too large. • Median won’t work: it is always a power of 2, thus, if the correct estimate is between

operators can be placed at any location in the query plan • Dropping near the source avoids wasting work but it might affect results of multiple queries if the source is connected to multiple queries.

Edge endpoints must have different signs • When merging components, if flipping all signs doesn’t work => the graph is not bipartite Bipartite graph checking ??? Vasiliki Kalavri | Boston University

the size of intermediate results • execute selective joins first => follow-up joins will have less work to do Algebraic re-orderings ??? Vasiliki Kalavri | Boston University 2020 20 Safety • Ensure

for data streams • patterns, transformations, declarative • traditional blocking operators don’t work on streams • non-blocking versions or windows • how to define non-blocking aggregates • NB-SQL