. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.2.1 O1: How to prepare Python Virtual Environment . . . . . . . . . . . . . . . . . . . 24 1.3.2.2 O2: How to add Python Files . . . following: 3 pyflink-docs, Release release-1.15 python3 --version Create a Python virtual environment Virtual environment gives you the ability to isolate the Python dependencies of different projects supported to use Python virtual environment in your PyFlink jobs, see PyFlink Dependency Management for more details. Create a virtual environment using virtualenv To create a virtual environment using virtualenv

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.2.1 O1: How to prepare Python Virtual Environment . . . . . . . . . . . . . . . . . . . 24 1.3.2.2 O2: How to add Python Files . . . following: 3 pyflink-docs, Release release-1.16 python3 --version Create a Python virtual environment Virtual environment gives you the ability to isolate the Python dependencies of different projects supported to use Python virtual environment in your PyFlink jobs, see PyFlink Dependency Management for more details. Create a virtual environment using virtualenv To create a virtual environment using virtualenv

SQL and define queries over tables. • stream-to-relation: define tables by selecting portions of a stream. • relation-to-stream: create streams through querying tables Declarative language: CQL 4 Language • Ad-hoc SQL queries • Updates on database tables • Continuous queries on data streams • New streams (derived) are defined as virtual views in SQL • Semantics are equivalent to having an state. • TERMINATE: produce the result. Note that it is allowed to define and maintain local tables as state. 36 Vasiliki Kalavri | Boston University 2020 Example: AVG UDA AGGREGATE myavg(Next

throughput is limited by the processing rate of the slowest task. • Parallel tasks are connected via virtual channels multiplexed over TCP connections: • In the presence of skew, a single overload channel link-by-link, per virtual channel congestion control technique used in ATM network switches. • To exchange data through an ATM network, each pair of endpoints first needs to establish a virtual circuit (VC) the credit of a receiver drops to zero (or a specified threshold), backpressure appears on its virtual channel. ??? Vasiliki Kalavri | Boston University 2020 29 Remarks on CFC • Bakcpressure is inflicted

stream? • In traditional data processing applications, we know the entire dataset in advance, e.g. tables stored in a database. A data stream is a data set that is produced incrementally over time, rather materialized views src dest total 1 2 20K sum src dest bytes 1 2 20K • Base streams update relation tables and derived streams update materialized views. • An operator outputs event streams that describe semantics of the operator. 19 Vasiliki Kalavri | Boston University 2020 • Base streams update relation tables and derived streams update materialized views. • An operator outputs event streams that describe

the-world-beyond-batch-streaming-102 • Watermarks, Tables, Event Time, and the Dataflow Model: https:// www.confluent.jp/blog/watermarks-tables-event-time-dataflow-model/ Further reading 22

are a Windows user, you are advised to use Windows subsystem for Linux (WSL), Cygwin, or a Linux virtual machine to run Flink in a UNIX environment. • A Java 8.x installation. To develop Flink applications

to the same parallel instance • Some kind of hashing is typically used • Maintaining routing tables or an index for all key mappings is usually impractical • Skewed load is challenging to handle