Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 2/11: Windows and Triggers Vasiliki Kalavri | Boston University 2020 • Practical way to perform operations on unbounded input have a start and an end timestamp. • All built-in window assigners provide a default trigger that triggers the evaluation of a window once the (processing or event) time passes the end of the window. Long, ctx: OnTimerContext, out: Collector[OUT]) is invoked when a previously registered timer triggers. The timestamp argument gives the timestamp of the firing timer and the Collector allows emitting

Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new

overhead, latency? • Consumer receives a notification when new data is available • how to implement triggers? • Direct messaging • Direct network communication, UDP multicast, TCP • HTTP or RPC if the

state by implementing the ListCheckpointed interface • snapshotState() is invoked when Flink triggers a checkpoint of the stateful function. • restoreState() is always invoked when the job is started

University 2020 45 • When a source task receives a checkpoint barrier, it pauses emitting records, triggers a checkpoint of its local state at the state backend, and broadcasts barriers to all outgoing

pyflink-docs Release release-1.15 PyFlink Nov 23, 2022 CONTENTS 1 How to build docs locally 3 1.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . pyflink-docs, Release release-1.15 PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads, such as real-time data processing pipelines, large-scale exploratory HOW TO BUILD DOCS LOCALLY 1. Install dependency requirements python3 -m pip install -r dev/requirements.txt 2. Conda install pandoc conda install pandoc 3. Build the docs python3 setup.py build_sphinx

pyflink-docs Release release-1.16 PyFlink Nov 23, 2022 CONTENTS 1 How to build docs locally 3 1.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . pyflink-docs, Release release-1.16 PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads, such as real-time data processing pipelines, large-scale exploratory HOW TO BUILD DOCS LOCALLY 1. Install dependency requirements python3 -m pip install -r dev/requirements.txt 2. Conda install pandoc conda install pandoc 3. Build the docs python3 setup.py build_sphinx

and processing guarantees of streaming systems • be proficient in using Apache Flink and Kafka to build end-to-end, scalable, and reliable streaming applications • have a solid understanding of how stream Vasiliki Kalavri | Boston University 2020 Final Project You will use Apache Flink and Kafka to build a real-time monitoring and anomaly detection framework for datacenters. Your framework will: •

same initial state and given the same sequence of input tuples • convergent-capable: it can re-build internal state in a way that it eventually converges to a non-failure execution output • repeatable: acknowledge reception of input tuples notify upstream of oldest logged tuples necessary to re-build current state Vasiliki Kalavri | Boston University 2020 Upstream backup Recovery time • The

faults caused by high congestion. • In the presence of bursty traffic, CFC causes backpressure to build up fast and propagate along congested VCs to their sources which can be throttled. • Essentially