3 Flink processes ??? Vasiliki Kalavri | Boston University 2020 • Flink requires a sufficient number of processing slots in order to execute all tasks of an application. • The JobManager cannot supports the following restart strategies: • The fixed-delay strategy restarts an application a fixed number of times and waits a configured time between two restart attempts. • The failure-rate strategy long as a configurable failure rate is not exceeded. The failure rate is specified as the maximum number of failures within a time interval. • e.g. you can configure that an application be restarted as

messages • buffer messages in a queue: what if the queue grows larger than available memory? • block the producer (back-pressure, flow control) 2 ??? Vasiliki Kalavri | Boston University 2020 Load balance for all their receivers and receivers regularly send notifications upstream containing their number of available credits. • One credit corresponds to some amount of buffer space so that a sender • This is crucial in the presence of data skew where a single overloaded task could otherwise block the flow of data to all other downstream operator instances. • On the downside, the additional

dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations to ensure result correctness ??? Vasiliki Kalavri | Boston University 2020 only one or few operators need to be rescaled • Partial pause and restart • only temporarily block the affected dataflow subgraph • usually the operator to be scaled and upstream channels • All-at-once a single task • Each stateful task is responsible for processing and state management 31 block channels and upstream operators ??? Vasiliki Kalavri | Boston University 2020 Pause-and-restart

of this series? 3 • the sum of all the values • the sum of the squares of the values • the number of observations var = ∑ (xi − μ)2 N ??? Vasiliki Kalavri | Boston University 2020 A simple and of this series? 3 • the sum of all the values • the sum of the squares of the values • the number of observations • μ = sum / count • var = (sum of squares / count) - μ2 Then var = ∑ (xi − μ)2 of this series? 3 • the sum of all the values • the sum of the squares of the values • the number of observations We can compute the three summary values in a single pass through the data. • μ

Counting distinct elements 2 ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one or or multiple webpages ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one or multiple solution: maintain a hash table ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one or

total number of full restarts since this job was submitted. numberOfCompletedCheckpoints job The number of successfully completed checkpoints. numberOfFailedCheckpoints job The number of failed Flink应用程序(入门) 监控 – 8 3.2 仪表盘示例 Figure 1: Uptime (35 minutes), Restarting Time (3 milliseconds) and Number of Full Restarts (7) caolei – 监控Apache Flink应用程序(入门) 监控 – 9 Figure 2: Completed Checkpoints Description numRecordsOutPerSecond task The number of records this operator/task sends per second. numRecordsOutPerSecond operator The number of records this operator sends per second. caolei

?? Vasiliki Kalavri | Boston University 2020 Operator selectivity 6 • The number of output elements produced per number of input elements • a map operator has a selectivity of 1, i.e. it produces different threads • The optimizer can interact with the scheduler and fuse operators according to the number of available cores / threads • Fused operators can share the address space but use separate threads by construction • Elastic scaling techniques enable dynamic operator fission by adjusting the number of parallel operator instances according to data rates • straight-forward for stateless operators

taskmanager.numberOfTaskSlots: The number of parallel operator or user function instances that a single TaskManager can run. This value is typically proportional to the number of physical CPU cores that the the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). 18 Vasiliki Kalavri | Boston University 2020 Start Flink:
 ./bin/start-cluster.sh Stop Flink:
 ./bin/stop-cluster records that is continually appended to—a structured commit log. An offset is a sequential id number assigned to records within a partition. It uniquely identifies records within each partition.

. . . . . . . . . . . . . . 22 1.3.1.2 O2: Java gateway process exited before sending its port number . . . . . . . . . . . 22 1.3.2 Usage issues . . . . . . . . . . . . . . . . . . . . . . . . . . from pyflink.common.watermark_strategy import WatermarkStrategy from pyflink.datastream.connectors.number_seq import NumberSequenceSource env = StreamExecutionEnvironment.get_execution_environment() seq_num_source documentation for more details. 1.3.1.2 O2: Java gateway process exited before sending its port number The exception stack is as following: Traceback (most recent call last): File "/Users/dianfu/c

. . . . . . . . . . . . . . 22 1.3.1.2 O2: Java gateway process exited before sending its port number . . . . . . . . . . . 22 1.3.2 Usage issues . . . . . . . . . . . . . . . . . . . . . . . . . . from pyflink.common.watermark_strategy import WatermarkStrategy from pyflink.datastream.connectors.number_seq import NumberSequenceSource env = StreamExecutionEnvironment.get_execution_environment() seq_num_source documentation for more details. 1.3.1.2 O2: Java gateway process exited before sending its port number The exception stack is as following: Traceback (most recent call last): File "/Users/dianfu/c