caolei – 监控Apache Flink应用程序(入门) 1 https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#registering-metrics 2 https://ci.apache.org/projects/flink/flink-docs-release-1 numberOfFailedCheckpoints > threshold caolei – 监控Apache Flink应用程序(入门) 进度和吞吐量监控 – 10 3 https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/#task-chaining-and-resource-groups Apache Flink, which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for various reasons including the following: 1. It

\ tar -xvf Python-3.7.9.tgz && \ cd Python-3.7.9 && \ ./configure --without-tests --enable-shared && \ make -j6 && \ make install && \ ldconfig /usr/local/lib && \ cd .. && rm -f Python-3.7.9.tgz && rm 0x7fcd1ad0c0f0> Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how how to eventually write data to a table sink. The declared pipeline can be printed, optimized, and eventually executed in a cluster. The pipeline can work with bounded or unbounded streams which enables

\ tar -xvf Python-3.7.9.tgz && \ cd Python-3.7.9 && \ ./configure --without-tests --enable-shared && \ make -j6 && \ make install && \ ldconfig /usr/local/lib && \ cd .. && rm -f Python-3.7.9.tgz && rm 0x7fcd1ad0c0f0> Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how how to eventually write data to a table sink. The declared pipeline can be printed, optimized, and eventually executed in a cluster. The pipeline can work with bounded or unbounded streams which enables

input rates and periodically estimates operator selectivities. • The load shedder assigns a cost, ci, in cycles per tuple, and a selectivity, si, to each operator i. • The statistics manager collects channel or source Adjust processing rate of all operators to that of the slowest part of the pipeline ??? Vasiliki Kalavri | Boston University 2020 23 Progress is controlled though buffer availability

Vasiliki Kalavri | Boston University 2020 Types of Parallelism 7 B A C A B D A A B split Pipeline: A || B Task: B || C Data: A || A ??? Vasiliki Kalavri | Boston University 2020 8 Distributed computational steps • beneficial if it enables other optimizations, e.g. re-ordering • if the pipeline parallelism pays off Safety Profitability ??? Vasiliki Kalavri | Boston University 2020 24 • serialization and transport B A B ??? Vasiliki Kalavri | Boston University 2020 29 • removes pipeline parallelism but saves communication and serialization cost • if operators are separate, throughput

SQL extensions, CQL Java, Scala, Python, SQL Execution centralized distributed Parallelism pipeline pipeline, task, data State limited, in-memory partitioned, virtually unlimited, persisted to backends

SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Pipeline 吞吐 TB-PB TB-PB 未经⼤大规模⽣生产验证 性能 ⼀一般(分钟⼩小时级别) 快(秒级) 优秀 x2 稳定性 好 ⼀一般 已在阿⾥里里内部验证 API 差(MR) 最丰富

a catalog of all IDs ever seen and checking it for de-duplication is expensive • In a healthy pipeline though, most records will not be duplicates • Each worker maintains a Bloom Filter of all IDs

range {1, 2, …, m} ??? Vasiliki Kalavri | Boston University 2020 22 for j=1 to p do i = hj(x) ci,j++ Adding an element to the sketch stream elements x All counters are initialized to 0s 0 0 average of all counters, but the minimum. let f: array of length p for j=1 to p do i = hj(x) f[j] = ci,j return min(f[1], f[2], …, f[p]) ??? Vasiliki Kalavri | Boston University 2020 24 Computing top-k

output and state update atomic Vasiliki Kalavri | Boston University 2020 • Working with State: https://ci.apache.org/projects/flink/flink-docs- release-1.10/dev/stream/state/state.html • Managing State