dataflow with sources S1, S2, … Sn and rates λ1, λ2, … λn identify the minimum parallelism πi per operator i, such that the physical dataflow can sustain all source rates. S1 S2 λ1 λ2 S1 S2 π=2 serialization send message waiting waiting 13 ??? Vasiliki Kalavri | Boston University 2020 14 o1 src o2 back-pressure target: 40 rec/s 10 rec/s 100 rec/s Which operator is the bottleneck? What if Boston University 2020 14 o1 src o2 back-pressure target: 40 rec/s 10 rec/s 100 rec/s Which operator is the bottleneck? What if we scale ο1 x 4? How much to scale ο2? o1 cannot keep up waiting for

Flink作业监控的基础是它的度量系统，该系统由两个部分组成: Metrics和MetricsReporters。 1.1 Metrics Flink提供了一套全面的内置Metrics: • JVM堆/非堆/直接内存的使用情况(任务粒度) • 作业重启次数(作业粒度) • 每秒处理的数据量(操作符粒度) • ...... 作为用户，您可以并且应该向函数中添加应用程序相关的metrics。这些metr above, it is also possible to use Flink’s metrics system to gather insights about system resources, i.e. memory, CPU & network-related metrics for the whole machine as opposed to the Flink processes alone as a starting point when you first think about how to successfully monitor your Flink application. I highly recommend to start monitoring your Flink application early on in the development phase. This

secondary I1 O1 N’i I1 Vasiliki Kalavri | Boston University 2020 Passive Standby • Each primary periodically checkpoints its state and sends it to the secondary 6 Ni primary secondary I1 O1 N’i N’i update checkpoint send state ??? Vasiliki Kalavri | Boston University 2020 How can we make sure that checkpoints are meaningful and coherent? 7 ??? Vasiliki Kalavri | Boston University 2020 University 2020 –Leslie Lamport The distributed snapshot algorithm described here came about when I visited Chandy, who was then at the University of Texas in Austin. He posed the problem to me over

shedding as an optimization problem N: query network I: set of input streams with known arrival rates C: system processing capacity H: headroom factor, i.e. a conservative estimate of the percentage of Load(N(I)): the load as a fraction of the total capacity C that network N(I) presents Uacc: the aggregate utility 6 Find a new network N' such that Load(N’(I))< H x C and Uacc(N(I)) - Uacc(N'I)) Uacc(N'I)) is minimized ??? Vasiliki Kalavri | Boston University 2020 Implementation • Load shedding is commonly implemented by a standalone component integrated with the stream processor • The load

. . . . . . . . . . . . . . . . . . . 22 1.3.1.1 O1: Scala Dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.3.1.2 O2: Java gateway process exited before sending its port . . . . . . . . . . . . . . . . . . . 24 1.3.2.1 O1: How to prepare Python Virtual Environment . . . . . . . . . . . . . . . . . . . 24 1.3.2.2 O2: How to add Python Files . . . . . . . . . . . . . issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.3.1 O1: InaccessibleObjectException: Unable to make field private final byte[] java.lang.String.value accessible:

. . . . . . . . . . . . . . . . . . . 22 1.3.1.1 O1: Scala Dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.3.1.2 O2: Java gateway process exited before sending its port . . . . . . . . . . . . . . . . . . . 24 1.3.2.1 O1: How to prepare Python Virtual Environment . . . . . . . . . . . . . . . . . . . 24 1.3.2.2 O2: How to add Python Files . . . . . . . . . . . . . issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.3.1 O1: InaccessibleObjectException: Unable to make field private final byte[] java.lang.String.value accessible:

nodes other apps I1 I2 O1 O2 N’1 N’K N’2 … I’1 I’2 O’1 O’2 6 Vasiliki Kalavri | Boston University 2020 Assumptions Ni primary secondary I1 I2 O1 O2 N’i I’1 I’2 O’1 O’2 • The communication order-preserving, reliable message transport, e.g. TCP. • Failures are single-node and fail- stop, i.e. no network partitions or multiple simultaneous failures are considered. • The secondary node let Of be the pre-failure execution of the primary and O’ the output produced by the secondary after recovery. • Precise recovery guarantees Of + O’ = Oe • Rollback recovery allows duplicate tuples downstream:

HBase 集u分析 CDC 数a 、CDC记录实时写入HBase。高吞P + 低延迟。 2、小vSg询延迟低。 3、集u可拓展 ci评C B点 、行存o引不适O分析A务。 2、HBase集ur护成e较高。 3、通过Re12o4Server定DHF23e， ServerlB化Rs存完H用不上。 4、数a格式q定HF23e，不cF拓展到 +arquet、Avro、Orcn。 t点 A3a/21 A3a/21 Kudu 维护 CDC 数据p 、支持L时更新数据，时效性佳。 2、CK加速，适合OLAP分析。 方案评估 优点 、cedKudup群，a较小众。维护 O本q。 2、H HDFS / S3 / OSS 等D裂。数据c e，且KAO本不如S3 / OSS。 3、Kudud批量P描不如3ar4u1t。 4、不支持增量SF。 h点 直接D入CDC到Hi2+分析 、流程能E作 GE=D.<, chan>=E.a<O快。 3、方便上S3 OSS，超高性价比。 方案s估 优点 1、增量和全量表割p，时效性不足。 2、r计和l护额外hChang+ S+4表。 3、计算引擎并非原g支UCDC。 4、不支U实时U13+24。

of output elements produced per number of input elements • a map operator has a selectivity of 1, i.e. it produces one output element for each input element it processes • an operator that tokenizes estimate the cost of different strategies? • before execution or during runtime Query optimization (I) ??? Vasiliki Kalavri | Boston University 2020 10 Optimization strategies • enumerate equivalent remove idempotent operations, e.g. two selections on the same predicate • remove a dead subgraph, i.e. one that never produces output Redundancy elimination variations How can no-op or idempotent operators

Continuous Processing & Streaming Analytics Event-driven Applications ✔ ✔ 未来 Micro Services O_0 O_1 I_0 I_1 I_2 P_0 P_1 P_2 S_0 S_1 Order Inventory Payment Shipping Flow-Control Async Call