u0169V6m6NXcGsky8glShQLNX+er2E5bFKA0TVOuO56Y myKkynAmclLuZxpSyER1gx1JY9RBPjt1Qk6t0idRomx JQ2bq74mcxlqP49B2xtQM9aI3Ff/zOpmJLoOcyzQzKNl 8UZQJYhIy/Zv0uUJmxNgSyhS3txI2pIoyY9Mp2xC8xZeX iV+vXdW8u/Nq47pIowTHcAJn4MEFNOAWmuADgwE8wyu8 u0169V6m6NXcGsky8glShQLNX+er2E5bFKA0TVOuO56Y myKkynAmclLuZxpSyER1gx1JY9RBPjt1Qk6t0idRomx JQ2bq74mcxlqP49B2xtQM9aI3Ff/zOpmJLoOcyzQzKNl 8UZQJYhIy/Zv0uUJmxNgSyhS3txI2pIoyY9Mp2xC8xZeX iV+vXdW8u/Nq47pIowTHcAJn4MEFNOAWmuADgwE8wyu8 u0169V6m6NXcGsky8glShQLNX+er2E5bFKA0TVOuO56Y myKkynAmclLuZxpSyER1gx1JY9RBPjt1Qk6t0idRomx JQ2bq74mcxlqP49B2xtQM9aI3Ff/zOpmJLoOcyzQzKNl 8UZQJYhIy/Zv0uUJmxNgSyhS3txI2pIoyY9Mp2xC8xZeX iV+vXdW8u/Nq47pIowTHcAJn4MEFNOAWmuADgwE8wyu8

• The performance of these operation is proportional to the size of the state. ▶ mapWithState • It is executed only on set of keys that are available in the last micro batch. • The performance is proportional • The performance of these operation is proportional to the size of the state. ▶ mapWithState • It is executed only on set of keys that are available in the last micro batch. • The performance is proportional • The performance of these operation is proportional to the size of the state. ▶ mapWithState • It is executed only on set of keys that are available in the last micro batch. • The performance is proportional

re-partitioning and migration • minimize communication • keep duration short • minimize performance disruption, e.g. latency spikes • avoid introducing load imbalance • Resource management Kalavri | Boston University 2020 12 • Detect environment changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict When and how much to adapt? 12 • Detect environment changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict

metrics.latency.granularity: subtask), enabling latency tracking can significantly impact the performance of the cluster. It is recommended to only enable it to locate sources of latency during debugging 1550652804788.1550652804788.1&__hssc=216506377.3.1551426921706&__hsfp=3017175250 hand, if you job’s performance is starting to degrade among the first metrics you want to look at are memory consumption and your TaskManagers are constantly under very high load, you might be able to improve the overall performance by decreasing the number of task slots per TaskManager (in case of a Standalone setup), by providing

to apply the re-configuration? 3 • Detect environment changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict requirements 7 ▸ Accuracy ▸ no over/under-provisioning ▸ Stability ▸ no oscillations ▸ Performance ▸ fast convergence scaling controller detect symptoms decide whether to scale decide MIMO too complex • Action • predictive, dataflow-wide The output signal is the delay time Performance depends on parameter selection, e.g. poles placement, sampling period, damping Cannot identify

Kalavri | Boston University 2020 Fault-tolerance trade-offs 12 Steady-state overhead • How is performance affected by the fault-tolerance mechanism under normal, failure- free operation? • How much been checkpointed, i.e. the user’s non- deterministic code is not re-executed Bloom filters for performance • Maintaining a catalog of all IDs ever seen and checking it for de-duplication is expensive

placement decisions • different algorithms, e.g. hash-based vs. broadcast join • What does performance depend on? • input data, intermediate data • operator properties • How can we estimate the Boston University 2020 13 • Profitability: under what conditions does the optimization improve performance? • can the decision be automatic? • Safety: under what conditions does the optimization preserve

have a solid understanding of how stream processing systems work and what factors affect their performance • be aware of the challenges and trade-offs one needs to consider when designing and deploying

Elasticity Selectively drop records: • Temporarily trades-off result accuracy for sustainable performance. • Suitable for applications with strict latency constraints that can tolerate approximate