<#WorldCup, 480> <#StarWars, 300> Any non-trivial streaming computation maintains state: • rolling aggregations • window contents • input offsets • machine learning models State in dataflow computations 480> <#StarWars, 300> <#Brexit> Any non-trivial streaming computation maintains state: • rolling aggregations • window contents • input offsets • machine learning models State in dataflow computations <#StarWars, 300> <#Brexit, 521> Any non-trivial streaming computation maintains state: • rolling aggregations • window contents • input offsets • machine learning models State in dataflow computations

(FileSink .for_row_format('/tmp/sink', Encoder.simple_string_encoder("UTF-8")) .with_rolling_policy(RollingPolicy.default_rolling_policy( part_size=1024 ** 3, rollover_interval=15 * 60 * 1000, inactivity_interval=5

(FileSink .for_row_format('/tmp/sink', Encoder.simple_string_encoder("UTF-8")) .with_rolling_policy(RollingPolicy.default_rolling_policy( part_size=1024 ** 3, rollover_interval=15 * 60 * 1000, inactivity_interval=5

<#StarWars, 300> <#Brexit, 521> Any non-trivial streaming computation maintains state: • rolling aggregations • window contents • input offsets • machine learning models State in dataflow computations

cause unnecessary result degradation! • Load shedding components rely on statistics gathered during execution: • A statistics manager module monitors processing and input rates and periodically estimates assigns a cost, ci, in cycles per tuple, and a selectivity, si, to each operator i. • The statistics manager collects metrics and estimates those parameters either continuously or by running the sampling rate • The rate defines the probability to discard a tuple and is computed based on statistics and operator selectivity • The optimization objective is to achieve the highest possible accuracy

filtering and alarm activation • Aggregation of multiple sensors and joins • Examples • Real-time statistics, e.g. weather maps • Monitor conditions to adjust resources, e.g. power generation • Monitor

Boston University 2020 Cost-based optimization 11 Parsed program representation Optimizer statistics input plan A plan B output Lowest-cost plan ??? Vasiliki Kalavri | Boston University 2020