Mechanism: How to apply the re-configuration? 3 • Detect environment changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions processing a tuple and all its derived results • Policy • each operator as a single-server queuing system • generalized Jackson networks • Action • predictive, at-once for all operators ??? Vasiliki processing a tuple and all its derived results • Policy • each operator as a single-server queuing system • generalized Jackson networks • Action • predictive, at-once for all operators Too fine-grained

broker: a system that connects event producers with event consumers. • It receives messages from the producers and pushes them to the consumers. • A TCP connection is a simple messaging system which which connects one sender with one recipient. • A general messaging system connects multiple producers to multiple consumers by organizing messages into topics. 7 Message Broker producer producer allow the use of wildcards. 21 Content-based Pub/Sub • Events are grouped according to event properties or contents. • data attributes or meta-data. • Consumers subscribe to events by specifying

stream elements on-the-fly using limited memory 2 Vasiliki Kalavri | Boston University 2020 Properties of data streams • They arrive continuously instead of being available a-priori. • They bear DSMS Database Management System • ad-hoc queries, data manipulation tasks • insertions, updates, deletions of single row or groups of rows Data Stream Management System • continuous queries •

2 GB. If the size of an archive file is more than 2 GB, you could upload it to a distributed file system and then use the path in the command line option -pyarch. • Mix use of the above options You could pyflink-docs, Release release-1.15 1.1.1.5 Kubernetes Kubernetes is a popular container-orchestration system for automating computer application deployment, scaling, and management. This page shows you how DataStream Connectors. [8]: from pyflink.common import Encoder from pyflink.datastream.connectors.file_system import FileSink, RollingPolicy def split(s): splits = s[1].split('|') for sp in splits: (continues

2 GB. If the size of an archive file is more than 2 GB, you could upload it to a distributed file system and then use the path in the command line option -pyarch. • Mix use of the above options You could pyflink-docs, Release release-1.16 1.1.1.5 Kubernetes Kubernetes is a popular container-orchestration system for automating computer application deployment, scaling, and management. This page shows you how DataStream Connectors. [8]: from pyflink.common import Encoder from pyflink.datastream.connectors.file_system import FileSink, RollingPolicy def split(s): splits = s[1].split('|') for sp in splits: (continues

no more delayed events will arrive. • Watermarks provide a logical clock which informs the system about the current event time. http://streamingbook.net/fig/2-9 Vasiliki Kalavri | Boston University with a timestamp T indicates that all subsequent records should have timestamps > T. Watermark properties 14 Vasiliki Kalavri | Boston University 2020 Watermarks are essential to both event-time windows

broadcast join • What does performance depend on? • input data, intermediate data • operator properties • How can we estimate the cost of different strategies? • before execution or during runtime

latency constraints that can tolerate approximate results. Slow down the flow of data: • The system buffers excess data for later processing, once input rates stabilize. • Requires a persistent process of discarding data when input rates increase beyond system capacity. • Load shedding techniques operate in a dynamic fashion: the system detects an overload situation during runtime and selectively streams with known arrival rates C: system processing capacity H: headroom factor, i.e. a conservative estimate of the percentage of resources required by the system at steady state Load(N(I)): the load

....................................................................................... 22 4.14 System Resources....................................................................................... is processed by Apache Flink, which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for various reasons including the TaskManager (in case of a containerized setup), or by providing more TaskManagers. In general, a system already running under very high load during normal operations, will need much more time to catch-up

& reconfiguration ??? Vasiliki Kalavri | Boston University 2020 • To recover from failures, the system needs to • restart failed processes • restart the application and recover its state 2 Checkpointing and all required metadata, such as the application’s JAR file, into a remote persistent storage system • Zookeeper also holds state handles and checkpoint locations 5 JobManager failures ??? Vasiliki Vasiliki Kalavri | Boston University 2020 12 • Detect environment changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions