commonly used Python virtual environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink jobs to YARN for execution that is, pre-install a few commonly used Python virtual environments on the cluster nodes and use custom Python virtual environment when there are some special requirements. 1.1. Getting Started 9 pyflink-docs

commonly used Python virtual environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink jobs to YARN for execution that is, pre-install a few commonly used Python virtual environments on the cluster nodes and use custom Python virtual environment when there are some special requirements. 1.1. Getting Started 9 pyflink-docs

University 2020 • Change parallelism • scale out to process increased load • scale in to save resources • Fix bugs or change business logic • Optimize execution plan • Change operator placement predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections

minimize disk access • scheduling Objectives • optimize resource utilization or minimize resources • decrease latency, increase throughput • minimize monetary costs (if running in the cloud) Safety • Ensure resource kinds: all resources required by a fused operator should remain available. • Ensure resource amounts: the total amount of resources required by the fused operator must be | Boston University 2020 35 Safety • Ensure resource availability: the host must have enough resources for all assigned operators • Ensure security constraints: what are the trusted hosts for each

.............................................................................. 22 4.14 System Resources................................................................................................ decreasing the number of task slots per TaskManager (in case of a Standalone setup), by providing more resources to the TaskManager (in case of a containerized setup), or by providing more TaskManagers. In general ease-1.7/monitoring/metrics.html#system-resources 10 https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html 4.14 System Resources In addition to the JVM metrics above, it

file systems, socket connections. 2. Advanced sources, e.g., Kafka, Flume, Kinesis, Twitter. 3. Custom sources, e.g., user-provided sources. 13 / 79 Input Operations ▶ Every input DStream is associated file systems, socket connections. 2. Advanced sources, e.g., Kafka, Flume, Kinesis, Twitter. 3. Custom sources, e.g., user-provided sources. 13 / 79 Input Operations - Basic Sources ▶ Socket connection quorum], [consumer group id], [number of partitions]) 15 / 79 Input Operations - Custom Sources (1/3) ▶ To create a custom source: extend the Receiver class. ▶ Implement onStart() and onStop(). ▶ Call

file:///home/user/wordcount_out 19 Flink commands Vasiliki Kalavri | Boston University 2020 Resources • Documentation • https://flink.apache.org/ • Community • https://flink.apache.org/community failures without losing any records committed to the log. Vasiliki Kalavri | Boston University 2020 Resources • Documentation • https://kafka.apache.org/ • Community • https://kafka.apache.org/contact

Scale resource allocation: • Addresses the case of increased load and additionally ensures no resources are left idle when the input load decreases. ??? Vasiliki Kalavri | Boston University 2020 Load system processing capacity H: headroom factor, i.e. a conservative estimate of the percentage of resources required by the system at steady state Load(N(I)): the load as a fraction of the total capacity

predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections

2020 input stream window assigner ... trigger evictor evaluation function result stream Custom windows 20 • Describe each component Vasiliki Kalavri | Boston University 2020 32 4 2 5 7 44 on… Vasiliki Kalavri | Boston University 2020 Advanced transformation functions used to implement custom logic for which predefined windows and transformations might not be suitable: • they provide access