file systems, socket connections. 2. Advanced sources, e.g., Kafka, Flume, Kinesis, Twitter. 3. Custom sources, e.g., user-provided sources. 13 / 79 Input Operations ▶ Every input DStream is associated file systems, socket connections. 2. Advanced sources, e.g., Kafka, Flume, Kinesis, Twitter. 3. Custom sources, e.g., user-provided sources. 13 / 79 Input Operations - Basic Sources ▶ Socket connection quorum], [consumer group id], [number of partitions]) 15 / 79 Input Operations - Custom Sources (1/3) ▶ To create a custom source: extend the Receiver class. ▶ Implement onStart() and onStop(). ▶ Call

commonly used Python virtual environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone py See Submitting PyFlink jobs for more details. 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported that is, pre-install a few commonly used Python virtual environments on the cluster nodes and use custom Python virtual environment when there are some special requirements. 1.1. Getting Started 9 pyflink-docs

commonly used Python virtual environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone py See Submitting PyFlink jobs for more details. 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported that is, pre-install a few commonly used Python virtual environments on the cluster nodes and use custom Python virtual environment when there are some special requirements. 1.1. Getting Started 9 pyflink-docs

https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/#task-chaining-and-resource-groups 4 进度和吞吐量监控 知道您的应用程序正在运行并且检查点正常工作是件好事，但是它并不能告诉您应用程序是否正在实际取得进 展并与上游系统保持同步。 4.1 吞吐量 Fli overall memory consumption of the Job- and TaskManager containers to ensure they don’t exceed their resource limits. This is particularly important, when using the RocksDB statebackend, since RocksDB allocates Flink processes alone. System resource monitoring is disabled by default and requires additional dependencies on the classpath. Please check out the Flink system resource metrics documentation9 for additional

hashing • indexing, pre-fetching • minimize disk access • scheduling Objectives • optimize resource utilization or minimize resources • decrease latency, increase throughput • minimize monetary Kalavri | Boston University 2020 28 Safety • Ensure resource kinds: all resources required by a fused operator should remain available. • Ensure resource amounts: the total amount of resources required by stateful Variations and dynamism ??? Vasiliki Kalavri | Boston University 2020 35 Safety • Ensure resource availability: the host must have enough resources for all assigned operators • Ensure security

2020 input stream window assigner ... trigger evictor evaluation function result stream Custom windows 20 • Describe each component Vasiliki Kalavri | Boston University 2020 32 4 2 5 7 44 on… Vasiliki Kalavri | Boston University 2020 Advanced transformation functions used to implement custom logic for which predefined windows and transformations might not be suitable: • they provide access

event in the last minute • Tumble windows are non-overlapping fixed-size • events every hour • Custom windows have neither fixed bounds nor fixed size • events in a period during which a user was active | Boston University 2020 User-Defined Aggregates (UDAs) Constructs that allow the definition of custom aggregations using three statement groups: • INITIALIZE: initialized local state. • ITERATE:

Time Window 2015 年阿里巴巴开始使用 Flink 并持续贡献社区 重构分布式架构 Client Dispatcher Job Manager Task Manager Resource Manager Cluster Manager Task Manager 1. Submit job 2. Start job 3. Request slots 4. Allocate

• minimize performance disruption, e.g. latency spikes • avoid introducing load imbalance • Resource management • utilization, isolation • Automation • continuous monitoring • bottleneck detection

stabilize. • Requires a persistent input source. • Suitable for transient load increase. Scale resource allocation: • Addresses the case of increased load and additionally ensures no resources are