Standalone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.1.1.4 YARN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1.1.5 Kubernetes 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink jobs to YARN for execution environment It requires Python 3.6 or above with PyFlink pre-installed to be available on the nodes of the YARN cluster. It’s sug- gested to use Python virtual environments to set up the Python environment. See

Standalone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.1.1.4 YARN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1.1.5 Kubernetes 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink jobs to YARN for execution environment It requires Python 3.6 or above with PyFlink pre-installed to be available on the nodes of the YARN cluster. It’s sug- gested to use Python virtual environments to set up the Python environment. See

job 2. Start job 3. Request slots 4. Allocate Container 5. Start Task Manager 6. Schedule Task YARN RM K8S RM 增量 Checkpoint 时间 全量状态 增量状态 增量 snapshot 基于 credit 的流控机制 Streaming SQL ------------------------- Dataflow Query Processor DAG & StreamOperator Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN Runtime Distributed Streaming Dataflow DataStream API Stream Processing DataSet API Batch Processing Relational Table API & SQL Relational Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN DataStream Physical 统一 Operator 抽象 Pull-based operator Push-based operator 算子可自定义读取顺序 Table API

enough slots become available. • Restart is automatic if there is a ResourceManager, e.g. in a YARN setup • A manual TaskManager re-start or a backup is required in standalone mode • The restart