����������� �������� ����������� �������� Retail spark search adds MR flink Sigma Fuxi Kubernetes YARN Node ���������� �� ��������� ������� ��� ����������� Co-located�40% Seperated �10% 30% ������ Kubernetes Focus on long running service. Driving current state towards desired state with control loops YARN Focus on scheduling jobs ���������� ���� Kubernetes Container centric – bottom up. Container is is the primitive. Other primitives such as replicaset, deployment are built around containers. YARN Application centric: top down. Scheduling sequence: Queue -> user -> application -> container request

TensorFlow on Yarn 李远策 2017年4月17日 内容大纲 Ø TensorFlow使用现状及痛点� Ø TensorFlow on Yarn设计� Ø TensorFlow on Yarn技术细节揭秘� Ø 深度学习平台演进及SparkFlow介绍� 背景 坐标：360-系统部-⼤数据团队� 专业：Yarn、Spark、MR、HDFS …� 集群资源的管理（目前支持CPU、内存，需要扩展GPU 资源管理）� • 作业的统⼀管理、状态跟踪� • 资源组（Schedule Pool）的划分� • 作业进程的资源隔离� Yarn能解决什么问题：� TensorFlow on Yarn设计 • 同时支持单机和分布式TensorFlow程序� • 支持GPU资源管理和调度� • 不再需要⼿动配置CluserSpec信息，仅需要设置work 和ps的数量� TensorFlow on Yarn设计 • 支持GPU亲和性调度（提⾼通信效率）� • Web的⽅式查看作业的运⾏状况和作业日志� • 在线查看Tensorboard� • HistoryServer支持查看结束作业的日志和状态信息� • 控制已有的TensorFlow作业的迁移成本（最多改三⾏ 代码）� 扩展目标：� TensorFlow on Yarn设计 tensorflow-submit

resource with managed ACLs, e.g. Queue Access Control Lists, from cluster managers, e.g. Apache Hadoop YARN, Kubernetes (K8s) to create the Spark application; 2) a user account can only access data and metadata THREE RUN ANYWHERE Kyuubi can submit Spark applications to all supported cluster managers, including YARN, Mesos, Kubernetes, Stan- dalone, and local. The SPA policy also make it possible for you to launch you can run Spark/Flink/Trino SQL engines created by the Kyuubi on any cluster manager, including YARN, Kubernetes, Mesos, e.t.c... Or, you can manipulate data from different data sources with the Spark

resource with managed ACLs, e.g. Queue Access Control Lists, from cluster managers, e.g. Apache Hadoop YARN, Kubernetes (K8s) to create the Spark application; 2) a user account can only access data and metadata THREE RUN ANYWHERE Kyuubi can submit Spark applications to all supported cluster managers, including YARN, Mesos, Kubernetes, Stan- dalone, and local. The SPA policy also make it possible for you to launch you can run Spark/Flink/Trino SQL engines created by the Kyuubi on any cluster manager, including YARN, Kubernetes, Mesos, e.t.c... Or, you can manipulate data from different data sources with the Spark

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. 5 Metrics 16 Chapter 3. What’s Next Kyuubi, Release 1.9.0-SNAPSHOT Operation Server Session Spnego Yarn Zookeeper 3.2.3 Spark Configurations Via spark-defaults.conf Setting them in $SPARK_HOME/conf/spark-defaults spark.master=local spark.sql.adaptive.enabled=true # For a user named kent ___kent___.spark.master=yarn ___kent___.spark.sql.adaptive.enabled=false (continues on next page) 3.2. Configurations 19 Kyuubi

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. 5 Frontend Ha Kinit Kubernetes Lineage Metadata Metrics Operation Server Session Spnego Yarn Zookeeper 3.2.3 Spark Configurations 16 Chapter 3. What’s Next Kyuubi, Release 1.8.0 Via spark-defaults spark.master=local spark.sql.adaptive.enabled=true # For a user named kent ___kent___.spark.master=yarn ___kent___.spark.sql.adaptive.enabled=false # For a user named bob ___bob___.spark.master=spark://master:7077

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. 5 Frontend Ha Kinit Kubernetes Lineage Metadata Metrics Operation Server Session Spnego Yarn Zookeeper 3.2.3 Spark Configurations 16 Chapter 3. What’s Next Kyuubi, Release 1.8.0 Via spark-defaults spark.master=local spark.sql.adaptive.enabled=true # For a user named kent ___kent___.spark.master=yarn ___kent___.spark.sql.adaptive.enabled=false # For a user named bob ___bob___.spark.master=spark://master:7077

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. 5 Frontend Ha Kinit Kubernetes Lineage Metadata Metrics Operation Server Session Spnego Yarn Zookeeper 3.2.3 Spark Configurations 16 Chapter 3. What’s Next Kyuubi, Release 1.8.1 Via spark-defaults spark.master=local spark.sql.adaptive.enabled=true # For a user named kent ___kent___.spark.master=yarn ___kent___.spark.sql.adaptive.enabled=false # For a user named bob ___bob___.spark.master=spark://master:7077

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. 5 Frontend Ha Kinit Kubernetes Lineage Metadata Metrics Operation Server Session Spnego Yarn Zookeeper 3.2.3 Spark Configurations 16 Chapter 3. What’s Next Kyuubi, Release 1.8.0 Via spark-defaults spark.master=local spark.sql.adaptive.enabled=true # For a user named kent ___kent___.spark.master=yarn ___kent___.spark.sql.adaptive.enabled=false # For a user named bob ___bob___.spark.master=spark://master:7077

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. 5 visit different Hive metastore server instance. Similarly, this works for other services like HDFS, YARN too. Limitation: As most Hive configurations are final and unmodifiable in Spark at runtime, this $KYUUBI_HOME/work) # - HADOOP_CONF_DIR Directory containing the Hadoop / YARN configuration to use. # - YARN_CONF_DIR Directory containing the YARN configuration to use. # # - SPARK_HOME Spark distribution which