2.1 弹性分布数据集（RDD） RDD 是 Spark 的最基本抽象,是对分布式内存的抽象使用，实现了以操作本地集合的方式 来操作分布式数据集的抽象实现。RDD 是 Spark 最核心的东西，它表示已被分区，不可变的 并能够被并行操作的数据集合，不同的数据集格式对应不同的 RDD 实现。RDD 必须是可序 列化的。RDD 可以 cache 到内存中，每次对 RDD 数据集的操作之后的结果，都可以存放到 交互式数据挖掘来说，效率提升比较大。 1.2.2 RDD 的转换与操作 对于 RDD 可以有两种计算方式：转换（返回值还是一个 RDD）与操作（返回值不是一个 RDD） 1. 转换(Transformations) (如：map, filter, groupBy, join 等)，Transformations 操作是 Lazy 的，也就是说从一个 RDD 转换生成另一个 RDD 的操作不是马上执行，Spark 操作会返回结果或把 RDD 数据写 到存储系统中。Actions 是触发 Spark 启动计算的动因。 1.2.3 血统（Lineage） 利用内存加快数据加载,在众多的其它的 In-Memory 类数据库或 Cache 类系统中也有实 现，Spark 的主要区别在于它处理分布式运算环境下的数据容错性（节点实效/数据丢失）问 题时采用的方案。为了保证 RDD 中数据的鲁棒性，RDD 数据集通过所谓的血统关系(Lineage)

up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations. • Discretized Stream Processing (DStream) 7 / 79 Spark Streaming ▶ Run a streaming up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations. • Discretized Stream Processing (DStream) 7 / 79 Spark Streaming ▶ Run a streaming up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations. • Discretized Stream Processing (DStream) 7 / 79 DStream (1/2) ▶ DStream: sequence

ensorFlow gRPC server） • Worker通过读取RDD获取训练样本 • RDD的数据cache到内存或者磁盘供多次迭代训练使用 SparkFlow介绍 SparkFlow与TensorFlow on Yarn对比：� SparkFlow TensorFlow on Yarn 通过RDD读取训练样本数据，关心 文件存储格式 直接读取HDFS数据，不关心文件存

Pipeline 中的 mapPartitions 在进行下一部分讲解之前，我们先简要介绍一下 Spark 的懒执行机制。Spark 的算 子分为 Action 和 Transformation 两大类。RDD 的依赖关系构成了数据处理的有 向无环图 DAG。只有当 Action 算子出现时，才会执行 Action 算子与前面一系列 Transformation 算子构成的 DAG。Spark 还会根据 为例，最后 runTask() 方法调用了 rdd.iterator()。 ResultTask.scala override def runTask(context: TaskContext): U = { ...... // 源码缩略不进行展示：初始化一些需要的对象 val (rdd, func) = ser.deserialize[(RDD[T], (TaskContext, Iterator[T]) getCurrentThreadCpuTime - deserializeStartCpuTime } else 0L // 这里的 func() 调用了 rdd.iterator() func(context, rdd.iterator(partition, context)) } 而 RDD 的 iterator 方法的源码如下，其调用逻辑最终都会进入 computeOrRead- Checkpoint 方法，若没有

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the custom RDD resource profile. 1.8.0 spark.sql.finalWrit eStageExecutorCor es fallback spark.exec utor.cores Specify the executor core request for final write stage. It would be passed to the RDD resource memo ry Specify the executor on heap memory request for final write stage. It would be passed to the RDD resource profile. 1.8.0 spark.sql.finalWrit eStageExecutorMe moryOverhead fallback spark.exec utor

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the custom RDD resource profile. 1.8.0 spark.sql.finalWrit eStageExecutorCor es fallback spark.exec utor.cores Specify the executor core request for final write stage. It would be passed to the RDD resource memo ry Specify the executor on heap memory request for final write stage. It would be passed to the RDD resource profile. 1.8.0 spark.sql.finalWrit eStageExecutorMe moryOverhead fallback spark.exec utor

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the custom RDD resource profile. 1.8.0 spark.sql.finalWrit eStageExecutorCor es fallback spark.exec utor.cores Specify the executor core request for final write stage. It would be passed to the RDD resource memo ry Specify the executor on heap memory request for final write stage. It would be passed to the RDD resource profile. 1.8.0 spark.sql.finalWrit eStageExecutorMe moryOverhead fallback spark.exec utor

吞吐 TB-PB TB-PB 未经⼤大规模⽣生产验证 性能 ⼀一般(分钟⼩小时级别) 快(秒级) 优秀 x2 稳定性 好 ⼀一般 已在阿⾥里里内部验证 API 差(MR) 最丰富 (RDD/DataSet/DataFrame) Python/Scala/R/Java 丰富 (TableAPI) Scala/Java SQL HiveSQL SparkSQL ANSI SQL

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the enabled false When true, make final wr RDD resource profile. spark.sql.finalWriteStageExecutorCores fallback spark.executor.cores Specify the executor core passed to the RDD resour spark.sql.finalWriteStageExecutorMemory

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the enabled false When true, make final wr RDD resource profile. spark.sql.finalWriteStageExecutorCores fallback spark.executor.cores Specify the executor core passed to the RDD resour spark.sql.finalWriteStageExecutorMemory