Hortonworks Hadoop 簡史 5 / 74 The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It 成是一個中立的機制，方便支援各種不同 Framework。 23 / 74 YARN - Yet Another Resource Negotiator A General-Purpose Distributed Application Management Framework Data Operating System for Enterprise Hadoop 24 / 74 Resource 26 / 74 Hadoop 家族 27 / 74 Windows 家族 28 / 74 Hadoop 動物園 29 / 74 HDFS：Distributed File System MapReduce：Distributed Data Analysis Engine Avro：Language-Neutral Data Serialization System (2010-05

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 26 Multi-processing and Distributed Computing 329 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1301 68 Delimited Files 1332 69 Distributed Computing 1337 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 26 Multi-processing and Distributed Computing 329 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1301 68 Delimited Files 1332 69 Distributed Computing 1337 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 26 Multi-processing and Distributed Computing 329 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1301 68 Delimited Files 1332 69 Distributed Computing 1337 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 26 Multi-processing and Distributed Computing 332 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1354 69 Delimited Files 1385 70 Distributed Computing 1390 70.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 26 Multi-processing and Distributed Computing 333 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1352 69 Delimited Files 1383 70 Distributed Computing 1388 70.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 26 Multi-processing and Distributed Computing 332 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1351 69 Delimited Files 1382 70 Distributed Computing 1387 70.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 26 Multi-processing and Distributed Computing 332 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1351 69 Delimited Files 1382 70 Distributed Computing 1387 70.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 26 Multi-processing and Distributed Computing 332 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1343 68 Delimited Files 1374 69 Distributed Computing 1379 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 25 Multi-processing and Distributed Computing 310 25.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1114 66 Delimited Files 1139 67 Distributed Computing 1144 67.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system