Hadoop 迁移到阿里云 MaxCompute 技术方案 （V2.8.5） 编写人：MaxCompute 产品团队 日 期：2019.05 Alibaba Cloud MaxCompute 解决方案 2 目录 1 概要 .................................. ............. 6 2 阿里云大数据与开源生态对比 .................................................................................................................. 7 2.1 Hadoop 及开源生态与阿里云大数据生态对比 ................. ................................................................................. 9 2.1.4 阿里云大数据与 Hadoop 生态的产品映射 ......................................................................... 9 2.2 MaxCompute

数据节点 Chunkserver 数据存储 副本一致性 • 客户端 Client 对元数据增删改查 对数据增删改查 • 快照克隆服务器MDS各个组件 MDS是中心节点，负责元数据管理、集群状态收集与调度。MDS包含以下几个部分： • Topology: 管理集群的 topo 元数据信息。 • Nameserver: 管理文件的元数据信息。 • Copyset: 副本放置策略。 • Heartbeat: 储系统中多副 本PageFile支持块设备、三副本AppendFile（待开发）支持在线对象存储、AppendECFile（待开发）支持 近线对象存储可以共存。 如上所示LogicalPool与pool为多对一的关系，一个物理pool可以存放各种类型的file。当然由于curve支持 多个pool，可以选择一个logicalPool独享一个pool。 通过结合curve的用户系统，Log 通过上述信息的定期更新，作为schedule 模块进行均衡及配置变更的依据 • 通过chunkserver定期上报copyset的copyset的epoch， 检测chunkserver的copyset与mds差异，同 步两者的copyset信息 • 支持配置变更功能，在心跳回复报文中下发mds发起的配置变更命令，并在后续心跳中获取配置 变更进度。HEARTBEAT MDS端：mds 端的心跳主要由三个部分组成：

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 7.12 Using Sprite in Stereotype You can use sprites within stereotype components #777777 } () "Data Access" as DA DA - [First Component] [First Component] ..> () HTTP : use HTTP - [Web Server] << Apache >> @enduml @startuml [AA] <> Guía de Referencia del Lenguaje PlantUML

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 7.12 Using Sprite in Stereotype You can use sprites within stereotype components #777777 } () "Data Access" as DA DA - [First Component] [First Component] ..> () HTTP : use HTTP - [Web Server] << Apache >> @enduml @startuml [AA] <> Guía de Referencia del Lenguaje PlantUML

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 7.12 Using Sprite in Stereotype You can use sprites within stereotype components ArrowFontName Impact ArrowColor #FF6655 ArrowFontColor #777777 } () "Data Access" as DA Component "Web Server" as WS << Apache >> DA - [First Component] [First Component] ..> () HTTP : use HTTP - WS @enduml

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 7.12 Using Sprite in Stereotype You can use sprites within stereotype components #777777 } () "Data Access" as DA DA - [First Component] [First Component] ..> () HTTP : use HTTP - [Web Server] << Apache >> @enduml @startuml [AA] <> Guía de Referencia del Lenguaje PlantUML

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 6.12 Using Sprite in Stereotype You can use sprites within stereotype components #777777 } () "Data Access" as DA DA - [First Component] [First Component] ..> () HTTP : use HTTP - [Web Server] << Apache >> Guía de Referencia del Lenguaje PlantUML (1.2020.23) 111 / 304 6.13 Personalización

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 6.12 Using Sprite in Stereotype You can use sprites within stereotype components #777777 } () "Data Access" as DA DA - [First Component] [First Component] ..> () HTTP : use HTTP - [Web Server] << Apache >> Guía de Referencia del Lenguaje PlantUML (1.2020.22) 100 / 292 6.13 Personalización

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 6.10 Using Sprite in Stereotype You can use sprites within stereotype components PlantUML (1.2019.4) 81 / 165 6.11 Personalización (Skinparam) 6 DIAGRAMA DE COMPONENTES HTTP - [Web Server] << Apache >> @enduml @startuml [AA] <> [BB] <> [CC] <>

"Data Access" as DA DA - [First Component] [First Component] ..> HTTP : use note left of HTTP : Web Service only note right of [First Component] A note can also be on several lines end note @enduml individuales Puedes especificar un color despues de la definición del componente. @startuml component [Web Server] #Yellow @enduml 6.10 Using Sprite in Stereotype You can use sprites within stereotype components PlantUML (1.2019.3) 81 / 157 6.11 Personalización (Skinparam) 6 DIAGRAMA DE COMPONENTES HTTP - [Web Server] << Apache >> @enduml @startuml [AA] <> [BB] <> [CC] <>