Performance of Apache Ozone on NVMe Wei-Chiu Chuang (jojochuang) Ritesh Shukla (kerneltime) Agenda • Overview of how Ozone and how it scales • Why NVME is important for Ozone for scaling • Benefits improvements results from NVME clusters • Summary • Questions Ozone Architecture Why does Ozone Scale? Separation of concerns Ozone Manager Ozone Client Datanode Storage Container Manager Recon Foreground Background No background load Datanode Datanode Datanode hadoop-hdds hadoop-ozone Scale out No foreground load Why does Ozone Scale? Aggregation via containers Chunks Chunks Chunks Blocks User Chunks

Apache Ozone Erasure Coding(EC) The Modern Big Data Object Store with More Than 50% Storage Space Savings Uma Maheswara Rao Gangumalla Sr. Engineering Manager, Cloudera Inc Stephen O’Donnell Sr. Staff at Cloudera ❏ Apache Software Foundation Member ❏ More than 10 years with Apache Projects. ❏ Apache Hadoop Project Management Committee(PMC) Member ❏ Apache Ozone PMC Member ❏ Apache Incubator PMC chair - 2021, 2022 What is Ozone? ➢ Apache Ozone is a distributed, scalable, and high performance object store ➢ Ozone is designed and optimized for Big Data workloads. ➢ Ozone can scale up to billions

What’s new in Apache Ozone 1.3 陈怡 Apache Ozone PMC 主席 3 ⽬录 I. Ozone 构架 II. Ozone 1.3 新功能 III. 未来展望 4 Ozone 构架 5 Ozone 1.3 新功能 I. 纠删码(Erasure coding) II. 系统均衡器(Container Balancer) III RS(6,3) 3 67% EC RS(10, 4) 4 71% EC RS(3,2) 2 60% 以计算为代价，满⾜数据可靠性的同时， 降低数据存储成本 数据可靠性 vs. 存储效率 7 Ozone条带纠删码 I. 物理块：每个DN磁盘上的数据块，默认256MB II. 逻辑EC块：满⾜EC策略的⼀个⽤户数据块。例如RS-3-2，⼀个逻辑块3*256MB⼤⼩ III. 条带：条带的默认粒度1MB，可配置 0 11 Ozone⽀持的纠删码策略 I. 内建⽀持的策略 I. RS-3-2-1024K II. RS-6-3-1024K III.XOR-2-1-1024K II. 可定制新的策略 III.策略设置⽀持 I. 全局策略设置 II. 桶级别策略设置 III.对象/⽂件级别策略设置 12 Container Balancer 时机 I. 新的节点加⼊Ozone集群 II. 删除⼤量数据后

Apache Ozone 的最近进展和实 践分享 刘岩 陈怡 2022.07.29 ⽬录 • Apache Hadoop HDFS⾯临的问题 • Apache Ozone介绍 • Apache Ozone适⽤场景 • Apache Ozone的最近进展 • Apache Ozone的实践分享 ⼤数据存储的需求 能否提供⾼并发读取和写⼊ 是否兼容主流API，如HDFS/S3 ⽆法接受私有化 的数据存储系统 公有云的对象存储服务 ⽆法在线下部署 ⽬录 • Apache Hadoop HDFS⾯临的问题 • Apache Ozone介绍 • Apache Ozone适⽤场景 • Apache Ozone的最近进展 • Apache Ozone的实践分享 Apache Ozone • Ozone是 ⼀个分布式的KV对象存储 可扩展⾄数⼗亿个对象，从⽽对云原⽣类的应⽤更友好 ⽤更友好 强⼀致性 与HDFS 和 S3 API兼容 可在存储密集型设备中部署进⽽极⼤的减少设备开⽀ Apache Ozone – 数据存储的路径设计 Ozone的存储路径为 volumes, buckets, 和 keys. Volumes 类似与⽤户账号. 只有Admin 可以创建或删除Volumes Buckets 类似与S3 的 Buckets， ⼀个Buckets中可以包含任意多个Key，但不能包含其

/ 51 Confidential—Restricted Nov 10, 2022 Ozone User Group Summit / 51 2 Confidential—Restricted / 51 THE HYBRID DATA COMPANY We believe that data can make what is impossible today, possible tomorrow Lakehouse Data Fabric Data Mesh SDX Multi-cloud & on-premises data management and analytics Ozone / 51 5 Confidential—Restricted BIG DATA STORAGE REQUIRES ... Can it handle large workloads Does Security Encryption Is the data protected at rest and in-transit? / 51 7 Confidential—Restricted Apache Ozone Scalable, redundant distributed object store Designed for data applications to store structured

Ozone：Hadoop 原生分布式对象存储 Spark大数据博客 - https://www.iteblog.com Ozone：Hadoop 原生分布式对象存储 Hadoop 社区推出了新一代分布式Key-value对象存储系统 Ozone，同时提供对象和文件访问的接 口，从构架上解决了长久以来困扰HDFS的小文件问题。本文作为Ozone系列文章的第一篇，抛个 砖，介绍Ozone的产生背景，主要架构和功能。 。 基于这些背景，Hadoop 社区推出了新的分布式存储系统 Ozone，从构架上解决这个问题。 Ozone的设计原则 Ozone 由一群对大规模Hadoop集群有着丰富运维和管理经验的工程师和构架师设计和实现。他 们对大数据有深刻的洞察力，清楚的了解HDFS的优缺点，这些洞察力自始自终影响了Ozone的设 计和实现。Ozone的设计遵循一下原则： / 10 Ozone：Hadoop 原生分布式对象存储 Spark大数据博客 - https://www.iteblog.com 强一致性 构架简洁性： 当系统出现问题时，一个简单的架构更容易定位，也容易调试。Ozone尽可能的保持架构 的简单，即使因此需要可扩展性上做一些妥协。但是在Ozone在扩展性上绝不逊色，目标 是支持单集群1000亿个对象。 构架分层 Ozone 采用分层的文件系统。Namespace

Apache OFBiz® The Apache OFBiz Project Version Trunk Table of Contents 1. System requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . License License Apache 2.0 Apache 2.0 [badge] If you are reading this file in AsciiDoc format you may want to see it at HTML or PDF format Welcome to Apache OFBiz! A powerful top level Apache software project reading section. 3 3. Security If you find a security issue, please report it to: security @ ofbiz.apache.org. Once proper mitigations to the security issues are complete the OFBiz team will disclose this

解读Apache 解读Apache 演讲者 Craig Russell Justin Mclean 姜宁 本演讲包含了 Bertrand Delacretaz, Roman Shaposhnik 以及其 它ASF贡献者的⼯作 Craig Russell介绍 Craig Russell介绍 软件架构师 Object Data Management Group Java Data Objects (MySQL) Apache Committer - from 2005 Apache Member - from 2007 Apache Secretary - from 2010-2019 Apache 孵化器管理委员成员 Apache 董事会主席 Apache 软件基⾦会: Apache 软件基⾦会: 世界上最⼤的开源基⾦会 Apache基⾦会的使命 Apache基⾦会的使命 Apache软基⾦⾦会 Apache软基⾦⾦会 (ASF) 是⼀个根据美国法案 501(c)(3)注册公益性组织。 ASF的使命 是通过向加⼊ASF的志同道合的软件项⽬社区提供服务，为公众提供开源软件。 ASF提供了⼀个独⽴于任何公司影响⼒的中⽴空间，保证其中的 项⽬可以在商业友好Apache许可证2.0下繁荣发展， 为公众利益 创建开源软件。 公开成⽴于1999年的公益组织 公开成⽴于1999年的公益组织 使命:

Apache Explained Presented by Roman Shaposhnik Craig Russell Justin Mclean Including original work of Bertrand Delacretaz and other amazing ASF contributors ➔ Engineer and Entrepreneur ◆ Founder Pivotal Inc. ◆ Ex- Sun microsystems, Huawei, Yahoo!, Cloudera ➔ Apache Committer - from 2010 ➔ Apache Member - from 2014 ➔ Apache Board Member and VP of Legal Affairs - from 2018 ➔ Member, Incubator Management Committee, Bigdata Who is Roman Shaposhnik? Apache Software Foundation: The World’s Largest Open Source Foundation The ASF’s Mission The Apache Software Foundation (ASF) is a US 501(c)(3) charitable

Apache Explained Apache Explained Presented by Craig Russell Justin Mclean Willem Jiang Including original work of Bertrand Delacretaz, Roman Shaposhnik and other amazing ASF contributors Who is Craig Jones (MySQL) Apache Committer - from 2005 Apache Member - from 2007 Apache Secretary - 2010 - 2019 Member, Incubator Project Management Committee Chairman, Apache Board of Directors Apache Software Foundation: Foundation: Apache Software Foundation: The World’s Largest Open The World’s Largest Open Source Foundation Source Foundation The ASF’s Mission The ASF’s Mission The Apache Software Foundation (ASF)