CurveBS I/O processing flow Before introducing IO processing flow, we first describe the overall architecture, data organization and topology structure of CURVE. CurveBS uses the central and hardware, the need for decentralization is not particularly strong in most cases. The CurveBS core consists of three parts: 1. Metadata Server (MDS) l Manages and stores metadata information snapshots/mirrors l Support rollback from snapshot CurveBS data organization Data organization of virtual block device in CurveBS1. CurveBS maps the virtual block device to a file. For example,

Estimation of availability and reliability in CurveBS CurveBS uses the RAFT protocol to maintain consistency of stored data. It generally takes the form of 3 replicas of data. If one replica fails Time of disks is MTBF (Mean Time Before Failure). 𝐴𝐹𝑅 = 1 𝑀𝑇𝐵𝐹/（24 ∗ 365） ∗ 100 In CurveBS, the data on one disk is distributed in about 50 copysets, which means when a disk fails, up to ∗ 𝑃-) The probability of no data loss in a cluster is as follows: 1 − 𝑃 Assume that in a CurveBS cluster consisting of 1200 disks, MTBF of each disk is 1.2 million hours, and the data recovery

Distributed storage system https://www.opencurve.io/Agenda • What is Curve • Use Cases • CurveBS • Key Features • Comparing to Ceph • CurveFS • Comparing to Ceph • Current Status • RoadmapWhat is Curve • Curve is an distributed storage system • Components • Curve Block Storage (CurveBS) • CurveBS: a high performance cloud native distributed block storage • Curve File System (CurveFS) moved to OSS, and frequent data is moved to high speed storage transparentlyData backup • Curve (CurveBS, CurveFS) can backup data to remote public cloud and to on-prem OSSCurveBS • high performance

opencurve.io/Agenda • CurveBS Architecture • CurveBS Topology • CurveBS Data Organization • MetaData Server (MDS) • ChunkServer • Client • CurveBS IO processing flow • CurveBS Performance considerations Cooperate with Alibaba to support high-performance polardb for postgresql using CurveBS • Cloud-native support for CurveBS • CurveBS clusters and related monitoring services can be configured using CRDs in and on- premises environments • Use the Curve operator to install, upgrade, backup, and expand CurveBS clusters • Use Curve Discover to discover resource changes and to collect the system’s runtime

Storage (CurveBS) • CurveBS: a high performance cloud native distributed block storage • Curve File System (CurveFS) • CurveFS: a high performance cloud native file system (Based on CurveBS / S3 compatible Features for Cluster • Features for CurveBS • Features for CurveFS • Features for monitorFeature list for cluster • Install / uninstall/ upgrade and configure CurveBS/CurveFS through helm chart • upgrade for cluster • CurveBS mirroring configured with CRDs • Different Curve clusters may share MDS and ETCD server • Replication between CurveFS configured with CRDsFeature list for CurveBS • CSI plugin

1 基于SPDK的CurveBS PFS存储引擎10/17/22 2 Why ●为了减少使用cpu做内存copy，减少系统调用 ●发挥某些被操作系统屏蔽的功能，例如nvme write zero ●根据阿里《When Cloud Storage Meets RDMA》的说法 ●在100Gbps网络带宽时，内存带宽成为瓶颈 ●Intel Memory Latency Checker (MLC)测试得到的CPU内存带宽是 为什么用PFS ●对代码比较熟悉 ●找一个能管理裸盘，具有产品级可靠性的代码挺难的 ●PFS支持类POSIX文件的接口，与使用EXT4的存储引擎代码很像， 所以容易移植现有代码到PFS存储引擎 ●CurveBS对文件系统元数据的操作非常少，对文件系统的要求不高， 所以不需要元数据高性能，这方面PFS也合适10/17/22 6 对PFS的修改 ●基于阿里开源的PFS ●不再基于daemon模式，而是直接使用pfs vector的接口主要是为了与brpc的iobuf对接，iobuf由若干地址不连 续的block组成，一次IO提交可以提高效率。10/17/22 8 PFS+SPDK 的部分读写的实现 ●某些盘只支持4k单位读写，但是CurveBS支持512字节读写 ●可能存在部分写的并发冲突 ●引入并发的range lock解决冲突10/17/22 9 PFS+SPDK 的DMA支持 ●ssize_t pfs_writev_dma(int

使用廉价硬件搭建。iSCSI软件 • Client端: iscsi initiator，系统自带 • Linux open-iscsi • Windows iSCSI 发起者 • 服务器端 • 必须是CurveBS原生支持的平台，因为需要curve原生接口，目前是LinuxiSCSI target服务器 • LINUX LILO • 一般用于输出内核本地块设备 • TCMU • 作为LILO支持用户态的接口

TCP 1 32 85 2 39 86 4 46 128 8 47 170 16 67 291 32 119 378 64 237 45263 4K大小时延比较(us)64 CurveBS FIO65 现状 ●开源分支:https://github.com/opencurve/incubator-brpc.git –curve主干分支 –ucx_am当前rdma分支 ●