CurveFS client 向上提供两层接口，分别是© XXX Page 3 of 11 对接fuse，提供通用文件系统接口。对于fuse接口，先前进行了一些调研，见FUSE调研 提供lib库，提供对接分布式数据库接口，这一部分，可参考polarfs的接口，如下图所示。 根据讨论，我们首先对接fuse的lowlevel operators，对于数据库的lib库接口，后续可以在此基础上再做一层对接。lowlevel +readdirplus +copy_file_range +lseek 关键接口分析 init void (*init) (void *userdata, struct fuse_conn_info *conn); 根据挂载信息，从mds获取文件系统信息（或superblock），块分配器（bitmap）和root inode所在的copyset、 metaserver block），缓存到client端。 destroy void (*destroy) (void *userdata); 清理init缓存的文件系统信息。 lookup void (*lookup) (fuse_req_t req, fuse_ino_t parent, const char *name); 根据parent inode id和name从denty缓存中找到对应的denty结构； 如果dentry缓存

ds/curvefs_mds wanghai+ 2642837 2589230 0 13:47 pts/156 00:00:00 ./bazel-bin/curvefs/src/client/fuse_client -f -o volume=/fs -o user=test -o conf=./curvefs/conf/curvefs_client.conf /tmp/fsmount 问题1：root用户无法访问挂载目录 4 of 33 查阅资料发现这是fuse的一种安全策略，默认是只有filesystem owner拥有该文件系统的访问权限，如果想要其他用户有权访问，需要在挂载参数中指定‘-o allow-root’ 或'-o allow-other'以允许相应用户有权访问该文件系统，如果挂载者不是root还需要在/etc/fuse.conf（/usr/local/etc/fuse.conf）中增加配置项“use 配置项是无值的）。详见libfuse官方文 档：https://github.com/libfuse/libfuse#security-implications # The file /etc/fuse.conf allows for the following parameters: # # user_allow_other - Using the allow_other mount option

: 用于生产临时的无名的普通文件，pathname指定一个目录。 O_TMPFILE libfuse open void(* fuse_lowlevel_ops::open)(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)© XXX Page 6 of 23 Open a file Open flags are available file is opened. See fuse_file_info structure in <fuse_common.h> for more details. If this request is answered with an error code of ENOSYS and FUSE_CAP_NO_OPEN_SUPPORT is set in fuse_conn_info.capable, and release will also succeed without being sent to the filesystem process. Valid replies: fuse_reply_open fuse_reply_err Parameters req request handle ino the inode number fi file information© XXX Page

⽀持低延迟的⽂件数据访问Curve⽂件系统⾯临的问题 • ⽤户态实现 • 稳定性/可靠性⾼ • 容易更新及维护 • 基于FUSE提供POSIX兼容⽂件接⼝ • 问题 • 相对kernel⽂件系统的实现(ext4, xfs)性能 差异⼤，延迟⾼FUSE⽂件IO读写流程 • 场景1 pytorch example word_language_model • LOOKUP inode 关闭⽂件时会发送FLUSH请求和RELEASE请求FUSE⽂件IO读写流程FUSE的IO路径及瓶颈分析 • 对⽐测试 • ⽂件访问测试直接访问ext4 • 通过FUSE访问passthrough_ll底层ext4 • 内核调⽤延迟测试 • 与FUSE Daemon通讯120us左右，FUSE Daemon⼤概10us以内 • 瓶颈在/dev/fuse通讯开销基于FUSE可能的优化点 • 降低内核与libfuse通讯延迟 cache？实现POSIX兼容API途径及问题 • 基于FUSE的实现 • curve / ceph / gluster • LD_PRELOAD重载⽂件系统系统调⽤ • vpp / f-stack / DirectFUSE • Kernel版本实现 • BentoFS 基于rust的实现采⽤LD_Preload⽅式瓶颈分析 • 环境 • FUSE daemon使⽤ passthrough_ll

分布式系统，参考并设计解决上述遗留问题。 当前删除接口代码如下：© XXX Page 3 of 15 CURVEFS_ERROR FuseClient::RemoveNode(fuse_req_t req, fuse_ino_t parent, const char *name) { Dentry dentry; CURVEFS_ERROR ret = dentryMa 指的是文件的访问计数。当文件/目录被打开时， ，该文件/目录仍然可以被打开的进程访问，不会造成崩溃或报错，我们的curvefs也需要实现 即使文件/目录已经被另一个进程删除了（nlink==0） 这样的语义。 这部分内容在fuse的相关接口中也有描述如下： /** * Forget about an inode * * This function is called when the kernel removes internal caches. * * The inode's lookup count increases by one for every call to * fuse_reply_entry and fuse_reply_create. The nlookup parameter * indicates by how much the lookup count should be

(“cache-based”) Representative system Code optimization strategy Hand-optimize key loops in assembly Fuse loops to amortize latency & maximize instruction-level parallelism Maximize data reuse & minimize (“cache-based”) Representative system Code optimization strategy Hand-optimize key loops in assembly Fuse loops to amortize latency & maximize instruction-level parallelism Maximize data reuse & minimize (“cache-based”) Representative system Code optimization strategy Hand-optimize key loops in assembly Fuse loops to amortize latency & maximize instruction-level parallelism Maximize data reuse & minimize

consumption pattern to automatically fuse computation. 46The Big Idea: Enrich function interfaces with data production and consumption pattern to automatically fuse computation. 47 Only function signatureThe Big Idea: Enrich function interfaces with data production and consumption pattern to automatically fuse computation. 48 Compute independent subsetsFern! 49Fern! 50 manya227 June 2024 Pipeline pipeline({ model complicated dependencies BLUR X BLUR Y INPUT Fuse TilesEnough to model complicated dependencies Overlapping Inputs BLUR X INPUT BLUR Y Fuse TilesEnough to model complicated dependencies Unfused

bubblewrap buildbox-worker WORKER FARM BuildBox-CASD (Local Caching Proxy) BuildBox-Fuse (FUSE filesystem) + Distributed Cache + Distributed Build:© 2024 Bloomberg Finance L.P. All rights

write operation. CurveFS architecture Curvefs implements the file system interface based on FUSE. 1. Fs-meta Cluster is used to manage the inode and dentry metadata of files. The architecture metadata scalability is very good in this way. 2. Fs-data cluster is used to store file data. Curve-fuse Supports Object storage by S3 apis and CurveBS CurveBS performance considerations 1. CurveBS

01 对接 firecracker cilium openebs 等技术 无性能损失的网络层计量 与隔离 rust 自研分布式文件系统 sealfs 直接对接 rustvmm 绕 开 fuse 02 client manger manger metadata data data metadata metadata Data and metadata store in different 不在磁盘级别提供高可用 所以高可用交给 KB rocketmq 这些应用自身去解决 业务起的有状态容器可用 sealfs 文件系统 sealfs 直接从容器拦截 IO，对接 sealfs 分布式文件系统，避免 fuse 用户态内核态反复横跳 在 Sealos 上使用 GPU 在 Sealos 上利用 Cilium + BPF 实现流量统计 Slide source credit to: How to Make