Ozone on NVMe Wei-Chiu Chuang (jojochuang) Ritesh Shukla (kerneltime) Agenda • Overview of how Ozone and how it scales • Why NVME is important for Ozone for scaling • Benefits of using NVME • Impala Impala performance results from NVME clusters • Write path improvements results from NVME clusters • Summary • Questions Ozone Architecture Why does Ozone Scale? Separation of concerns Ozone Manager limitations, working set can be cached in memory and unused data can be destaged to disk • OM uses NVME to store RocksDBs • Future projects such as Snapshots leverage RocksDB to preserve simplicity of

0:87:00.0-nvme-1 - /dev/disk/by-path/pci-0000:88:00.0-nvme-1 optionalPaths: - /dev/disk/by-path/pci-0000:89:00.0-nvme-1 - /dev/disk/by-path/pci-0000:90:00.0-nvme-1 thinPoolConfig: 000:87:00.0-nvme-1 - /dev/disk/by-path/pci-0000:88:00.0-nvme-1 optionalPaths: - /dev/disk/by-path/pci-0000:89:00.0-nvme-1 - /dev/disk/by-path/pci-0000:90:00.0-nvme-1 thinPoolConfig: name: thin-pool-1 sizePercent: 90 overprovisionRatio:

主机之间的通信。 3.3.1. 通过 OpenStack 使用 PCI 透传向 etcd 节点提供存储 要为 etcd 节点提供快速存储以便 etcd 在大规模稳定，使用 PCI 透传将非易失性内存表达(NVMe)设备直 接传递给 etcd 节点。要使用 Red Hat OpenStack 11 或更高版本进行设置，请在存在 PCI 设备的 OpenStack 节点上完成以下内容。 1. 确定在 BIOS all_filters [pci] alias = { "vendor_id":"144d", "product_id":"a820", "device_type":"type-PCI", "name":"nvme" } 6. 在控制器上重启 nova-api 和 nova-scheduler。 7. 在 /etc/nova/nova.conf 中的计算节点上： [pci] passthrough_whitelist "product_id":"a820", "device_type":"type-PCI", "name":"nvme" } 要检索您想要透传的 NVMe 设备的 address、vendor_id 和 product_id 值，请运行： 8. 重启计算节点上的 nova-compute。 9. 将您正在运行的 OpenStack 版本配置为使用 NVMe 并启动 etcd 节点。 3.4. 使用 TUNED 配置集扩展主机

. . . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special: