Services built for S3 – Object store workloads IMPALA + OZONE Featuring FSO Buckets 30 © 2022 Cloudera, Inc. All rights reserved. IMPALA + OZONE • Impala: SQL engine built to run in Hadoop clusters will store Impala’s data in Ozone instead of HDFS 31 © 2022 Cloudera, Inc. All rights reserved. IMPALA-9400: IMPALA OZONE SUPPORT Jira Description IMPALA-10212 ofs support in Impala IMPALA-9448 Test encryption IMPALA-10213 Support data locality of Impala daemons on Ozone IMPALA-10214 Support file handle cache for Ozone 32 © 2022 Cloudera, Inc. All rights reserved. CHOOSING BUCKET TYPE • Impala has native

Ozone and how it scales • Why NVME is important for Ozone for scaling • Benefits of using NVME • Impala performance results from NVME clusters • Write path improvements results from NVME clusters • Summary measure network saturation when using S3 • Impala TPCDS benchmark • Ratis streaming performance tests How much does disk read cost with NVME? Impala TPCDS Why Impala and Ozone? • Data Warehouse is the most most common use case. ($$$) • Impala historically optimized on HDFS -> what will it do on Ozone Software under test CDP Private Cloud Base 7.1.8 + • IMPALA-11457 Fix regression with unknown disk

Hadoop 富二代 46 / 74 Parallel Processing： Tez Spark ... User Interface： Hue SQL on Hadoop： Impala Presto Drill/Dremel/BigQuery ... Data Collector： Flume Chukwa Scribe ... Machine Learning： Mahout Cloudera 主導 Online Demo：http://demo.gethue.com/ 50 / 74 Hue - Interactive SQL & Dashboard 51 / 74 Impala - Real-Time Queries in Hadoop Cloudera 主導，做了兩年才在 2012 年正式發表 支援 HDFS/HBase 的 Distributed Parallel MapReduce，直接透過 In-Memory Process 來處理 Compliant with ANSI-92 SQL Standard，所以透過 Cloudera ODBC Driver for Impala，就可以跟既有的 BI/DW 工具整合 52 / 74 Presto Facebook 主導，2012 年秋天開始發展，2013 年春天開始推 廣，作為 Facebook Data Warehouse

Ozone – 使⽤场景 #1 HDFS (300M FILES) AI/ML HIVE/IMPALA/SPARK KAFKA / FLINK 计算 OZONE (2 BILLION Objects) AI/ML HIVE/IMPALA/SPARK KAFKA / FLINK 计算 OTHER WORKLOADS OTHER WORKLOADS 业务价值 • 集约化的⼀套存储来⾯向不同的业务负载 • 更易于运维的控制⾯ • 只需要⼀个运维团队⽽不是多个 运维价值 OZONE STORAGE AI/ML HIVE/IMPALA/ SPARK KAFKA / Flink 计算 数据科学 数据仓库 S3 应⽤ S3 API OTHER WORKLOADS ⽬录 • Apache Hadoop HDFS⾯临的问题

流处理 Spark Streaming Flink Storm 实时计算(原流计算) EMR(开源流计算组件） 分析型数据存储 数据仓库： GreenPlum/Impala/Presto/Hive NoSQL：Hbase 数据仓库：MaxCompute/ Hologres/分析 型数据库 NoSQL:云数据库 Hbase 版/表格存储 分析与报表 Cloud MaxCompute 解决方案 16 MapReduce MaxCompute MR Apache Spark MaxCompute Spark 交互式分析 Impala Presto Hawk GreenPlum 等交互式分析 MaxCompute Lightning，提供只读的交互式查 询服务 图计算 Spark GraphX MaxCompute 解决方案 25  网络环境（私有网络、经典网络、VPC 专）  有无专线  常用组件（Hive、Spark、Storm、HBase、Flink、Kafa、Impala、Sqoop、Kylin、Flume）  机器配置（CPU 核数、内存大小）  数据量及存储类型  作业量及作业类型（SQL 脚本上传）  调度系统及周期（Pipeline

following configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[[sparksql]]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

following configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[[sparksql]]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

following configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[[sparksql]]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

following configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[[sparksql]]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

following configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[[sparksql]]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes