Parquet and DataFusion 朱霜 2023.06.18 1. Introduction 2. Duo - Observability duet: Logging and Tracing • What is Duo? • How does it work? 3. Apache Arrow, Parquet and DataFusion • A brief introduction introduction to Arrow, Parquet, and DataFusion • How does Duo store and query log, span data? 4. The vision of Duo Content Introduction • ID: Folyd • GitHub: @folyd • 博客: https://folyd.com • ⼯作：字节跳动 Logging and Tracing Duo - How does it work? Duo - How does it work? Apache Arrow, Parquet, and DataFusion Apache Arrow • Created by Wes McKinney, creator of Pandas (2016) • A language-independent

1.3 基于 DataFusion 的高性能查询引擎 9 SQL Result DataFusion Parser DataFusion SqlToRel DataFusion Logical Optimizer DataFusion Physical Planner DataFusion Logical Optimizer DataFusion Scheduler Scheduler DataFusion Execution Injectable parser rules Injectable transform rules & Customizable logical node Injectable logical opt rules Injectable transform rules & Customizable physical rules Pluggable Cascade Optimizer Pluggable Scheduler Logical transform Physical transform 1.4基于DataFusion的高性能查询引擎 • 扩展数据源 • 扩展 SQL 语句 • 扩展流处理引擎 • 扩展优化规则 • 扩展时序函数 1.5 分布式 1. Shared nothing 2.

Ballista is a distributed SQL query engine powered by the Rust implementation of Apache Arrow and DataFusion. It’s mainly for interactive queries of low latency. • Support DAG and fault tolerance • Support snowflake.pdf • Apache Arrow https://arrow.apache.org/ • Apache Arrow DataFusion https://github.com/apache/arrow-datafusion • Apache Arrow Ballista https://github.com/apache/arrow-ballista Thank

chumsky, nom, winnow? Still exploring ? 03 the Engine Which way to rewrite? Apache Arrow & DataFusion Diff between models REWRITE Around 1/7 of Prometheus is PromQL A straightforward 3-layer exceeded 1/10 of Golang’s With some key infrastructure like WebAssembly WEBASSEMBLY waynexia / datafusion-playground INTEROPERABILITY @coprocessor(returns=['value']) def add(**params) -> vector[i64]: