MaterializeMySQL Database engine in ClickHouse WinterZhang(张健) About me • Active ClickHouse Contributor • MaterializeMySQL Database Engine • Custom HTTP Handler • MySQL Database Engine • BloomFilter Skipping https://github.com/zhang2014 MySQL Table Engine • Mapping to MySQL table • Fetch table struct from MySQL • Fetch data from MySQL when execute query MySQL Database Engine • Mapping to MySQL database • Database Engine • Mapping to MySQL database • Consume MySQL BINLOG and store to MergeTree • Experimental feature (20.8, recommend latest stable version) MaterializeMySQL Database Engine MaterializeMySQL

最先将SQL和Map-Reduce的功能整合到统一的数 据处理框架中 • 可以帮助企业采集所有数据，并在竞争中获得出色 的洞察力 41 Parallel Dataflow Engine • General purpose parallel-dataflow engine natively executes SQL & MapReduce • Optimized for commodity compute or FC connected storage Local compute Local storage gNet Interconnect Greenplum Parallel Dataflow Engine gNet Software Interconnect • Supercomputing-based “soft switch” interconnect • Utilizes commodity Highly scalable protocol supports 1000s of nodes gNet Software Interconnect SG (Scatter/Gather) Streaming 1) Scatter Phase • No need for dedicated loading servers • Performance scales with number of

dependency • Job Open Ecosystem – Unify job api for extension – Support rich job type lib, such as dataflow, script, HTTP, file, big data – Focus business SDK, can work with Spring IOC • Admin Console to implement the job interface to realize business logic. Typical representatives: Simple type, Dataflow type. Type-based Jobs only need to provide the type name, developers do not need to implement the running on the job server, etc. ElasticJob provides two class‐based job types which are Simple and Dataflow; and also provides a type‐based job which is Script. Users can extend job types by implementing

ElasticJob 的作业可划分为基于 class 类型和基于 type 类型两种。 Class 类型的作业由开发者直接使用，需要由开发者实现该作业接口实现业务逻辑。典型代表：Simple 类 型、Dataflow 类型。Type 类型的作业只需提供类型名称即可，开发者无需实现该作业接口，而是通过外 置配置的方式使用。典型代表：Script 类型、HTTP 类型。 5.5. 作业开放生态 20 Apache getShardingTotalCount(), getShardingItem() 等方法分别获取分片总数，运行在本作业服务 器的分片序列号等。 ElasticJob 目前提供 Simple、Dataflow 这两种基于 class 的作业类型，并提供 Script、HTTP 这两种基于 type 的作业类型，用户可通过实现 SPI 接口自行扩展作业类型。 简单作业 意为简单实现，未经任何封装的类型。需实现 processData(ShardingContext shardingContext, List data) { // process data // ... } } 流式处理 可通过属性配置 streaming.process 开启或关闭流式处理。 如果开启流式处理，则作业只有在 fetchData 方法的返回值为 null 或集合容量为空时，才停止抓取，否 则作业将一直运行下去；如果关闭流式处

Greenplum-Spark Connector Greenplum-Informatica Connector Greenplum-Kafka Integration Greenplum Streaming Server Data Direct ODBC/JDBC Drivers Greenplum PostGIS Extension gpcopy utility for copying or management) Related Products Tanzu Greenplum Command Center Tanzu Greenplum Text Tanzu Greenplum Streaming Server Tanzu Greenplum Connector for Apache Spark Tanzu Greenplum Connector for Apache NiFi Tanzu the Greenplum 5 packages plus the addition of database/role/language commands and the Greenplum Streaming Server command utilities. Refer to Greenplum Client and Loader Tools Package for installation and

Greenplum-Spark Connector Greenplum-Informatica Connector Greenplum-Kafka Integration Greenplum Streaming Server Support for Greenplum-sourced or enhanced contrib modules: advanced_password_check diskquota VMware Tanzu Greenplum v6.20 Documentation VMware, Inc. 131 Tanzu Greenplum Text Tanzu Greenplum Streaming Server Tanzu Greenplum Connector for Apache Spark Tanzu Greenplum Connector for Apache NiFi Tanzu the Greenplum 5 packages plus the addition of database/role/language commands and the Greenplum Streaming Server command utilities. Refer to Greenplum Client and Loader Tools Package for installation and

Greenplum Connector for Apache Spark VMware Greenplum Connector for Informatica VMware Greenplum Streaming Server (Kafka, RabbitMQ, S3) Support for Greenplum-sourced or enhanced contrib modules: advanced_password_check Greenplum Disaster Recovery VMware Greenplum Platform Extension Framework (PXF) VMware Greenplum Streaming Server VMware Greenplum Text VMware Greenplum Upgrade Virtual Memory Calculator (for resource upgrade instructions. VMware Greenplum Streaming Server (GPSS) version 1.10.4 is included, which includes changes and bug fixes. Refer to the VMware Greenplum Streaming Server Documentation for more information

Greenplum-Spark Connector Greenplum-Informatica Connector Greenplum-Kafka Integration Greenplum Streaming Server Support for Greenplum-sourced or enhanced contrib modules: advanced_password_check diskquota Center Tanzu Greenplum Text VMware Greenplum 6 Documentation VMware, Inc 131 Tanzu Greenplum Streaming Server Tanzu Greenplum Connector for Apache Spark Tanzu Greenplum Connector for Apache NiFi Tanzu distributed tables. VMware Greenplum Streaming Server (GPSS) version 1.9.0 is bundled, which includes new features, changes, and bug fixes. Refer to the Greenplum Streaming Server Documentation for more information

Greenplum-Spark Connector Greenplum-Informatica Connector Greenplum-Kafka Integration Greenplum Streaming Server Support for Greenplum-sourced or enhanced contrib modules: advanced_password_check diskquota Copy Utility VMware Greenplum 6 Documentation VMware, Inc. 137 Greenplum Upgrade Greenplum Streaming Server Greenplum Platform Extension Framework (PXF) Greenplum Text Greenplum Connector for Apache distributed tables. VMware Greenplum Streaming Server (GPSS) version 1.9.0 is bundled, which includes new features, changes, and bug fixes. Refer to the Greenplum Streaming Server Documentation for more information