1：直接落盘，异步merge - background_pool_size 2：一个Insert Request，涉及N个分区的数据，在磁盘上就会生成N个数据目录，merge跟不上 3：一个目录，一个zxid，zookeeper集群的压力大，插入速度严重变慢 解决： 1：增大background_pool_size治标不治本 2：设置分区的时候需要思考，数据的特性需要了解 我们遇到的问题 查询过程中clickhouse-server进程挂掉 我们遇到的问题 zookeeper相关的问题 问题一：zookeeper的snapshot文件太大，follower从leader同步文件时超时 问题二：zookeeper压力太大，clickhouse表处于”read only mode”，插入失败 分析： clickhouse对zookeeper的依赖还是很重的，有大量的数据需要写到zookeeper上面，数据Part都在 zookeeper上面有个节点与之对应以及表的元数据信息等等 zookeeper上面有个节点与之对应以及表的元数据信息等等. 解决： 1：zookeeper机器的snapshot文件和log文件最好分盘存储(推荐SSD)提高ZK的响应 2：zookeeper的snapshot文件存储盘不低于1T 3：做好zookeeper集群和clickhouse集群的规划，可以多套zookeeper集群服务一套clickhouse集群 3.1：zookeeper集群的znode最好能在400w以下(这个时候snapshot文件到达2G+)

What does ClickHouse look like on Kubernetes? Shard 1 Replica 1 Zookeeper Services Zookeeper-0 Zookeeper-2 Zookeeper-1 Replica Service Load Balancer Service Shard 1 Replica 2 Shard $AWS_ELB_HOST_NAME Replication requires Zookeeper Install minimal Zookeeper in separate namespace. kubectl create ns zoons kubectl apply -f zookeeper-1-node.yaml -n zoons watch kubectl -n zoons zoons get all Note ZK node DNS name: zookeeper-0.zookeepers.zoons You can also install using helm *or* use external ZK cluster After inserting a ‘zookeepers’ clause we can add replicas apiVersion:

What does ClickHouse look like on Kubernetes? Shard 1 Replica 1 Zookeeper Services Zookeeper-0 Zookeeper-2 Zookeeper-1 Replica Service Load Balancer Service Shard 1 Replica 2 Shard $AWS_ELB_HOST_NAME Replication requires Zookeeper to be enabled Install minimal Zookeeper in separate namespace. kubectl create ns zoons kubectl apply -f zookeeper-1-node.yaml -n zoons watch kubectl kubectl -n zoons get all Note ZK node DNS name: zookeeper-0.zookeepers.zoons You can also install using helm *or* use external ZK cluster After inserting a ‘zookeepers’ clause we can add replicas

(Apache, Nginx) › Cache (Memcached) › Message Broker (Kafka, Amazon SQS) › Coordination system (Zookeeper, etcd) https://github.com/donnemartin/system-design-primer 5 / 97 Highload Architecture › Webserver (Apache, Nginx) › Cache (Memcached) › Message Broker (Kafka, Amazon SQS) › Coordination system (Zookeeper, etcd) › MapReduce (Hadoop, Spark) › Network File System (S3, HDFS) https://github.com/donnema (Apache, Nginx) › Cache (Memcached) › Message Broker (Kafka, Amazon SQS) › Coordination system (Zookeeper, etcd) › MapReduce (Hadoop, Spark) › Network File System (S3, HDFS) › Key-Value Storage (Redis,

идеальная. Репликация требует ZooKeeper 12 ZooKeeper — отдельный от ClickHouse компонент, написанный на Java, требующий тщательной настройки и отдельных серверов. ZooKeeper уходит! 13 1. Clickhouse-keeper Clickhouse-keeper — 100% совместимый с ZooKeeper по протоколу и модели данных • Compressed logs and snapshots • No issues with zxid overflow • No issues with large packets • Better memory usage • No issues

идеальная. Репликация требует ZooKeeper 12 ZooKeeper — отдельный от ClickHouse компонент, написанный на Java, требующий тщательной настройки и отдельных серверов. ZooKeeper уходит! 13 1. Clickhouse-keeper Clickhouse-keeper — 100% совместимый с ZooKeeper по протоколу и модели данных • Compressed logs and snapshots • No issues with zxid overflow • No issues with large packets • Better memory usage • No issues

too big(max_ast_elements, max_expanded_ast_elements) ● Mutations are stuck (KILL MUTATION) ● Zookeeper OOM because of SQL length (Put ids in a Memory Engine temp table) Final Product ● Only one config

Тестирование ClickHouse, которого мы заслуживаем Особенности репликации › Асинхронная мастер-мастер › Zookeeper как хранилище метаданных › Работает потаблично › Eventually Consistent › Highly Available: - Переживает