spark-submit \ --master k8s://https://192.168.45.93:5443 \ --deploy-mode cluster \ --name query \ --class com.databricks.spark.sql.perf.BenchmarkQuery \ --conf spark.kubernetes.volcano.enable=true \ --conf spark podgrup4 Solution: Pod delay creation l Spark-sql-perf(TPC-DS, master) l 26 concurrent l 4 nodes(8core, 64G, HDD) l Kubernetes 1.13 l Spark-sql-perf(TPC-DS, master) l 104 concurrent l 4 nodes(8core

& Priority Policy. Ø NodeResourceUsage Annotator scores node periodically. Ø Dynamic-Scheduler query node score directly from node to make decisions during predicate and priority stage. Dynamic Scheduler ThreadPressure • KubeletProblem • DockerdProblem • ContainerdProblem • KernelDeadlock • … Ø Distributed Ping Test • Self-agent • Self-holder de-scheduler npd self-holder Pod Kernel Kubelet Dockerd

提供了从开发到生产的安全状态的统一视图，因此我们将其放入试验阶段。 31. Trino 试验 Trino 以前被称之为 PrestoSQL，是一个专为面向大数据交互式分析查询而设计的开源分布式 SQL 查询引擎。经 过优化后，它可以在本地或者云上环境运行，并支持对 Hive、Cassandra、关系型数据库、甚至专有数据存储 等多种不同的数据源进行查询。它支持基于密码的认证、LDAP 和 OAuth Thoughtworks, Inc. All Rights Reserved. 26 42. dbt 采纳 dbt 仍是我们在 ETL 工作流程中进行数据转换的首选工具。我们喜欢它的工程严谨性和它在 SQL 数据转换中实 践模块化、可测试性、和可复用性的能力。 dbt 有开源和商业化 SaaS 产品两种版本和健康的生态，包括一个 提供了许多用于单元测试、数据质量、数据可观测性等软件包的社区。这些包中尤为值得注意的是用于监测数 dbt-unit-testing。dbt 很好地集成了各种云数 据仓库、数据湖和数据库，包括 Snowflake，BigQuery，Redshift，Databricks 和 Postgres。当需要处理结构 化数据并且能使用 SQL 进行数据转换时，我们的团队们倾向于 dbt，因此我们将它移至采纳阶段。 43. Mermaid 采纳 Mermaid 通过使用类似 Markdown 的标记语言来生成图表。自从上次在技术雷达中介绍以来，Mermaid

• ��������Session�� • �����������TraceId�� ������� ������� ������ ����� ������� ���������� SQL92�� ��/�� Join����� IP���� ���� ������ ��� ���� ��� ���� �������� ����(1/2) • ������ • ��������� OSS Hybrid Storage Array Massive Structured, Unstructured & Semi- structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages

对资源进行分配、调度、配置、部署等工作。 Lable就是给资源对象打一个标签，然后通过Label Secletor（标签选择器）查询和筛选拥有某些Label的资源对象， k8s通过这种方式实现了类似SQL的简单和通用的对象查询机制。 标签选择器可以类比SQL语句中的where查询条件，例如name=redis-salve 表示只查询名字为redis-salve的资源对 象。 env != production: 匹配所以不具有标签

Solution is Kubernetes: • Highly Scalable • Easy to explore hyper-parameters space • Easy to do distributed training But really, Data Scientists shouldn’t have to care about containers, kubernetes and that stuff • Pachyderm can version datasets and trigger new trainings when changes occur • Distributed File Systems • NFS • HDFS • … Classic DevOps solutions: • Containers • CI/CD • Autoscaling

the zones in your cluster have uneven available resources due to node variations or unevenly distributed pre-existing workloads, this might prevent perfectly even spreading of your pods across zones systemd) 20 Supplement Kubernetes Resource Management with vSphere DRS What is DRS? The vSphere Distributed Resource Scheduler (DRS) is a load balancer for VMs deployed on a hypervisor cluster. It has advanced

the zones in your cluster have uneven available resources due to node variations or unevenly distributed pre-existing workloads, this might prevent perfectly even spreading of your pods across zones systemd) 20 Supplement Kubernetes Resource Management with vSphere DRS What is DRS? The vSphere Distributed Resource Scheduler (DRS) is a load balancer for VMs deployed on a hypervisor cluster. It has advanced

Kubernetes-based Scheduler Framework》 The IEEE ISPA 2018 (16th IEEE International Symposium on Parallel and Distributed Processing with Applications) 能力扩展：GPU支持 资源-访问代价树 四类通信方式分类中，通信开销最大的是SOC，其次是PXB，再次是PHB，PIX通信方式 GPUs in Container Clouds》 The IEEE ISPA 2018 (16th IEEE International Symposium on Parallel and Distributed Processing with Applications) 生态 Next Tencent Kubernetes Engine

Application architecture Application architecture PHP-based Admin application Shared state in Redis and SQL database Scalable userfacing API in GoLang Queuing system Application technologies Best of breed-technologies