vkalavri@bu.edu Spring 2020 3/17: High availability, recovery semantics, and guarantees Vasiliki Kalavri | Boston University 2020 Today’s topics • High-availability and fault-tolerance in distributed state consists of • input queues • operator state • output queues • Short recovery time • High runtime overhead • The checkpoint interval determines the trade-off 14 Ni primary secondary state consists of • input queues • operator state • output queues • Short recovery time • High runtime overhead • The checkpoint interval determines the trade-off 14 Ni primary secondary

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . cluster managers. 7 Kyuubi, Release 1.6.1-incubating 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.6.1-incubating 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . cluster managers. 7 Kyuubi, Release 1.6.0-incubating 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.6.0-incubating 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . cluster managers. 7 Kyuubi, Release 1.5.0-incubating 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.5.0-incubating 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . cluster managers. 7 Kyuubi, Release 1.5.1-incubating 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.5.1-incubating 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . cluster managers. 7 Kyuubi, Release 1.5.2-incubating 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.5.2-incubating 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . against different cluster managers. 7 Kyuubi, Release 1.3.0 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent execution plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.3.0 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . against different cluster managers. 7 Kyuubi, Release 1.3.0 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent execution plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.3.0 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . cluster managers. 7 Kyuubi, Release 1.4.1-incubating 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.4.1-incubating 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained

GUIDE 1 Multi-tenancy 3 2 Ease of Use 5 3 Run Anywhere 7 4 High Performance 9 5 Authentication & Authorization 11 6 High Availability 13 6.1 Quick Start . . . . . . . . . . . . . . . . . . . . . against different cluster managers. 7 Kyuubi, Release 1.3.0 8 Chapter 3. Run Anywhere CHAPTER FOUR HIGH PERFORMANCE Kyuubi is built on the Apache Spark, a lightning-fast unified analytics engine. • Concurrent execution plan: fully supports Spark SQL Catalyst Optimizer, 9 Kyuubi, Release 1.3.0 10 Chapter 4. High Performance CHAPTER FIVE AUTHENTICATION & AUTHORIZATION With strong authentication and fine-grained