processing challenging? 28 Vasiliki Kalavri | Boston University 2020 Using pseudocode (or the programming language of your choice), write a program that reads a stream of integers and computes: 29 1. the

appended. ▶ Built on the Spark SQL engine. ▶ Perform database-like query optimizations. 56 / 79 Programming Model (1/2) ▶ Two main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on checks for new data (new row in the input table), and incrementally updates the result. 57 / 79 Programming Model (1/2) ▶ Two main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on checks for new data (new row in the input table), and incrementally updates the result. 57 / 79 Programming Model (1/2) ▶ Two main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on

beneficial? ??? Vasiliki Kalavri | Boston University 2020 • Use equivalence transformation rules if the language allows • selection operations are commutative • theta-join operations are commutative • natural Chromium/25.0.1364.160 Chrome/ 25.0.1364.160 Safari/537.22 Referer: https://www.google.be/ Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 GET /dumprequest HTTP/1.1 Host: Chromium/25.0.1364.160 Chrome/ 25.0.1364.160 Safari/537.22 Referer: https://www.google.be/ Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 GET /dumprequest HTTP/1.1 Host:

type, content, timing constraints. • Actions define how to produce results from the matches. Language Types 3 Vasiliki Kalavri | Boston University 2020 Three classes of operators: • relation-to-relation: portions of a stream. • relation-to-stream: create streams through querying tables Declarative language: CQL 4 Vasiliki Kalavri | Boston University 2020 Select IStream(*) From S1 [Rows 5], S2 [Rows τ> whenever tuple s is in R at time τ. 6 Vasiliki Kalavri | Boston University 2020 Imperative language: Aurora SQuAl Queries are represented in graphical representation using boxes and arrows Tumble

Boston University 2020 1. Process events online without storing them 2. Support a high-level language (e.g. StreamSQL) 3. Handle missing, out-of-order, delayed data 4. Guarantee deterministic (on Dataflow Streaming Relational Dataflow Input in-order out-of-order Results approximate exact Language SQL extensions, CQL Java, Scala, Python, SQL Execution centralized distributed Parallelism pipeline

attributes or meta-data. • Consumers subscribe to events by specifying filters in a subscription language. • Filters define constraints in the form of name-value pairs and basic comparison operators