userSentPayment 4 Connecting producers to consumers • Indirectly • Producer writes to a file or database • Consumer periodically polls and retrieves new data • polling overhead, latency? • Consumer for temporary storage • Messages stored on the queue until they are processed and deleted • transactional, timing, and ordering guarantees • Each message is processed only once, by a single consumer Databases • DBs keep data until explicitly deleted while MBs delete messages once consumed. • Use a database for long-term data storage! • MBs assume a small working set. If consumers are slow, throughput

traditional data processing applications, we know the entire dataset in advance, e.g. tables stored in a database. A data stream is a data set that is produced incrementally over time, rather than being available not know when the stream ends. 3 Vasiliki Kalavri | Boston University 2020 DW DBMS SDW DSMS Database Management System • ad-hoc queries, data manipulation tasks • insertions, updates, deletions Vasiliki Kalavri | Boston University 2020 Synopsis maintenance & Stream Query Processing Engine Synopsis for R1 Synopsis for Rr … Query Q(R1, …, Rr) Approximate answers to Q … 31 Stream

operations 30 / 79 Output Operations (1/4) ▶ Push out DStream’s data to external systems, e.g., a database or a file system. ▶ foreachRDD: the most generic output operator • Applies a function to each live data stream as a table that is being continuously appended. ▶ Built on the Spark SQL engine. ▶ Perform database-like query optimizations. 56 / 79 Programming Model (1/2) ▶ Two main steps to develop -spark.html] 60 / 79 Structured Streaming Example (2/3) ▶ We could express it as the following SQL query. SELECT action, WINDOW(time, "1 hour"), COUNT * FROM events GROUP BY action, WINDOW(time, "1

persistent message queue, before it is processed by Apache Flink, which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for each network shuffle, takes time and adds to latency. 5. If the application emits through a transactional sink, the sink will only commit and publish transactions upon successful checkpoints of Flink consider latency, which is introduced inside the Flink topology and cannot be attributed to transactional sinks or events being buffered for functional reasons (4.). To this end, Flink comes with a feature

TableEnvironment is the entry point and central context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a TableEnvironment User-defined function management: User-defined function registration, dropping, listing, etc. • Executing SQL queries • Job configuration • Python dependency management • Job submission For more details of [5]: root |-- id: BIGINT |-- data: STRING Create a Table from DDL statements [6]: table_env.execute_sql(""" CREATE TABLE random_source ( id TINYINT, data STRING ) WITH ( 'connector' = 'datagen', 'fields

TableEnvironment is the entry point and central context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a TableEnvironment User-defined function management: User-defined function registration, dropping, listing, etc. • Executing SQL queries • Job configuration • Python dependency management • Job submission For more details of [5]: root |-- id: BIGINT |-- data: STRING Create a Table from DDL statements [6]: table_env.execute_sql(""" CREATE TABLE random_source ( id TINYINT, data STRING ) WITH ( 'connector' = 'datagen', 'fields

Boston University 2020 Three classes of operators: • relation-to-relation: similar to standard SQL and define queries over tables. • stream-to-relation: define tables by selecting portions of a answers when it detects the end of its input. • NOT IN, set difference and division, traditional SQL aggregates • A Non-blocking query operator can produce answers incrementally as new input records continuous queries. 29 Vasiliki Kalavri | Boston University 2020 Non-blocking SQL Let NB-SQL be the non-blocking subset of SQL that excludes non- monotonic constructs: • EXCEPT, NOT EXIST, NOT IN and ALL

or more streams of possibly different type A series of transformations on streams in Stream SQL, Scala, Python, Rust, Java… ??? Vasiliki Kalavri | Boston University 2020 Logic State context of streaming? • queries run continuously • streams are unbounded • In traditional ad-hoc database queries, the query plan is generated on- the-fly. Different plans can be used for two consecutive

management • checkpointing state to remote and persistent storage, e.g. a distributed filesystem or a database system • Available state backends in Flink: • In-memory • File system • RocksDB State backends choose? 9 Vasiliki Kalavri | Boston University 2020 RocksDB 10 RocksDB is an LSM-tree storage engine with key/value interface, where keys and values are arbitrary byte streams. https://rocksdb.org/

可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Scala/Java SQL HiveSQL SparkSQL ANSI SQL 易易⽤用性 ⼀一般 易易⽤用 ⼀一般 ⼯工具/⽣生态 ⼀一般 丰富 ⼀一般 Flink Batch应⽤用 - 数据湖 Data Lake vs. Data Warehouse Flink Batch应⽤用 - 数据湖 Flink Batch应⽤用 - 数据湖 Blink SQL+UDF Queue