Vasiliki Kalavri | Boston University 2020 Synopsis maintenance & Stream Query Processing Engine Synopsis for R1 Synopsis for Rr … Query Q(R1, …, Rr) Approximate answers to Q … 31 Stream number of distinct users who have visited a website? • The top-10 queries inserted in a search engine? • The connected components of accounts in a stream of financial transactions? What synopsis No particular basic stream model (time-series, turnstile…) is imposed by the dataflow execution engine. • The burden of representation and denotations if left to the application developer/user.

clusters • tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances. • Distributing event notifications • a service that accepts user signups can send update the IDs of objects that have changed. • Logging to multiple systems • a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.

Treating a live data stream as a table that is being continuously appended. ▶ Built on the Spark SQL engine. ▶ Perform database-like query optimizations. 56 / 79 Programming Model (1/2) ▶ Two main steps Clusters”, HotCloud’12. ▶ P. Carbone et al., “Apache flink: Stream and batch processing in a single engine”, 2015. ▶ Some slides were derived from Heather Miller’s slides: http://heather.miller.am/teac

context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a TableEnvironment. TableEnvironment is central concept for creating DataStream API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a StreamExecutionEnvironment. StreamExecutionEnvironment

context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a TableEnvironment. TableEnvironment is central concept for creating DataStream API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a StreamExecutionEnvironment. StreamExecutionEnvironment

choose? 9 Vasiliki Kalavri | Boston University 2020 RocksDB 10 RocksDB is an LSM-tree storage engine with key/value interface, where keys and values are arbitrary byte streams. https://rocksdb.org/

answers … S1 S2 Sr Input Manager Scheduler QoS Monitor Load Shedder Query Execution Engine Qm Q2 Q1 Ad-hoc or continuous queries Input streams … ??? Vasiliki Kalavri | Boston University

generated as a stream of edges? • How can we perform iterative computation in a streaming dataflow engine? How can we propagate watermarks? • Do we need to run the computation from scratch for every new

the queries in advance • we can store a fixed proportion of the stream, e.g. 1/10th 7 search engine query stream Example use-case: Web search user behavior study Q: How