point of all Spark Streaming functionality. ▶ The second parameter, Seconds(1), represents the time interval at which streaming data will be divided into batches. val conf = new SparkConf().setAppName(appName) point of all Spark Streaming functionality. ▶ The second parameter, Seconds(1), represents the time interval at which streaming data will be divided into batches. val conf = new SparkConf().setAppName(appName) window is defined by two parameters: window length and slide interval. ▶ A tumbling window effect can be achieved by making slide interval = window length 24 / 79 Window Operations (2/3) ▶ window(windowLength

operator state • output queues • Short recovery time • High runtime overhead • The checkpoint interval determines the trade-off 14 Ni primary secondary I1 O1 N’i update checkpoint send state operator state • output queues • Short recovery time • High runtime overhead • The checkpoint interval determines the trade-off 14 Ni primary secondary I1 O1 N’i update checkpoint send state re-process many tuples • all tuples that contributed to lost state • a complete queue-trimming interval worth of tuples, if level-0 and level-1 acks are periodically transmitted Overhead • Low bandwidth

Collector> out) throws Exception { if (!ride.isStart) { Interval rideInterval = new Interval(ride.startTime, ride.endTime); Minutes duration = rideInterval.toDuration()

empty windows! Flink’s built-in window assigners create windows of type TimeWindow. : a time interval between the two timestamps, where start is inclusive and end is exclusive. 7 Built-in Window Kalavri | Boston University 2020 non-overlapping buckets of fixed size 12:10 12:00 12:20 fixed time interval key 3 key 2 key 1 Tumbling windows 8 Vasiliki Kalavri | Boston University 2020 val sensorData:

with_rolling_policy(RollingPolicy.default_rolling_policy( part_size=1024 ** 3, rollover_interval=15 * 60 * 1000, inactivity_interval=5 *␣ ˓→60 * 1000)) .build()) ds.map(lambda i: (i[0] + 1, i[1]), Types.TUPLE([Types

with_rolling_policy(RollingPolicy.default_rolling_policy( part_size=1024 ** 3, rollover_interval=15 * 60 * 1000, inactivity_interval=5 *␣ ˓→60 * 1000)) .build()) ds.map(lambda i: (i[0] + 1, i[1]), Types.TUPLE([Types

is not exceeded. The failure rate is specified as the maximum number of failures within a time interval. • e.g. you can configure that an application be restarted as long as it did not fail more than

transactions upon successful checkpoints of Flink, adding latency usually up to the checkpointing interval for each record. In practice, it has proven invaluable to add timestamps to your events at multiple

mechanism in case of failures 44 input stream time-based micro-batches D-Streams • During an interval, input data received is stored using RDDs • A D-Stream is a group of such RDDs which can be processed

0 enters the system and also an item of type B with Y = 10 is detected, followed (in a time interval of 5–15 s) by an item of type C with Z < 5. 8 Vasiliki Kalavri | Boston University 2020 Streaming