6.3 Q3: Types.BIG_INT() VS Types.LONG() . . . . . . . . . . . . . . . . . . . . . . 32 1.4 API reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 githubusercontent.com/apache/flink/master/flink-python/pyflink/ ˓→examples/table/word_count.py -o word_count.py python3 word_count.py # You will see outputs as following: # Use --input to specify file input. # githubusercontent.com/apache/flink/master/flink-python/pyflink/ ˓→examples/table/word_count.py -o word_count.py python3 word_count.py If there any any problems, you could check the logging messages in the log

6.3 Q3: Types.BIG_INT() VS Types.LONG() . . . . . . . . . . . . . . . . . . . . . . 32 1.4 API reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 githubusercontent.com/apache/flink/master/flink-python/pyflink/ ˓→examples/table/word_count.py -o word_count.py python3 word_count.py # You will see outputs as following: # Use --input to specify file input. # githubusercontent.com/apache/flink/master/flink-python/pyflink/ ˓→examples/table/word_count.py -o word_count.py python3 word_count.py If there any any problems, you could check the logging messages in the log

2020 Counting distinct elements 2 ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting visiting one or multiple webpages ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one Naive solution: maintain a hash table ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting

= ssc.receiverStream(new CustomReceiver(host, port)) val words = customReceiverStream.flatMap(_.split(" ")) 18 / 79 Operations on DStreams ▶ Input operations ▶ Transformation ▶ Output operations the records of the source DStream on which func returns true. 21 / 79 Transformations (3/4) ▶ count • Returns a new DStream of single-element RDDs by counting the number of elements in each RDD of DStream that contains the union of the elements in two DStreams. 22 / 79 Transformations (3/4) ▶ count • Returns a new DStream of single-element RDDs by counting the number of elements in each RDD of

their contents as a function of time. • average price of items bought within the last 5 minutes • Count-based (physical) windows define their contents according to the number of events. • average price commonly to be used as input to multiple downstream operators. • Group by / Partition Operators split a stream into sub-streams according to a function or the event contents. • one stream per customer Vasiliki Kalavri | Boston University 2020 CQL GroupBy Example Select IStream(Count(*)) From S1 [Rows 1000] Group By S1.B Count the number or events in the last 1000 rows for each value of B 20

3.Output to Sinks 3 Vasiliki Kalavri | Boston University 2020 Streaming word count textStream .flatMap {_.split("\\W+")} .map {(_, 1)} .keyBy(0) .sum(1) .print() “live and let

merge, split, query, subscribe, … State operations and types 4 Consider you are designing a state interface. What operations should state support? What state types can you think of? • Count, sum,

?? Vasiliki Kalavri | Boston University 2020 Types of Parallelism 7 B A C A B D A A B split Pipeline: A || B Task: B || C Data: A || A ??? Vasiliki Kalavri | Boston University 2020 8 Distributed applying B and then A. • holds if both operators are stateless Re-ordering split and merge split merge merge split merge split When might this be beneficial? ??? Vasiliki Kalavri | Boston University deadlocks: if split cannot push data because one channel is full and merge cannot receive data because another channel is empty Operator fission Data parallelism, replication A A A split merge ???

the sum of the squares of the values • the number of observations • μ = sum / count • var = (sum of squares / count) - μ2 Then var = ∑ (xi − μ)2 N ??? Vasiliki Kalavri | Boston University 2020 compute the three summary values in a single pass through the data. • μ = sum / count • var = (sum of squares / count) - μ2 Then var = ∑ (xi − μ)2 N ??? Vasiliki Kalavri | Boston University 2020 compute the three summary values in a single pass through the data. • μ = sum / count • var = (sum of squares / count) - μ2 Then var = ∑ (xi − μ)2 N ??? Vasiliki Kalavri | Boston University 2020

parameter of an operator defines the number of key groups into which the keyed state of the operator is split. • The number of key groups limits the maximum number of parallel tasks to which keyed state can