Machine Learning (ML) pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas, then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem tgz && \ tar -xvf Python-3.7.9.tgz && \ cd Python-3.7.9 && \ ./configure --without-tests --enable-shared && \ make -j6 && \ make install && \ ldconfig /usr/local/lib && \ cd .. && rm -f Python-3.7.9.tgz

Machine Learning (ML) pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas, then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem tgz && \ tar -xvf Python-3.7.9.tgz && \ cd Python-3.7.9 && \ ./configure --without-tests --enable-shared && \ make -j6 && \ make install && \ ldconfig /usr/local/lib && \ cd .. && rm -f Python-3.7.9.tgz

separate threads of control • avoid communication cost without losing pipeline parallelism • use a shared buffer for communication • Fused filters / projections at the source can significantly reduce I/O disk I/O • fixed number of random state accesses, 32K L1 cache • the throughput of the non-shared version degrades first State sharing B A Β Α Profitability ??? Vasiliki Kalavri | Boston University

producer only needs to receive ack from broker 9 Communication patterns (I) Load balancing or shared subscription • A logical producer/consumer can be implemented by multiple physical tasks running

connect(priceRequests).flatMap( new CoFlatMapFunction[(String, Double), Item, Offer] { // shared state between the two streams val factorValues: HashMap[String, Double] = HashMap.empty

Vasiliki Kalavri | Boston University 2020 Distributed dataflow model • Exploit data parallelism and shared-nothing architectures to scale stream processing to high-volume streams and large state • Streams