Transformation ▶ Output operations 12 / 79 Input Operations ▶ Every input DStream is associated with a Receiver object. • It receives the data from a source and stores it in Spark’s memory for processing. ▶ , user-provided sources. 13 / 79 Input Operations ▶ Every input DStream is associated with a Receiver object. • It receives the data from a source and stores it in Spark’s memory for processing. ▶ partitions]) 15 / 79 Input Operations - Custom Sources (1/3) ▶ To create a custom source: extend the Receiver class. ▶ Implement onStart() and onStop(). ▶ Call store(data) to store received data inside Spark

whether receivers have the required capacity to handle data messages. • When the credit of a receiver drops to zero (or a specified threshold), backpressure appears on its virtual channel. ??? Vasiliki

• TaskManagers have a pool of network buffers to send and receive data. • If the sender and receiver run in separate processes, they communicate via permanent TCP connections. • If they run in