GPU Tile 1 Tile 0 Xe LinkProject Goals - Offer high-level, standard C++ distributed data structures - Support distributed algorithms - Achieve high performance for both multi-GPU, NUMA, and multi-node reduce(par_unseq, z, 0, std::plus()); }Outline - Background (Ranges, Parallelism, Distributed Data Structures) - Distributed Ranges (Concepts) - Implementation (Algorithms and views) - Complex sparse matrices) - Lessons learnedOutline - Background (Ranges, Parallelism, Distributed Data Structures) - Distributed Ranges (Concepts) - Implementation (Algorithms and views) - Complex

program for a supercomputer? Introduce PGAS Model, RDMA Building Remote Pointer Types Building Distributed Data Structures Extending to GPUsThis Talk Background: how do we write a program for a supercomputer supercomputer? Introduce PGAS Model, RDMA Building Remote Pointer Types Building Distributed Data Structures Extending to GPUsThis Talk Background: how do we write a program for a supercomputer? Introduce Introduce PGAS Model, RDMA Building Remote Pointer Types Building Distributed Data Structures Extending to GPUsThis Talk Background: how do we write a program for a supercomputer? Introduce PGAS Model

blocks in the ledger and returns the first block’s index or BlockCount on failure std::size_t find_contiguous_blocks(std::size_t n) const noexcept; // Marks n blocks in the ledger as “in-use” starting void set_blocks_in_use(std::size_t index, std::size_t n) noexcept; // Marks n blocks in the ledger as “free” starting at ‘index’ void set_blocks_free(std::size_t index, std::size_t n) noexcept; m_data{nullptr}; // Reserves one bit per block to indicate whether it is in-use std::byte* m_ledger{nullptr}; }; © 2020 Apex.AI, Inc.Construction/Destruction bucket::bucket(std::size_t block_size

Subscription2 Consumer 1 2 3 4 5 6 7 readEntries Ack/N-Ack Broke BookKeepe 1 2 3 4 5 6 7 Ledger stores Data A B C D E Ledger stores Cursor changes Curso Subscription1 … Write Ack ManagedLedger + -/+ streamnative

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 25 Multi-processing and Distributed Computing 309 25.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1113 66 Delimited Files 1138 67 Distributed Computing 1143 67.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 25 Multi-processing and Distributed Computing 309 25.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1113 66 Delimited Files 1138 67 Distributed Computing 1143 67.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 25 Multi-processing and Distributed Computing 310 25.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1114 66 Delimited Files 1139 67 Distributed Computing 1144 67.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 26 Multi-processing and Distributed Computing 328 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1298 68 Delimited Files 1329 69 Distributed Computing 1334 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 26 Multi-processing and Distributed Computing 328 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1298 68 Delimited Files 1329 69 Distributed Computing 1334 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 26 Multi-processing and Distributed Computing 329 26.1 Code Availability and Loading Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1301 68 Delimited Files 1332 69 Distributed Computing 1337 69.1 Cluster Manager Interface . . . . . . . . . . . . . . . . . . . . . . . . need to vectorize code for performance; devectorized code is fast • Designed for parallelism and distributed computation • Lightweight "green" threading (coroutines) • Unobtrusive yet powerful type system