Remote Execution Caching Compiler (RECC) CppCon 2024 September 19, 2024 Shivam Bairoliya Software Engineer© 2024 Bloomberg Finance L.P. All rights reserved. What is RECC? ● Remote Execution Caching source build tool that wraps compiler commands and optionally forwards them to a remote build execution service ○ Encompasses the capabilities of both ccache and distcc ○ Supports remote linking and CC) ○ Supports multiple operating systems (Linux, macOS, Solaris) ● Compatible with any remote execution API server supported by Bazel ○ Single Host Server/Proxy: BuildBox-CASD ○ Distributed Server:

We need a system Orchestrator to implement the robot behaviorRobot Behaviors This Coordinator or Task Planner is also the highest level of abstraction of our system, where we focus on WHAT the robot current state of the tree in real-time to an external process (GUI). ● Collect statistics about the execution of the tree and the actions in a non-intrusive way.Code injection and interactive debuggingUser

used when we created them.” - Albert EinsteinSo what is there to Rethink?Rethinking: Task Queues Problem #1 - Task Queues Do Not Scale Well: ● Contention: ○ Even the most meticulously designed lock-free problems: ○ Task starvation ○ Load balancing ○ Forfeits strict FIFO behaviour ○ Increases memory footprint (or requires allocations) ○ Terrible task selection “fairness” Task Thread Thread Thread Task Task Task Task Task Task Task Task Back Front Task Queue Execute Task() Thread Thread PoolRethinking: Task Queues Problem #2 - No Inherent Support For Prioritization: ● Priority queues address this but

in a top-down task graph What is Task Graph-based Programming System (TGPS) Code 4• TGPS encapsulates function calls and their dependencies in a top-down task graph What is Task Graph-based Programming top-down task graph What is Task Graph-based Programming System (TGPS) Code A B C D B A C D 6• TGPS encapsulates function calls and their dependencies in a top-down task graph What is Task Graph-based sched; 2 task_a = sched.emplace([](&){ 3 // Code block A; 4 }); 5 task_b = sched.emplace([](&){ 6 // Code block B; 7 }); 8 task_c = sched.emplace([](&){ 9 // Code block C; 10 }); 11 task_d = sched

112 9 API Reference 113 9.1 Task Decorators - celery.decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 9.2 Defining Tasks - celery.task.base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 9.3 Task Sets, Subtasks and Callbacks - celery.task.sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 9.4 Executing Tasks - celery . . . . . 122 9.5 Task Result - celery.result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 9.6 Task Information and Utilities - celery.task . . . . . . . . . .

older version of Celery (2.1). For the latest stable version please go here. Celery - Distributed Task Queue Contents: Getting Started Introduction Broker Installation First steps with Celery Resources Reference Task Decorators - celery.decorators Defining Tasks - celery.task.base Task Sets, Subtasks and Callbacks - celery.task.sets Executing Tasks - celery.execute Task Result - celery.result Task Information Information and Utilities - celery.task Configuration - celery.conf Remote Management of Workers - celery.task.control HTTP Callback Tasks - celery.task.http Periodic Task Schedule Behaviors - celery.schedules

documentation » http://cloud.github.com/downloads/ask/celery/celery_favicon_128.png Celery - Distributed Task Queue Contents: Getting Started Introduction Broker Installation First steps with Celery Periodic (Webhooks) Routing Tasks Configuration and defaults Example configuration file Concurrency settings Task result backend settings Database backend settings AMQP backend settings Cache backend settings Tokyo settings Redis backend settings MongoDB backend settings Messaging settings Task execution settings Worker: celeryd Periodic Task Server: celerybeat Monitor Server: celerymon Cookbook Creating Tasks Running

http://github.com/ask/celery/ Keywords task queue, job queue, asynchronous, rabbitmq, amqp, redis, python, webhooks, queue, dis- tributed – Celery is an open source asynchronous task queue/job queue based on distributed distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers. Tasks can execute workload. The result of the task can be stored for later retrieval (called its “tombstone”). 1.1.2 Example You probably want to see some code by now, so here’s an example task adding two numbers: from

http://github.com/ask/celery/ Keywords task queue, job queue, asynchronous, rabbitmq, amqp, redis, python, webhooks, queue, dis- tributed – Celery is an open source asynchronous task queue/job queue based on distributed distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on one or more worker nodes using multiprocessing of the task can be stored for later retrieval (called its “tombstone”). 1.1.2 Example You probably want to see some code by now, so here’s an example task adding two numbers: from celery.task import

. . . . . . . . . . . 192 10.4 celery.task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 10.5 celery.task.base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 10.6 celery.task.sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 10.7 celery.result . . . . . . . . . . . 202 10.8 celery.task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 10.9 celery.task.control . . . . . . . . . . . . . . .