– 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Baselines  #1 Primary Baseline  Existing algorithm or system infrastructure  Main comparison point, usually operations  Beware: Avoid speedup-only results (need absolute numbers for grounding)  #2 Additional Baselines  Alternative systems w/ different runtime and compiler  Usually, not directly comparable but important for grounding  E.g.,: for SystemDS  R, Julia, Spark, TensorFlow, PyTorch  Problem of Weak Baselines  Authors want to show improvements  Successive improvements over state-of-the-art don’t add up

conclusions 1. Support building from source 2. Build a verified binary cache 3. Version using baselines 4. Build open-source with a package manager 5. Cache build assets internally 6. Monitor, prevent version ? Consuming projectProblem 3: Version conflicts – The Diamond problem Solution 3: Manage baselines, not individual libraries boost@v1 openssl@v1 gtest@v1 … Catalog baseline: b60f003ccf5fe8613 CppCon 2021 talk from BloombergProblem 3: Version conflicts – The Diamond problem Solution 3: Manage baselines, not individual libraries Test version updates using the “cones of destruction” method Library

external systems. As with CM on traditional programs, there needs to be rigor on managing changes to baselines, designs, engineering, and program documentation. Some additional CM methods for Agile include: place to ensure rigorous configuration management of requirements, code, technical designs, and baselines?  What is the continuous improvement strategy? See also: Agile architecture articles 10 meets the design-to or build-to specifications defined in the functional, allocated, and product baselines. In an Agile environment verification is an integral element of each sprint. At the end of each

real-world application • Conclusion 74Hardware Platform & Baselines • Eight CPU threads (3.2 GHz) • One NVIDIA 3080 ti GPU • Baselines - Microbenchmarks • Taskflow • 𝐓𝐚𝐬𝐤𝐟𝐥𝐨𝐰𝐜𝐨𝐫𝐨 1

matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the corresponding optimizer hint for this bindings for these queries. Enable capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: Because the automatic binding creation function relies capture_plan_baselines_blacklist �→ . Then the filtering conditions take effect in the entire cluster immediately. 1337 -- Filter by table name INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type

“1024GB”. 2.2.1.3 Others 35 • Before the upgrade, check whether the value of the tidb_evolve_plan_baselines system variable is ON. If the value is ON, set it to OFF; otherwise, the upgrade will fail. • matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the corresponding optimizer hint for this show warnings; 9.3.4.3.2 Baseline capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: Because the automatic binding creation function relies

matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the 680 corresponding optimizer hint for warnings; 685 9.3.4.3.2 Baseline capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: Because the automatic binding creation function relies to enable automatic binding evolution: set global tidb_evolve_plan_baselines = on; The default value of tidb_evolve_plan_baselines is off. Warning: • Baseline evolution is an experimental feature.

Reproducible Experiments  Create, use, and share dedicated benchmarks / datasets  Avoid weak baselines, start early w/ baseline comparisons  Automate your experiments as much as possible  Keep repository

connection between deep learning and machine learning). Deep Learning models have beaten previous baselines significantly in many tasks in computer vision, natural language understanding, speech, and so on

matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the corresponding optimizer hint for this bindings for these queries. Enable capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: 1615 Because the automatic binding creation function relies mysql.capture_plan_baselines_blacklist �→ . Then the filtering conditions take effect in the entire cluster immediately. -- Filter by table name INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type