– 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Baselines  #1 Primary Baseline  Existing algorithm or system infrastructure  Main comparison point, usually operations  Beware: Avoid speedup-only results (need absolute numbers for grounding)  #2 Additional Baselines  Alternative systems w/ different runtime and compiler  Usually, not directly comparable but important for grounding  E.g.,: for SystemDS  R, Julia, Spark, TensorFlow, PyTorch  Problem of Weak Baselines  Authors want to show improvements  Successive improvements over state-of-the-art don’t add up

conclusions 1. Support building from source 2. Build a verified binary cache 3. Version using baselines 4. Build open-source with a package manager 5. Cache build assets internally 6. Monitor, prevent version ? Consuming projectProblem 3: Version conflicts – The Diamond problem Solution 3: Manage baselines, not individual libraries boost@v1 openssl@v1 gtest@v1 … Catalog baseline: b60f003ccf5fe8613 CppCon 2021 talk from BloombergProblem 3: Version conflicts – The Diamond problem Solution 3: Manage baselines, not individual libraries Test version updates using the “cones of destruction” method Library

external systems. As with CM on traditional programs, there needs to be rigor on managing changes to baselines, designs, engineering, and program documentation. Some additional CM methods for Agile include: place to ensure rigorous configuration management of requirements, code, technical designs, and baselines?  What is the continuous improvement strategy? See also: Agile architecture articles 10 meets the design-to or build-to specifications defined in the functional, allocated, and product baselines. In an Agile environment verification is an integral element of each sprint. At the end of each

real-world application • Conclusion 74Hardware Platform & Baselines • Eight CPU threads (3.2 GHz) • One NVIDIA 3080 ti GPU • Baselines - Microbenchmarks • Taskflow • 𝐓𝐚𝐬𝐤𝐟𝐥𝐨𝐰𝐜𝐨𝐫𝐨 1

matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the corresponding optimizer hint for this bindings for these queries. Enable capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: Because the automatic binding creation function relies capture_plan_baselines_blacklist �→ . Then the filtering conditions take effect in the entire cluster immediately. 1337 -- Filter by table name INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type

“1024GB”. 2.2.1.3 Others 35 • Before the upgrade, check whether the value of the tidb_evolve_plan_baselines system variable is ON. If the value is ON, set it to OFF; otherwise, the upgrade will fail. • matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the corresponding optimizer hint for this show warnings; 9.3.4.3.2 Baseline capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: Because the automatic binding creation function relies

sampling model running in 30us on single server CPU core - Beat hand-written, highly optimized baselines (https://github.com/mozilla/LPCNet) by ~40% - Bonus: Real-time on mobile CPUs for free 6 TVM

matches one of the bound SQL statements in the system table and the system variable tidb_use_plan_baselines is set to on (the default value is on), TiDB then uses the 680 corresponding optimizer hint for warnings; 685 9.3.4.3.2 Baseline capturing To enable baseline capturing, set tidb_capture_plan_baselines to on. The default value is off. Note: Because the automatic binding creation function relies to enable automatic binding evolution: set global tidb_evolve_plan_baselines = on; The default value of tidb_evolve_plan_baselines is off. Warning: • Baseline evolution is an experimental feature.

compaction bytes 达到该阈值时， 流控机制开始 拒绝所有写入 请求并报错。 默认值为 “1024GB”。 2.2.1.3 其他 • 升级前，请检查系统变量tidb_evolve_plan_baselines 的值是否为 ON。如果为 ON，需要将其改成 OFF， 否则会导致升级失败。 • v4.0 集群升级到 v5.2 集群后，tidb_multi_statement_mode 变量的默认值由 SQL 进行 “标准化” 处理，然后存储到系统表中。在处理 SQL 查询时，只要 “标准化” 后的 SQL 和系统表中某个被绑 定的 SQL 语句一致，并且系统变量 tidb_use_plan_baselines 的值为 on（其默认值为 on），即可使用相应的优 化器 Hint。如果存在多个可匹配的执行计划，优化器会从中选择代价最小的一个进行绑定。 标准化：把 SQL 中的常量变成变量参数，对空格 select * from t; show warnings; 9.3.4.3.2 自动捕获绑定 (Baseline Capturing) 通过将 tidb_capture_plan_baselines 的值设置为 on（其默认值为 off）可以打开自动捕获绑定功能。 注意： 自动绑定功能依赖于Statement Summary，因此在使用自动绑定之前需打开 Statement Summary

Reproducible Experiments  Create, use, and share dedicated benchmarks / datasets  Avoid weak baselines, start early w/ baseline comparisons  Automate your experiments as much as possible  Keep repository