– 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Baselines  #1 Primary Baseline  Existing algorithm or system infrastructure  Main comparison point, usually operations  Beware: Avoid speedup-only results (need absolute numbers for grounding)  #2 Additional Baselines  Alternative systems w/ different runtime and compiler  Usually, not directly comparable but important for grounding  E.g.,: for SystemDS  R, Julia, Spark, TensorFlow, PyTorch  Problem of Weak Baselines  Authors want to show improvements  Successive improvements over state-of-the-art don’t add up

sampling model running in 30us on single server CPU core - Beat hand-written, highly optimized baselines (https://github.com/mozilla/LPCNet) by ~40% - Bonus: Real-time on mobile CPUs for free 6 TVM

Reproducible Experiments  Create, use, and share dedicated benchmarks / datasets  Avoid weak baselines, start early w/ baseline comparisons  Automate your experiments as much as possible  Keep repository

letters ”sit” and below which descenders extend” (Wikipedia). In HTML, there are several kinds of baselines (alphabetic, ideographic, bottom…). Here, consider that baseline is ”bottom” and determines the