– 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Baselines  #1 Primary Baseline  Existing algorithm or system infrastructure  Main comparison point, usually operations  Beware: Avoid speedup-only results (need absolute numbers for grounding)  #2 Additional Baselines  Alternative systems w/ different runtime and compiler  Usually, not directly comparable but important for grounding  E.g.,: for SystemDS  R, Julia, Spark, TensorFlow, PyTorch  Problem of Weak Baselines  Authors want to show improvements  Successive improvements over state-of-the-art don’t add up

Reproducible Experiments  Create, use, and share dedicated benchmarks / datasets  Avoid weak baselines, start early w/ baseline comparisons  Automate your experiments as much as possible  Keep repository