Variance If p-value ≤ 5% we reject the null hypothesis p-value = 26.4% -O3 -O2 vs one in four experiments will show an effect that does not exist!Analysis of Variance If p-value ≤ 5% we reject the know causes this effect?� � � � � � Performance Experiments � � � � If we could magically speed up … �� � � � � � Performance Experiments � � � � If we could magically speed up � � � Performance Experiments � � � � If we could magically speed up … � More speedup in … � leads to a larger program speedup.� � � � � � Performance Experiments � � � � If we

THE OPTIMIZER SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Yes (at least in my experiments) A: Yes (at least in my experiments) 56HOW DID WE DO? HOW DID WE DO? No macros? No manual control �ow? Declarative SEE SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Again yes (at least in my experiments) A: Again yes (at least in my experiments) 64A CONCLUSION FROM THIS A CONCLUSION FROM THIS EXPERIMENT? EXPERIMENT

algorithm This is just error propagation (the bane of all freshman physics labs) 24Qualities of experiments 1. Precision 2. Reproducibility 3. Accuracy 25Precision Precision maximizes information content the code in question 26Reproducibility (handling nondeterministic results) Most scientific experiments have measurement error or noise. Interference from the environment: noisy signals Inherent in

meant we could not share saved replays between Xbox and PC.” - Shawn Hargreaves, MSDN Blog “Experiments are usually reproducible only on the same machine with the same system library and the same compiler

unprotected garbage that has accumulated Optimizing latency (work in progress) • Throughput experiments show that the average latency is quite low. • What about the tail latency? Is there anything

[13] Persistent Programming in Real Life 2019 (PIRL 2019), Persistent Memory Evaluation and Experiments (https://www.youtube.com/watch?v=M_kCL1OZjko). Retrieved 3/22/2021. [14] C. Click, “A lock-free

Demand-driven implementation ▪ Collection of the generalizable part of our vectorization Platform for experiments and discussions of future developments ▪ Single header include provided 17 Olaf Krzikalla, DLR

warning becomes a build blocker :( => we had to disable /RTCs and /RTC1 so we could do the ASan experiments /RTCs and /RTC1 Runtime Checks102 2020 Victor Ciura | @ciura_victor - 2020: The Year of Sanitizers

it matters. There’s a more “grown up” way to say this…” “ Hölzle 1994 in the course of our experiments we discovered that the trigger mechanism (“when”) is much less important for good recompilation