results  refutable by evidence  Continuous Experiments  Run experiments during survey / prototype building  Systematic experiments  observations and ideas for improvements  Don’t be afraid of throw away aspects in controlled and understandable scope  Bottom-up approach  #3 Benchmarks  Evaluate on community/own benchmarks  Examples: TPC-C, TPC-H, TPC-DS, JOB, MLPerf  #4 End-to-end Applications  Evaluate Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Benchmarks  Overview  Community- and organization-driven creation of agreed benchmarks  Benchmarks can define a field and foster

bringing these ideas to the fore, we hope to ground discussions and make it clearer to our broader community why the rules are in place and why particular decisions have been made. If you understand what goals is outweighed by the cost of having people argue over them. Be consistent with the broader C++ community when appropriate Consistency with the way other organizations use C++ has value for the same reasons prevail. By this we specifically refer to the established conventions of the entire Google C++ community, not just your personal preferences or those of your team. Be skep- tical about and reluctant to

read, share, and verify. The rules below were designed in collaboration with the entire R user community at Google. Summary: R Style Rules 1. File Names: end in .R 2. Identifiers: variable.name (or variableName)

falsifiability of scientific results  Continuous Experiments  Run experiments during survey / prototype building  Systematic experiments  observations and ideas for improvements  Don’t be afraid of throw away analytical queries. SIGMOD 2014  #15.5 Yannis Klonatos, Christoph Koch, Tiark Rompf, Hassan Chafi: Building Efficient Query Engines in a High-Level Language. PVLDB 7(10) 2014  #15.6 Thomas Neumann: Efficiently