Open benchmark of solvers for energy planning

The aim of this project is to compare and spur development in open source optimization solvers, and to track the gap between open source and proprietary solvers on problems of interest to the energy planning community. Thus, we currently have 5 solvers on the platform: 4 popular open source solvers and a single commercial proprietary solver. We welcome including any open source solver to our platform, and can support community contributions via pull requests. As we do not wish this platform to become a competition between commercial solvers, we restrict the platform to a single proprietary solver, which is Gurobi by direct agreement. Other proprietary solvers exist, and users are welcome to use our benchmarking tools to benchmark their problems on all available solvers.

We chose to run the benchmark on the cloud for several reasons: it is more cost-efficient than physical machines or bare metal cloud servers; it allows parallel runs, which speeds up the process and allows us to scale to many more benchmark problems; it is automatable using infrastructure-as-code; it is transparent and reproducible by anyone with a cloud account; and it reflects the experience of most energy modellers, who use cloud compute or shared high performance computing clusters. We are aware that runtimes vary depending on the other workloads running on the same cloud zones, and have run experiments to estimate the error in runtime. We estimate that all our results have less than 6% error in runtime measurements.