Open benchmark of solvers for energy planning

The aim of this project is to compare and spur development in open source optimization solvers, and to track the gap between open source and proprietary solvers on problems of interest to the energy planning community. Thus, we currently have 5 solvers on the platform: 4 popular open source solvers and a single commercial proprietary solver. We welcome including any open source solver to our platform, and can support community contributions via pull requests. As we do not wish this platform to become a competition between commercial solvers, we restrict the platform to a single proprietary solver, which is Gurobi by direct agreement. Other proprietary solvers exist, and users are welcome to use our benchmarking tools to benchmark their problems on all available solvers.

We chose to run benchmarks on the cloud for several reasons: it is more cost-efficient than physical machines or bare metal cloud servers; it allows us to run benchmarks in parallel, which speeds up our run and allows us to scale to many more benchmark instances; it is automatable using infrastructure-as-code; it is transparent and reproducible by anyone with a cloud account; and it reflects the experience of most energy modellers, who use cloud compute or shared high performance computing clusters. We are aware that runtimes vary depending on the other workloads running on the same cloud zones, and have run experiments to estimate the error in runtime. We estimate that all our results have less than 6% error in runtime measurements.

We use the following nomenclature on this platform: An energy modelling framework, e.g. PyPSA or TIMES, is a software system that allows one to input country or region-specific data and model the energy system of interest. An energy model, e.g. eTIMES-EU or TIMES-NZ, is an instantiation of a modelling framework for a particular application or study. A benchmark (problem) is a single model scenario of an energy model, captured as an LP or MPS file that is given as input to an optimization solver. In order to study the scaling behavior of solvers, and to provide solver developers with smaller versions of problems of interest, we group benchmark problems obtained by varying size parameters such as spatial or temporal resolution as being size instances of the same benchmark. The full collection of benchmarks on our platform is the benchmark set.