Repository navigation
Precompile Catch2, link with mold and run the tests once (~1.7x faster) - #114
ihid-assistant[bot] wants to merge 3 commits into
Conversation
The default make target builds the test executable and then runs it. run.sh then ran it a second time to write the XML. Building only the executable halves the time spent running tests. If the parser reports an error, run.sh runs the default target as before, so that crash messages keep make's output. Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every test file includes catch2/catch_all.hpp, which took about a second to parse on every run. The image now holds a precompiled copy, and a compiler launcher makes the test file include it first. The solution's files compile unchanged. The browser runner has no precompiled header and compiles as before. Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GNU ld took about 0.3 s to link each test executable, and mold takes about 0.06 s. run.sh only asks for mold when it is installed, so the browser runner links as before. Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ahans
left a comment
There was a problem hiding this comment.
Good effort, I always like when things get faster. However, here I find it questionable if this is the right approach. Is touching test-specific files is off-limits? If not, I think cleaning up CMakeList.txts of the tests (over in cpp) and changing the tests to not include catch_all.hpp, but only what they actually use, would be better. It would probably make sense to get rid of the bundled Catch in that context and have CMake find it. We would have to change the instructions to require students to install it locally. The test runner would then also install a catch locally. We could link against Catch2::Catch2WithMain and would maybe save some more time.
We could also move the track to C++20 at the same time.
In general, I'd appreciate some discussion beforehand. I wasn't aware that runtimes are an issue. It seemed to always be about image sizes. Now this PR almost halves the runtime, but more than doubles the image size. With some thought, I think we can have both, lower runtimes and lower (or at least identical) images sizes, and leave the track in a more maintainable state in the process.
Anyway, if there's interest, I can create a few tickets here and over at cpp and draft some PRs.
| # mold links the test executable in about 0.06 s, against about 0.3 s for | ||
| # GNU ld, because it uses all cores and does less work per symbol. | ||
| if command -v mold > /dev/null; then | ||
| cmake_options="${cmake_options} -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=mold" |
There was a problem hiding this comment.
I think this would be cleaner:
| cmake_options="${cmake_options} -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=mold" | |
| cmake_options="${cmake_options} -DCMAKE_LINKER_TYPE=MOLD" |
In general, doing things in the CMakeLists.txt I would find preferable, since then everyone benefits from improvements, not just the Docker runner.
| # write the XML. Slow solutions took twice as long and timed out sooner. | ||
| # An exercise with no target named after the slug builds everything as before. | ||
| # The browser runner has no grep, so this uses sed. | ||
| built_target_only=false |
There was a problem hiding this comment.
This is known and something I found surprising when starting working on exercises and track maintenance. I think we could discuss if we'd want to change that. Normally, you build a project with a plain make. To run anything, you'd have to explicitly say so. Or, maybe even better, we could register the test as an actual CMake test and then use ctest. But again, that would require changing all the CMakeLists.txt files.
Tests are typically fast, so I never considered this an issue. But I do see that for slow solutions we pay too much. However, I don't see why it needs to be so complicated. Always doing
make "${slug}" 2> "${compilation_errors_file_name}"followed by the line that produces the XML should work? All CMakeLists.txt are identical, so assuming that there is a target named ${slug} is save.
| # these are the options used by the exercises' CMakeLists.txt in Debug mode. | ||
| # -Wpedantic matters because it changes which macros GCC defines. | ||
| # bin/compile-with-pch.sh makes the test file include it first. | ||
| RUN c++ -std=c++17 -g -Wall -Wextra -Wpedantic -Werror -x c++-header \ |
There was a problem hiding this comment.
This sounds very fragile. I guess there's a reason why pre-compiled headers are not exactly popular ...
Hello 🙂 I'm working with Claude to try and get our tooling faster.
This PR makes this project around 1.7x faster. I've run it in production to benchmark things in the real world against the most recent few hundred solutions, so I'm confident that it works robustly. However, I've not written the code. I have tried my hardest to get Claude to write sensible messages and comments that explains stuff, but I'd ask for your patience in the fact I'm sending AI code to you. Everything below here is from your friendly neighbourhood LLM...
What changed
Most of a C++ run goes on compiling the test file, and most of that goes on parsing Catch2's
catch_all.hpp. Every test file includes it, and every run parsed it from scratch. The Dockerfile now precompiles that header once, with the same compiler options as the exercises' Debug builds. GCC only uses a precompiled header if it is included before anything else. So when the test file is compiled, a small compiler launcher (bin/compile-with-pch.sh) adds-include catch2/catch_all.hpp. The launcher only changes how the test file is compiled. The solution's.cppfiles are compiled as before. If an exercise uses different compiler options (such as a Release build), GCC ignores the precompiled header and parses the header as before.The test executable is now linked with mold, which links it several times faster than GNU ld.
run.shnow builds only the test executable. The default make target also runs the tests, andrun.shthen ran them again to write the XML. So every run executed the tests twice, and slow solutions timed out sooner. Ifexercism_parserreports an error in the test results (for example after a crash or an uncaught exception),run.shruns the default target as before, so that the message still contains make's output.The browser runner uses the same
run.sh. It has no precompiled header and no mold, sorun.shonly uses them when they are installed, and it compiles and links as before there. The browser runner's sysroot has nogrep, so the checks inrun.shfor the precompiled header and mold usesedandcase.A few messages change:
all(for examplemake: *** [Makefile:124: two-fer] Error 2). The compiler's own errors are unchanged.<string>. A header-only solution missing one of these includes will now compile, where it failed before. If the solution has a.cppfile, compiling that file will still catch the missing include.The image grows from 583 MB to 709 MB, mostly because of the precompiled header.
This has been tested in production, and has reduced the median run from 3.95s to 2.3s.
🤖 Generated with Claude Code