Skip to content

Precompile Catch2, link with mold and run the tests once (~1.7x faster) - #114

Open
ihid-assistant[bot] wants to merge 3 commits into
mainfrom
speed-up-runs
Open

ihid-assistant[bot] wants to merge 3 commits into
mainfrom
speed-up-runs

Conversation

@ihid-assistant

Copy link
Copy Markdown

Hello 🙂 I'm working with Claude to try and get our tooling faster.

This PR makes this project around 1.7x faster. I've run it in production to benchmark things in the real world against the most recent few hundred solutions, so I'm confident that it works robustly. However, I've not written the code. I have tried my hardest to get Claude to write sensible messages and comments that explains stuff, but I'd ask for your patience in the fact I'm sending AI code to you. Everything below here is from your friendly neighbourhood LLM...


What changed

Most of a C++ run goes on compiling the test file, and most of that goes on parsing Catch2's catch_all.hpp. Every test file includes it, and every run parsed it from scratch. The Dockerfile now precompiles that header once, with the same compiler options as the exercises' Debug builds. GCC only uses a precompiled header if it is included before anything else. So when the test file is compiled, a small compiler launcher (bin/compile-with-pch.sh) adds -include catch2/catch_all.hpp. The launcher only changes how the test file is compiled. The solution's .cpp files are compiled as before. If an exercise uses different compiler options (such as a Release build), GCC ignores the precompiled header and parses the header as before.

The test executable is now linked with mold, which links it several times faster than GNU ld.

run.sh now builds only the test executable. The default make target also runs the tests, and run.sh then ran them again to write the XML. So every run executed the tests twice, and slow solutions timed out sooner. If exercism_parser reports an error in the test results (for example after a crash or an uncaught exception), run.sh runs the default target as before, so that the message still contains make's output.

The browser runner uses the same run.sh. It has no precompiled header and no mold, so run.sh only uses them when they are installed, and it compiles and links as before there. The browser runner's sysroot has no grep, so the checks in run.sh for the precompiled header and mold use sed and case.

A few messages change:

  • After a compile error, the last make lines name the test target instead of all (for example make: *** [Makefile:124: two-fer] Error 2). The compiler's own errors are unchanged.
  • Undefined references are reported by mold instead of GNU ld.
  • Catch2 is now included first in the test file, and it brings in standard headers such as <string>. A header-only solution missing one of these includes will now compile, where it failed before. If the solution has a .cpp file, compiling that file will still catch the missing include.

The image grows from 583 MB to 709 MB, mostly because of the precompiled header.

This has been tested in production, and has reduced the median run from 3.95s to 2.3s.

🤖 Generated with Claude Code

iHiD and others added 3 commits November 18, 2026 17:06
The default make target builds the test executable and then runs it. run.sh then ran it a second time to write the XML. Building only the executable halves the time spent running tests. If the parser reports an error, run.sh runs the default target as before, so that crash messages keep make's output.

Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every test file includes catch2/catch_all.hpp, which took about a second to parse on every run. The image now holds a precompiled copy, and a compiler launcher makes the test file include it first. The solution's files compile unchanged. The browser runner has no precompiled header and compiles as before.

Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GNU ld took about 0.3 s to link each test executable, and mold takes about 0.06 s. run.sh only asks for mold when it is installed, so the browser runner links as before.

Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ihid-assistant
ihid-assistant Bot requested a review from a team as a code owner November 18, 2026 08:15

@ahans ahans left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good effort, I always like when things get faster. However, here I find it questionable if this is the right approach. Is touching test-specific files is off-limits? If not, I think cleaning up CMakeList.txts of the tests (over in cpp) and changing the tests to not include catch_all.hpp, but only what they actually use, would be better. It would probably make sense to get rid of the bundled Catch in that context and have CMake find it. We would have to change the instructions to require students to install it locally. The test runner would then also install a catch locally. We could link against Catch2::Catch2WithMain and would maybe save some more time.

We could also move the track to C++20 at the same time.

In general, I'd appreciate some discussion beforehand. I wasn't aware that runtimes are an issue. It seemed to always be about image sizes. Now this PR almost halves the runtime, but more than doubles the image size. With some thought, I think we can have both, lower runtimes and lower (or at least identical) images sizes, and leave the track in a more maintainable state in the process.

Anyway, if there's interest, I can create a few tickets here and over at cpp and draft some PRs.

Comment thread bin/run.sh
# mold links the test executable in about 0.06 s, against about 0.3 s for
# GNU ld, because it uses all cores and does less work per symbol.
if command -v mold > /dev/null; then
cmake_options="${cmake_options} -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=mold"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be cleaner:

Suggested change
cmake_options="${cmake_options} -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=mold"
cmake_options="${cmake_options} -DCMAKE_LINKER_TYPE=MOLD"

In general, doing things in the CMakeLists.txt I would find preferable, since then everyone benefits from improvements, not just the Docker runner.

Comment thread bin/run.sh
# write the XML. Slow solutions took twice as long and timed out sooner.
# An exercise with no target named after the slug builds everything as before.
# The browser runner has no grep, so this uses sed.
built_target_only=false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is known and something I found surprising when starting working on exercises and track maintenance. I think we could discuss if we'd want to change that. Normally, you build a project with a plain make. To run anything, you'd have to explicitly say so. Or, maybe even better, we could register the test as an actual CMake test and then use ctest. But again, that would require changing all the CMakeLists.txt files.

Tests are typically fast, so I never considered this an issue. But I do see that for slow solutions we pay too much. However, I don't see why it needs to be so complicated. Always doing

make "${slug}" 2> "${compilation_errors_file_name}"

followed by the line that produces the XML should work? All CMakeLists.txt are identical, so assuming that there is a target named ${slug} is save.

Comment thread Dockerfile
# these are the options used by the exercises' CMakeLists.txt in Debug mode.
# -Wpedantic matters because it changes which macros GCC defines.
# bin/compile-with-pch.sh makes the test file include it first.
RUN c++ -std=c++17 -g -Wall -Wextra -Wpedantic -Werror -x c++-header \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds very fragile. I guess there's a reason why pre-compiled headers are not exactly popular ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants