Repository navigation
Architecture pass fixes: build tool - #477
Open
dimensionscape wants to merge 5 commits into
Open
dimensionscape wants to merge 5 commits into
dimensionscape wants to merge 5 commits into
Conversation
Two ways a parallel compile went wrong when something on a compile thread threw or exited: - When a compile failed, Compiler deleted the old object before calling setThreadError. If the delete threw, as it does for a read-only object or one another process has open on Windows, the error was never recorded, and the thread's catch only warned when no error had been recorded, so the build linked the old object and exited 0. The error is recorded first, the delete is guarded, and the catch records an error for anything thrown. - Tools.exit on a compile thread ran the exit hooks there, and ThreadPool's hook waits on the calling thread's own queue, which never fills, while the main thread waited for that thread: the build hung. HXCPP_EXIT_ON_ERROR on a compile error and any Log.error on a compile thread did this. On a compile thread, Tools.exit records the code and ends the job, and the main thread exits when the jobs have. HXCPP_EXIT_ON_ERROR is also read once the arguments and Options.txt are in, not only from the environment. Test: test/compile_errors/check.sh, run in CI, builds three projects that must fail: HXCPP_EXIT_ON_ERROR with a compile error and two optimisation tags on one file, which hung before, and on Windows a compile error over a read-only old object, which exited 0 before. All three fail within seconds now, on Windows x64. Fixes #471
Without the compile cache, the copy of Options.txt the build tool keeps in the obj folder is all that tells it the compile flags changed, and filterOptions keeps only defines containing "hxcpp" and a few others. Defines the toolchains compile by, or choose files by, were dropped: switching one reused objects built without it. Adding CPPIA_NO_JIT to a -D scriptable build recompiled only Main.cpp and failed to link against the old Cppia.obj (unresolved hx::getJitType), and removing it linked a host without its JIT, saying nothing. The defines the toolchain files test or use in flags, and which no obj folder or other kept define already covers, are kept too: CPPIA_NO_JIT, CPPIA_JIT, ABI, objc, OBJC_ARC, STDLIBCPP, exe_link, O3, LEGACY_MACOSX_SDK, armv6, rpi, winrt, noM32, noM64 and no_shared_libs. Test: a -D scriptable build, then the same with -D CPPIA_NO_JIT, compiled the cppia sources again and linked, then without it again, on Windows x64. Fixes #470
- A file with its own cxx_standard, which docs/build_xml/Files.md documents, in a group with a precompiled header was given both the file's standard and the PCH built with the group's. MSVC refused (C2855) and clang refuses a PCH built with other language options. Such a file compiles without the PCH. - group.mTags=="nvcc"; compared instead of assigning, so an <nvcc/> group kept the tags haxe,static and was given the host compiler's flags, which nvcc takes only through -Xcompiler. The tag is set. - Setup looked for NDKV6 to NDKV19 only, so -DNDKV25 was ignored and the newest NDK under ANDROID_NDK_DIR used. Any NDKV<n> is read, the lowest if several are given. An NDK without meta/platforms.json then stopped the build with "Invalid field access : toString" while warning about it; the warning prints now. Test, on Windows x64: a file with cxx_standard='17' in a PCH group compiles and runs; a stand-in nvcc.exe that prints its arguments gets no host flags; with fake NDKs 25 and 28, -DNDKV25 picks 25 and -DNDKV28 picks 28. Fixes #472
The build tool decides what to rebuild by comparing files against haxe-target.xml's depend lists, and the compile cache key hashes only the files listed. Headers hxcpp.h includes, directly or not, were missing from hxcpp-depends (hx/Functions.h, hx/FieldRef.h, hx/ErrorCodes.h, hx/Boot.h, hx/Undefine.h, cpp/CppInt32__.h, cpp/FastIterator.h, cpp/marshal/RootHandle.hpp and hx/Invoker.h), and headers the runtime's sources include were missing from its group (hx/DynamicImpl.h, hx/Memory.h, hx/NekoFunc.h, hx/CFFIJsPrime.h, hx/CFFINekoLoader.h, hx/TelemetryTracy.h and src/hx/FastMutex.h). So editing one, as an update of a haxelib git or dev checkout does, rebuilt nothing, and the cache relinked objects built before it. They are listed, and tools/check-depends.py, run in the Check headers job, fails when a header either reaches is missing again. Test: #error appended to include/hx/Functions.h now fails the next build, where it rebuilt nothing and exited 0 before; check-depends.py passes, and failed on main. Fixes #469
Before every build with the compile cache, the build tool stat'ed every file in it to keep it under HXCPP_CACHE_MB, even when the build had nothing to compile. With tens of thousands of files that took up to seconds a build. It trims when the cache's .trimmed stamp is older than ten minutes, and writes the stamp when it does. Measured with HXCPP_TIMES, a build with nothing to compile and a cache of 29792 files, three runs each on Windows x64: "setup cache" took 1138, 142 and 98 ms before, and 162 (the trim), 14 and 3 ms after. The other fixed cost in #473, running the MSVC setup script on every build (2.7 to 9.4 s in the same runs), is not changed here. Part of #473
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The build tool fixes from the architecture pass in #154. Each was shown on Windows x64, before and after.
HXCPP_EXIT_ON_ERROR, or an error such as two optimisation tags on one file) hung the build.HXCPP_EXIT_ON_ERRORalso works as a-Ddefine now. Test: test/compile_errors/check.sh, a new CI step, which catches all three against main's build tool.CPPIA_NO_JIT,ABI,OBJC_ARCand others, were not compared without the compile cache, so switching one reused stale objects; addingCPPIA_NO_JITfailed to link.cxx_standardin a precompiled header group failed to compile, an<nvcc/>group got the host compiler's flags, and-DNDKV20and up were ignored. An NDK without meta/platforms.json no longer stops the build with "Invalid field access".hxcpp.hand the runtime include were missing from the depend lists, so editing one rebuilt nothing. They are listed, and tools/check-depends.py, a new step in the Check headers job, keeps them complete.Fixes #469
Fixes #470
Fixes #471
Fixes #472