Skip to content

Architecture pass fixes: build tool - #477

Open
dimensionscape wants to merge 5 commits into
mainfrom
fix/arch-pass-4
Open

dimensionscape wants to merge 5 commits into
mainfrom
fix/arch-pass-4

Conversation

@dimensionscape

Copy link
Copy Markdown
Member

The build tool fixes from the architecture pass in #154. Each was shown on Windows x64, before and after.

Fixes #469
Fixes #470
Fixes #471
Fixes #472

Two ways a parallel compile went wrong when something on a compile
thread threw or exited:

- When a compile failed, Compiler deleted the old object before calling
  setThreadError. If the delete threw, as it does for a read-only object
  or one another process has open on Windows, the error was never
  recorded, and the thread's catch only warned when no error had been
  recorded, so the build linked the old object and exited 0. The error
  is recorded first, the delete is guarded, and the catch records an
  error for anything thrown.
- Tools.exit on a compile thread ran the exit hooks there, and
  ThreadPool's hook waits on the calling thread's own queue, which
  never fills, while the main thread waited for that thread: the build
  hung. HXCPP_EXIT_ON_ERROR on a compile error and any Log.error on a
  compile thread did this. On a compile thread, Tools.exit records the
  code and ends the job, and the main thread exits when the jobs have.

HXCPP_EXIT_ON_ERROR is also read once the arguments and Options.txt
are in, not only from the environment.

Test: test/compile_errors/check.sh, run in CI, builds three projects
that must fail: HXCPP_EXIT_ON_ERROR with a compile error and two
optimisation tags on one file, which hung before, and on Windows a
compile error over a read-only old object, which exited 0 before. All
three fail within seconds now, on Windows x64.

Fixes #471
Without the compile cache, the copy of Options.txt the build tool keeps
in the obj folder is all that tells it the compile flags changed, and
filterOptions keeps only defines containing "hxcpp" and a few others.
Defines the toolchains compile by, or choose files by, were dropped:
switching one reused objects built without it. Adding CPPIA_NO_JIT to
a -D scriptable build recompiled only Main.cpp and failed to link
against the old Cppia.obj (unresolved hx::getJitType), and removing it
linked a host without its JIT, saying nothing.

The defines the toolchain files test or use in flags, and which no obj
folder or other kept define already covers, are kept too: CPPIA_NO_JIT,
CPPIA_JIT, ABI, objc, OBJC_ARC, STDLIBCPP, exe_link, O3,
LEGACY_MACOSX_SDK, armv6, rpi, winrt, noM32, noM64 and
no_shared_libs.

Test: a -D scriptable build, then the same with -D CPPIA_NO_JIT,
compiled the cppia sources again and linked, then without it again,
on Windows x64.

Fixes #470
- A file with its own cxx_standard, which docs/build_xml/Files.md
  documents, in a group with a precompiled header was given both the
  file's standard and the PCH built with the group's. MSVC refused
  (C2855) and clang refuses a PCH built with other language options.
  Such a file compiles without the PCH.
- group.mTags=="nvcc"; compared instead of assigning, so an <nvcc/>
  group kept the tags haxe,static and was given the host compiler's
  flags, which nvcc takes only through -Xcompiler. The tag is set.
- Setup looked for NDKV6 to NDKV19 only, so -DNDKV25 was ignored and
  the newest NDK under ANDROID_NDK_DIR used. Any NDKV<n> is read, the
  lowest if several are given. An NDK without meta/platforms.json then
  stopped the build with "Invalid field access : toString" while
  warning about it; the warning prints now.

Test, on Windows x64: a file with cxx_standard='17' in a PCH group
compiles and runs; a stand-in nvcc.exe that prints its arguments gets
no host flags; with fake NDKs 25 and 28, -DNDKV25 picks 25 and
-DNDKV28 picks 28.

Fixes #472
The build tool decides what to rebuild by comparing files against
haxe-target.xml's depend lists, and the compile cache key hashes only
the files listed. Headers hxcpp.h includes, directly or not, were
missing from hxcpp-depends (hx/Functions.h, hx/FieldRef.h,
hx/ErrorCodes.h, hx/Boot.h, hx/Undefine.h, cpp/CppInt32__.h,
cpp/FastIterator.h, cpp/marshal/RootHandle.hpp and hx/Invoker.h), and
headers the runtime's sources include were missing from its group
(hx/DynamicImpl.h, hx/Memory.h, hx/NekoFunc.h, hx/CFFIJsPrime.h,
hx/CFFINekoLoader.h, hx/TelemetryTracy.h and src/hx/FastMutex.h). So
editing one, as an update of a haxelib git or dev checkout does,
rebuilt nothing, and the cache relinked objects built before it.

They are listed, and tools/check-depends.py, run in the Check headers
job, fails when a header either reaches is missing again.

Test: #error appended to include/hx/Functions.h now fails the next
build, where it rebuilt nothing and exited 0 before; check-depends.py
passes, and failed on main.

Fixes #469
Before every build with the compile cache, the build tool stat'ed
every file in it to keep it under HXCPP_CACHE_MB, even when the build
had nothing to compile. With tens of thousands of files that took up
to seconds a build. It trims when the cache's .trimmed stamp is older
than ten minutes, and writes the stamp when it does.

Measured with HXCPP_TIMES, a build with nothing to compile and a cache
of 29792 files, three runs each on Windows x64: "setup cache" took
1138, 142 and 98 ms before, and 162 (the trim), 14 and 3 ms after.

The other fixed cost in #473, running the MSVC setup script on every
build (2.7 to 9.4 s in the same runs), is not changed here.

Part of #473
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant