Skip to content

Bug: mon.csv write race causes flaky parallel ctest failures #390

@lukelowry

Description

@lukelowry

Description how to reproduce the bug

Failure mode varies between runs: Failed, SEGFAULT, or Subprocess aborted. I got each at least once. Sequential (-j 1) always passes. Re-running the parallel command in a loop reproduces within a few iterations.

$ ctest --test-dir build -j10 -R "TwoBusBasic"
...
The following tests FAILED:
	 17 - TwoBusBasic_using_app (Subprocess aborted)
Details
$ ctest --test-dir build -j$(nproc) -R "TwoBusBasic" --output-on-failure
...
17: Test command: /home/lukel/GridKit/build/application/PhasorDynamics/PDSim "TwoBusBasic.solver.json"
17: Working Directory: /home/lukel/GridKit/build/examples/PhasorDynamics/Tiny/TwoBus/Basic
17: [ERROR] Files "mon.csv" and "TwoBusBasic.ref.csv" have different number of variables.
17: terminate called after throwing an instance of 'std::length_error'
17:   what():  cannot create std::vector larger than max_size()

The following tests FAILED:
	 17 - TwoBusBasic_using_app (Subprocess aborted)

GridKit™ version

develop @ aaf6f81 ("Add INSTALL.md file. (#383)")

System and environment details

  • OS: Ubuntu 24.04.3 LTS on WSL2 (kernel 6.6.87.2-microsoft-standard-WSL2)
  • CPU: 12th Gen Intel Core i9-12900K (24 logical cores)
  • Compiler: clang/clang++ 16.0.6 (Ubuntu 16.0.6-23ubuntu4)
  • CMake: 3.28.3
  • SUNDIALS: 7.4.0 with KLU sparse support
  • Enzyme: via Spack env
  • Build preset: enzyme (RelWithDebInfo, -march=haswell -mtune=native)

Additional information

Affected directories (any with multiple tests sharing one mon.csv in the same WORKING_DIRECTORY):

  • examples/PhasorDynamics/Tiny/TwoBus/Basic/ — tests 14–17
  • examples/PhasorDynamics/Tiny/ThreeBus/Basic/ — tests 24–27

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions