Skip to content

Conversation

@FlorianDeconinck
Copy link
Contributor

Description

Move maps to Sequential when GT4PY_CARTESIAN_DEACTIVATE_THREADING=True for dace:X stencil backends

Requirements

  • All fixes and/or new features come with corresponding tests.

@FlorianDeconinck
Copy link
Contributor Author

@edopao This PR allows us to deactivate threading on the dace:X backends which seems to be at the heart of the issues we see on the daily CI + MacOS.

It's on an env var: GT4PY_CARTESIAN_DEACTIVATE_THREADING=True

Can you help out modify the CI config and do a dry run to see if it helps?

@edopao
Copy link
Contributor

edopao commented Jan 14, 2026

I have tried setting this variable on the branch dace_ci_cartesian, but the CI still fails:
https://github.com/GridTools/gt4py/actions/runs/20986218797/job/60320828991

tests/cartesian_tests/integration_tests/multi_feature_tests/test_code_generation.py::test_k_only_access_stencil[dace:cpu_kfirst] OMP: Error #179: Function pthread_key_create failed:
OMP: System error #35: Resource temporarily unavailable

I can see from the log that the variable is set:

Run uv python upgrade 3.14
  uv python upgrade 3.14
  ./noxfile.py -s 'test_cartesian-3.14' -t 'cpu'
  shell: /bin/bash -e {0}
  env:
    UV_CACHE_DIR: /Users/runner/work/_temp/setup-uv-cache
    CC: gcc
    CXX: g++
    CMAKE_PREFIX_PATH: /opt/homebrew/opt/libomp
    CXXFLAGS: -Wno-missing-template-arg-list-after-template-kw
    OPENMP_CPPFLAGS: -Xclang -fopenmp -Xclang -I/opt/homebrew/opt/libomp/include
    OPENMP_LDFLAGS: -Xclang -fopenmp /opt/homebrew/opt/libomp/lib/libomp.dylib
    GT4PY_CARTESIAN_DEACTIVATE_THREADING: 1
    UV_RESOLUTION: highest
    UV_PYTHON_PREFERENCE: only-managed

I have also tried with GT4PY_CARTESIAN_DEACTIVATE_THREADING=True to be sure, although I see from the code that it makes no difference.

It seems to me that after the first error of this kind, the pytest worker cannot recover and all tests on this pytest worker fail. Could this suggest that some memory is corrupted / in wrong state?

@edopao
Copy link
Contributor

edopao commented Jan 14, 2026

What is strange to me is that this problem only happens with Python "highest" strategy for dependency resolution, so there might be some newer python package that somehow leads to different generated code. I will try locally on my Mac.

@edopao
Copy link
Contributor

edopao commented Jan 14, 2026

I don't see this problem locally on my Mac, using OpenMP from brew installation as in the CI job.

I will now try to use a newer MacOS image in the CI, which is the next candidate release (macos-26).

@edopao
Copy link
Contributor

edopao commented Jan 14, 2026

I don't see this problem locally on my Mac, using OpenMP from brew installation as in the CI job.

I will now try to use a newer MacOS image in the CI, which is the next candidate release (macos-26).

Unfortunately, it still fails on macos-26. Next test, I could switch to gcc, installed through brew.

@FlorianDeconinck
Copy link
Contributor Author

What is strange to me is that this problem only happens with Python "highest" strategy for dependency resolution, so there might be some newer python package that somehow leads to different generated code. I will try locally on my Mac.

So this was my first hunch, too but the code are exactly the same between those versions. I am on Linux and can't reproduce either

@edopao
Copy link
Contributor

edopao commented Jan 14, 2026

I don't see this problem locally on my Mac, using OpenMP from brew installation as in the CI job.
I will now try to use a newer MacOS image in the CI, which is the next candidate release (macos-26).

Unfortunately, it still fails on macos-26. Next test, I could switch to gcc, installed through brew.

I still see the same issue using gcc instead of default clang compiler:
what(): thread_specific_storage constructor: could not initialize the TSS key!

@FlorianDeconinck
Copy link
Contributor Author

If this is truly a number of threads problem coming from a combination of newer packages + interpreter + our code, can we split the tests?
We can run:

  • ./tests/cartesian_tests/unit_tests
  • ./tests/cartesian_tests/integration_tests/feature_tests
  • ./tests/cartesian_tests/integration_tests/multi_feature_tests

For our team, the fact that MacOS doesn't know how to threads is pretty low in the stack of issues...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants