Kokkos::View related segfaults when using the SYCL backend #330

ThomasJannaudCAM · 2025-03-07T12:25:49Z

ThomasJannaudCAM
Mar 7, 2025

Hi all,

In Cambridge we just received some new Intel Ponte Vecchio GPUs. With a software engineer, we have been trying to run IDEFIX on them. She was able to compile Idefix on a number of the test cases using the SYCL backend provided by Kokkos, but at runtime it segfaults with an error about Kokkos::View. We are wondering if this is a fault with Idefix or with Kokkos. She has been able to reproduce it on two different PVC systems (each with slightly different environments - see below).

Thanks in advance !

Below is her description of how to reproduce the error:

In the test/MHD/disk directory type:

cmake ${IDEFIX_DIR} -DIdefix_MPI=ON -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DCMAKE_CXX_COMPILER=mpiicpx

then when it has finished configuring the build type ‘make'.

Run the Idefix executable (I used 4 MPI ranks but it doesn’t really matter, I’ve also tried with 1):

mpirun -np 4 ./idefix

This is the output:


jk945@sw-pvc03:~/idefix/test/MHD/disk $ mpirun -np 4 ./idefix
[0] MPI startup(): ===== GPU topology on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): NUMA nodes     : 2
[0] MPI startup(): GPUs           : 4
[0] MPI startup(): Stacks (Tiles) : 8
[0] MPI startup(): Hierarchy mode : flat
[0] MPI startup(): NUMA Id
GPU Id         Stacks (tiles)                 Ranks on this NUMA
[0] MPI startup(): 0      0,1            (0,1)(2,3)                     0,1
[0] MPI startup(): 1      2,3            (4,5)(6,7)                     2,3
[0] MPI startup(): ===== GPU pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): Rank
Pin stack (tile)
[0] MPI startup(): 0
{0,1}
[0] MPI startup(): 1
{2,3}
[0] MPI startup(): 2
{4,5}
[0] MPI startup(): 3
{6,7}
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[0] MPI startup(): Intel(R) MPI Library, Version 2021.13  Build 20240515 (id: df72937)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[0] MPI startup(): libfabric version: 1.20.1-impi
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400_x1.dat" not found
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400.dat" not found
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx.dat" not found
[0] MPI startup(): Load tuning file: "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): Number of NICs:  1 
[0] MPI startup(): ===== NIC pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): Rank    Thread id  Pin nic
[0] MPI startup(): 0       0          mlx
[0] MPI startup(): 1       0          mlx
[0] MPI startup(): 2       0          mlx
[0] MPI startup(): 3       0          mlx
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank    Pid      Node name                 Pin cpu
[0] MPI startup(): 0       2296614  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27}
[0] MPI startup(): 1       2296615  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54
                                                ,55}
[0] MPI startup(): 2       2296616  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82
                                                ,83}
[0] MPI startup(): 3       2296617  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,
                                                108,109,110,111}
                                  .:HMMMMHn:.  ..:n..
                                .H*'``     `'%HM'''''!x.
         :x                    x*`           .(MH:    `#h.
        x.`M                   M>        :nMMMMMMMh.     `n.
         *kXk..                XL  nnx:.XMMMMMMMMMMML   .. 4X.
          )MMMMMx              'M   `^?M*MMMMMMMMMMMM:HMMMHHMM.
          MMMMMMMX              ?k    'X ..'*MMMMMMM.#MMMMMMMMMx
         XMMMMMMMX               4:    M:MhHxxHHHx`MMx`MMMMMMMMM>
         XM!`   ?M                `x   4MM'`''``HHhMMX  'MMMMMMMM
         4M      M                 `:   *>     `` .('MX   '*MMMM'
          MX     `X.nnx..                        ..XMx`     'M*X
           ?h.    ''```^'*!Hx.     :Mf     xHMh  M**MMM      4L`
            `*Mx           `'*n.x. 4M>   :M` `` 'M    `       %
             '%                ``*MHMX   X>      !
            :!                    `#MM>  X>      `   :x
           :M                        ?M  `X     .  ..'M
           XX                       .!*X  `x   XM( MMx`h
          'M>::                        `M: `+  MMX XMM `:
          'M> M                         'X    'MMX ?MMk.Xx..
          'M> ?L                     ...:!     MMX.H**'MMMM*h
           M>  #L                  :!'`MM.    . X*`.xHMMMMMnMk.
           `!   #h.      :L           XM'*hxHMM*MhHMMMMMMMMMM'#h
           +     XMh:    4!      x   :f   MM'   `*MMMMMMMMMM%  `X
           M     Mf``tHhxHM      M>  4k xxX'      `#MMMMMMMf    `M .>
          :f     M   `MMMMM:     M>   M!MMM:         '*MMf'     'MH*
          !     Xf   'MMMMMX     `X   X>'h.`          :P*Mx.   .d*~..
        :M      X     4MMMMM>     !   X~ `Mh.      .nHL..M#'%nnMhH!'`
       XM      d>     'X`'**h     'h  M   ^'MMHH+*'`  ''''   `'**'
    %nxM>      *x+x.:. XL.. `k     `::X
:nMMHMMM:.  X>  Mn`*MMMMMHM: `:     ?MMn.
    `'**MML M>  'MMhMMMMMMMM  #      `M:^*x
         ^*MMttnnMMMMMMMMMMMH>.        M:.4X
                        `MMMM>X   (   .MMM:MM!   .
                          `'''4x.dX  +^ `''MMMMHM?L..
                                ``'           `'`'`'`

              Idefix version 2.2.00-54482d58
              Built against Kokkos 40500
              Compiled on Mar  5 2025 at 17:00:09


Main: initialization stage.
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x14acd1a29d90] 
[0x14acd1a29e40] __libc_start_main
[0x40eec5] 
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x15455c029d90] 
[0x15455c029e40] __libc_start_main
[0x40eec5] 
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x14f2af629d90] 
[0x14f2af629e40] __libc_start_main
[0x40eec5] 
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x151e84429d90] 
[0x151e84429e40] __libc_start_main
[0x40eec5]

I do not see this error with the CUDA backend.

Here are the versions of Intel oneAPI packages that I have loaded:

jk945@sw-pvc03:~/idefix/test/MHD/disk $ module list
Currently Loaded Modulefiles:
 1) intel/tbb/2021.13                          
 2) intel/compiler-rt/2024.2.0                 
 3) intel/compiler-intel-llvm/2024.2.0         
 4) intel/mpi/2021.13                          
 5) intel/mkl/2024.2                           
 6) glibc/2.35/gcc/spmhgmal                    
 7) intel-oneapi-dpl/2022.5.0/oneapi/jxepuwvw  
 8) ncurses/6.2/gcc-7.2.0-awbmlbc              
 9) cmake/3.22.3/gcc-7.2.0-ffrdpvs

glesur · 2025-03-07T12:45:21Z

glesur
Mar 7, 2025
Maintainer

can't really help much as we don't have any intel gpu here. A few thoughts:

you should try to run in serial first (ie without MPI enabled)
try to enable IDEFIX_Debug=ON (in addition to KOKKOS_Debug, as done above)
can you tell me the output of cmake (when called from a clean build directory)

1 reply

glesur Mar 9, 2025
Maintainer

additional thoughts: maybe add -DKokkos_ENABLE_SERIAL=ON to cmake, not sure it is enabled by default with syscl.

julianakwan · 2025-03-11T11:25:51Z

julianakwan
Mar 11, 2025

Hi @glesur - thanks very much for your suggestions.
I have tried a serial build with both Idefix and Kokkos debugging enabled but the same error with Kokkos::View still occurs.
This is the line I use to compile:
cmake ${IDEFIX_DIR} -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DKokkos_ENABLE_DEBUG=ON -DIdefix_DEBUG=ON

Here is the output from cmake

-- The C compiler identification is IntelLLVM 2025.0.0
-- The CXX compiler identification is IntelLLVM 2025.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icpx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting default Kokkos CXX standard to 17
-- Kokkos version: 4.5.0
-- The project name is: Kokkos
-- Using internal gtest for testing
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=gnu++17 for C++17 extensions as feature
-- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL
-- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL - found
-- Looking for SYCL_EXT_ONEAPI_GRAPH
-- Looking for SYCL_EXT_ONEAPI_GRAPH - found
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::SYCL
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
--  INTEL_PVC
-- Found TPLLIBDL: /usr/include  
-- Looking for C++ include oneapi/dpl/execution
-- Looking for C++ include oneapi/dpl/execution - found
-- Looking for C++ include oneapi/dpl/algorithm
-- Looking for C++ include oneapi/dpl/algorithm - found
-- Performing Test KOKKOS_NO_TBB_CONFLICT
-- Performing Test KOKKOS_NO_TBB_CONFLICT - Success
-- Using internal desul_atomics copy
-- Experimental mdspan support is enabled
-- Looking for C++ include experimental/mdspan
-- Looking for C++ include experimental/mdspan - not found
-- Looking for C++ include mdspan
-- Looking for C++ include mdspan - not found
-- Using internal mdspan directory /lus/flare/projects/grteclyn/idefix/src/kokkos/tpls/mdspan/include
-- Kokkos Backends: SERIAL;SYCL
-- Including problem-specific CMakeLists in '/lus/flare/projects/grteclyn/idefix/test/MHD/sod'
--     Explicitely setting Idefix_MHD = ON
-- Idefix final configuration
--     MHD:  ON
--     MPI:  OFF
--     HDF5: OFF
--     Python: OFF
--     Reconstruction: Linear
--     Precision: Double
--     Version: 2.2.00-54482d58
--     Problem definitions: 'definitions.hpp'
-- Configuring done (27.4s)
-- Generating done (2.8s)

Then I build using make and everything is fine at this stage.
When I run the executable, this is the output:

./idefix
----> Profiler::Init...
----> ...returned
                                  .:HMMMMHn:.  ..:n..
                                .H*'``     `'%HM'''''!x.
         :x                    x*`           .(MH:    `#h.
        x.`M                   M>        :nMMMMMMMh.     `n.
         *kXk..                XL  nnx:.XMMMMMMMMMMML   .. 4X.
          )MMMMMx              'M   `^?M*MMMMMMMMMMMM:HMMMHHMM.
          MMMMMMMX              ?k    'X ..'*MMMMMMM.#MMMMMMMMMx
         XMMMMMMMX               4:    M:MhHxxHHHx`MMx`MMMMMMMMM>
         XM!`   ?M                `x   4MM'`''``HHhMMX  'MMMMMMMM
         4M      M                 `:   *>     `` .('MX   '*MMMM'
          MX     `X.nnx..                        ..XMx`     'M*X
           ?h.    ''```^'*!Hx.     :Mf     xHMh  M**MMM      4L`
            `*Mx           `'*n.x. 4M>   :M` `` 'M    `       %
             '%                ``*MHMX   X>      !
            :!                    `#MM>  X>      `   :x
           :M                        ?M  `X     .  ..'M
           XX                       .!*X  `x   XM( MMx`h
          'M>::                        `M: `+  MMX XMM `:
          'M> M                         'X    'MMX ?MMk.Xx..
          'M> ?L                     ...:!     MMX.H**'MMMM*h
           M>  #L                  :!'`MM.    . X*`.xHMMMMMnMk.
           `!   #h.      :L           XM'*hxHMM*MhHMMMMMMMMMM'#h
           +     XMh:    4!      x   :f   MM'   `*MMMMMMMMMM%  `X
           M     Mf``tHhxHM      M>  4k xxX'      `#MMMMMMMf    `M .>
          :f     M   `MMMMM:     M>   M!MMM:         '*MMf'     'MH*
          !     Xf   'MMMMMX     `X   X>'h.`          :P*Mx.   .d*~..
        :M      X     4MMMMM>     !   X~ `Mh.      .nHL..M#'%nnMhH!'`
       XM      d>     'X`'**h     'h  M   ^'MMHH+*'`  ''''   `'**'
    %nxM>      *x+x.:. XL.. `k     `::X
:nMMHMMM:.  X>  Mn`*MMMMMHM: `:     ?MMn.
    `'**MML M>  'MMhMMMMMMMM  #      `M:^*x
         ^*MMttnnMMMMMMMMMMMH>.        M:.4X
                        `MMMM>X   (   .MMM:MM!   .
                          `'''4x.dX  +^ `''MMMMHM?L..
                                ``'           `'`'`'`

              Idefix version 2.2.00-54482d58
              Built against Kokkos 40500
              Compiled on Mar 11 2025 at 11:14:32


Main: initialization stage.
----> Grid::Grid(Input)...
----> ...returned
----> GridHost::GridHost(Grid)...
----> ...returned
----> GridHost::MakeGrid...
----> ...returned
----> GridHost::SyncToDevice...
----> ...returned
----> DataBlock::DataBlock...
--------> GridHost::GridHost(Grid)...
--------> ...returned
--------> GridHost::SyncFromDevice...
--------> ...returned
--------> DataBlock::ExtractSubdomain...
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
--------> ...returned
--------> DataBlock::MakeGeometry()...
------------> idefix_for(Volumes)...
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_dx0")
Backtrace:
[0x6069d0] 
[0x5e02d2] 
[0x411d76] 
[0x41af1d] 
[0x51e5c8] 
[0x45879b] 
[0x5d0c2a] 
[0x14d60757124d] __libc_start_main
[0x40ef0a] 
Aborted

6 replies

julianakwan Mar 11, 2025

Hi @glesur - I'm happy to report that that worked! There was a lot of output because I compiled on debug so let me know if you want to see it. I also checked with MPI (but always 1 MPI rank to 1 GPU) and it was fine as well. Thanks so much!

ThomasJannaudCAM Mar 11, 2025
Author

Thank you both very much !

glesur Mar 12, 2025
Maintainer

Excellent, I'll propagate the fix to the develop branch. @ThomasJannaudCAM , whenever you have time, I'd like to know how it performs on Intel PVCs on the OT3D test with 128^3 points and no tracer.

ThomasJannaudCAM Mar 12, 2025
Author

What would you want ? Frequency of cell updates and parallel efficiency ?

glesur Mar 12, 2025
Maintainer

yes, cell update/s (don't really care about the parallel efficiency for now), as in https://idefix.readthedocs.io/latest/performances.html

glesur · 2025-03-13T20:22:02Z

glesur
Mar 13, 2025
Maintainer

A fix for this has been merged into develop (#331) and will be part of the next Idefix release.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Kokkos::View related segfaults when using the SYCL backend #330

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 7 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Kokkos::View related segfaults when using the SYCL backend #330

Uh oh!

ThomasJannaudCAM Mar 7, 2025

Replies: 3 comments · 7 replies

Uh oh!

glesur Mar 7, 2025 Maintainer

Uh oh!

glesur Mar 9, 2025 Maintainer

Uh oh!

julianakwan Mar 11, 2025

Uh oh!

julianakwan Mar 11, 2025

Uh oh!

ThomasJannaudCAM Mar 11, 2025 Author

Uh oh!

glesur Mar 12, 2025 Maintainer

Uh oh!

ThomasJannaudCAM Mar 12, 2025 Author

Uh oh!

glesur Mar 12, 2025 Maintainer

Uh oh!

glesur Mar 13, 2025 Maintainer

ThomasJannaudCAM
Mar 7, 2025

Replies: 3 comments 7 replies

glesur
Mar 7, 2025
Maintainer

glesur Mar 9, 2025
Maintainer

julianakwan
Mar 11, 2025

ThomasJannaudCAM Mar 11, 2025
Author

glesur Mar 12, 2025
Maintainer

ThomasJannaudCAM Mar 12, 2025
Author

glesur Mar 12, 2025
Maintainer

glesur
Mar 13, 2025
Maintainer