Openacc test main #693

basava70 · 2025-02-25T11:25:30Z

PR to test OpenACC compilation and running in main branch.

JanStreffing · 2025-02-25T11:28:17Z

GCC compile fails with:

[ 53%] Building Fortran object src/CMakeFiles/fesom.dir/oce_tracer_mod.F90.o
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:30:0:

   30 | #endif
      | 
Error: #endif without #if
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:41:0:

   41 | #endif
      | 
Error: #endif without #if
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:28:6:

   28 |     #ifdef ENABLE_OPENACC
      |      1
Error: Invalid character in name at (1)
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:39:6:

   39 |     #ifdef ENABLE_OPENACC
      |      1
Error: Invalid character in name at (1)
make[2]: *** [src/CMakeFiles/fesom.dir/build.make:1492: src/CMakeFiles/fesom.dir/oce_tracer_mod.F90.o] Error 1
make[2]: *** Waiting for unfinished jobs....

See: https://github.com/FESOM/fesom2/pull/693/checks#step:5:207

basava70 · 2025-02-25T11:29:27Z

GCC compile fails with:

[ 53%] Building Fortran object src/CMakeFiles/fesom.dir/oce_tracer_mod.F90.o
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:30:0:

   30 | #endif
      | 
Error: #endif without #if
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:41:0:

   41 | #endif
      | 
Error: #endif without #if
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:28:6:

   28 |     #ifdef ENABLE_OPENACC
      |      1
Error: Invalid character in name at (1)
/__w/fesom2/fesom2/src/oce_tracer_mod.F90:39:6:

   39 |     #ifdef ENABLE_OPENACC
      |      1
Error: Invalid character in name at (1)
make[2]: *** [src/CMakeFiles/fesom.dir/build.make:1492: src/CMakeFiles/fesom.dir/oce_tracer_mod.F90.o] Error 1
make[2]: *** Waiting for unfinished jobs....

See: https://github.com/FESOM/fesom2/pull/693/checks#step:5:207

I am going through them now. Thanks

env/levante.dkrz.de/shell.intel

env/levante.dkrz.de/shell.nvhpc

src/oce_tracer_mod.F90

basava70 · 2025-02-25T12:10:01Z

Is everything good ? @JanStreffing

JanStreffing · 2025-02-25T12:17:06Z

Tests pass. I'm happy. @mandresm and @dsidoren, may comment on the ACC changes.

basava70 · 2025-02-26T11:05:41Z

should I add the job_hpc_levante to the code as well ? I am using that file to run the code.

JanStreffing · 2025-02-26T11:23:12Z

should I add the job_hpc_levante to the code as well ? I am using that file to run the code.

How does it differ from job_levante?

basava70 · 2025-02-26T11:28:50Z

should I add the job_hpc_levante to the code as well ? I am using that file to run the code.

How does it differ from job_levante?

First difference would be it would use gpu's, so there are
#SBATCH --gpus=
#SBATCH --gpus_per_task
and few others
should I post the file as a comment ?
there are also few other export commands, that was needed on levante gpu for ICON to work, else segmentation fault. This file was created in consultation with @suvarchal a few months ago. May you dont need these exports anymore but I am not sure.

JanStreffing · 2025-02-26T11:31:09Z

add as job_gpu_levante

basava70 · 2025-02-26T11:59:12Z

add as job_gpu_levante

Should I also add a comment on how to build ?
./configure.sh levante.nvhpc -DENABLE_OPENACC=ON -DDISABLE_OPENACC_ATOMICS=OFF
currently I use this to build. You think it is necessary to add this a comment in the job_gpu_levante file ?

pgierz · 2025-02-26T14:10:03Z

Hi @basava70, I would put one working runscript and one working compile script into the examples folder so everything is documented. I have not built FESOM outside of the esm-tools framework in a while, but if I remember correctly they also have example build scripts...

basava70 · 2025-06-19T13:14:42Z

I have merged with the latest commits from main. The code is finally compiling with OpenACC and running with both nvhpc and intel compilers on GPU and CPU respectively.
Please review and merge at your latest convenience or let me know your comments and what changes would you like to see.

JanStreffing

Looks to be ready!

env/levante.dkrz.de/shell.nvhpc

suvarchal

could you take out the commits from #697 https://github.com/FESOM/fesom2/pull/697/commits because it is not ready and test again, remaining are optional if you promise to fix them in next PR about to come.

cmake/AddPreprocessTarget.cmake

src/gen_modules_partitioning.F90

suvarchal · 2025-06-19T16:47:57Z

src/oce_adv_tra_driver.F90

+#endif
+
+            !$ACC PARALLEL LOOP GANG VECTOR DEFAULT(PRESENT) VECTOR_LENGTH(acc_vl)
 !$OMP DO


ifndef ENABLE_OPENACC is missing here and few omp blocks below, not sure openmp and openacc will work with this file but if it is too much we can defer to next PR you intend to do.

suvarchal · 2025-06-19T16:49:58Z

src/oce_tracer_mod.F90

+#ifdef ENABLE_OPENACC
 !$ACC end parallel loop
+#endif
 !$OMP PARALLEL DO


not guarded openmp meaning we can´t use openmp with openacc for other parts. can defer to next PR if you want to.

suvarchal · 2025-06-19T16:50:28Z

src/oce_adv_tra_driver.F90

 !$OMP PARALLEL DEFAULT(SHARED) PRIVATE(n, nz)
-            !$ACC PARALLEL LOOP GANG VECTOR DEFAULT(PRESENT) DEFAULT(PRESENT) VECTOR_LENGTH(acc_vl)
+            !$ACC PARALLEL LOOP GANG VECTOR DEFAULT(PRESENT) VECTOR_LENGTH(acc_vl)
 !$OMP DO


not guarded openmp meaning we can´t use openmp with openacc for other parts. can defer to next PR if you want to.

suvarchal · 2025-06-19T17:13:56Z

@basava70 apart from testing for gpu, cpu on levante; can you also enable openmp without openacc and test - in case code is broken as there is lot of ifdefs nested with openmp and openacc. ./configure.sh --clean levante.intel -DENABLE_OPENMP=ON and in job script export OMP_NUM_THREADS=whatever_slurm_cpus_per_task you use before running fesom.x.

mandresm · 2025-06-19T17:17:32Z

Thanks a lot for your review @suvarchal!

…endif loops

(cherry picked from commit 237434c)

basava70 · 2025-06-27T07:03:46Z

Hi all,
I have finished the work I need to do in this branch.
I completed the benchmark comparing runtimes between cpu and gpu versions.

Run Length	Run Length Unit	Steps per Day	CPU (s)	GPU (s)
1	d	32	1.806	125.2253
1	m	32	55.408	4160.025
1	y	32	787.221	> 26400.0
2	y	32	1287.22	NA

Please do review and merge.
Thanks

credits: I used chatgpt to create the table in markdown.

JanStreffing · 2025-06-27T09:39:14Z

@suvarchal, I think this is ready for another round of review.

suvarchal · 2025-06-27T10:25:48Z

@basava70 have you tested with ./configure.sh --clean levante.intel -DENABLE_OPENMP=ON and in job script export OMP_NUM_THREADS=whatever_slurm_cpus_per_task just a check before if it can be merged, i mean with intel or gnu not nvhpc because you changed some definition naming slightly in cmakelists and in code. we can defer both ENABLE_OPENACC and OPENMP (which i think will not work now) to your next
PR.

basava70 · 2025-06-27T10:28:34Z

@suvarchal No, I didn't. I will update you in a couple of hours.

basava70 · 2025-06-27T12:48:07Z

@basava70 have you tested with ./configure.sh --clean levante.intel -DENABLE_OPENMP=ON and in job script export OMP_NUM_THREADS=whatever_slurm_cpus_per_task just a check before if it can be merged, i mean with intel or gnu not nvhpc because you changed some definition naming slightly in cmakelists and in code. we can defer both ENABLE_OPENACC and OPENMP (which i think will not work now) to your next PR.

I could compile it with this option.
OMP_NUM_THREADS=2
--ntasks=256
--ntasks-per-node=64
--cpus-per-task=2

cpu time = 92.67 seconds
run_length = 1
run_length_unit = 'm'

JanStreffing · 2025-06-29T06:30:45Z

Time to merge?

JanStreffing self-requested a review February 25, 2025 11:29

JanStreffing requested changes Feb 25, 2025

View reviewed changes

env/levante.dkrz.de/shell.intel Show resolved Hide resolved

env/levante.dkrz.de/shell.nvhpc Show resolved Hide resolved

src/oce_tracer_mod.F90 Outdated Show resolved Hide resolved

JanStreffing requested review from dsidoren and mandresm February 25, 2025 11:33

JanStreffing assigned basava70 Feb 25, 2025

JanStreffing added the enhancement New feature or request label Feb 25, 2025

basava70 requested a review from JanStreffing February 26, 2025 04:58

basava70 requested a review from suvarchal June 19, 2025 13:10

JanStreffing approved these changes Jun 19, 2025

View reviewed changes

env/levante.dkrz.de/shell.nvhpc Outdated Show resolved Hide resolved

suvarchal requested changes Jun 19, 2025

View reviewed changes

seshadri levante added 7 commits June 19, 2025 20:00

Removed commits from preprocess target PR

bb16869

Working OpenACC code after commenting 'set(CMAKE_EXE_LINKER_FLAGS)'

8d77d94

Removed some commits from #697 PR

5e5d394

Updated with Miguels's comment

729aa7f

Now, we can compile on levante with intel and gnu

980a688

Added suvi's changes back to env.sh (compilerId regex for levante)

6613e88

Solved some errors caused in oce_adv_tra_driver.F90

1c53781

seshadri levante and others added 7 commits June 19, 2025 20:05

Working OpenACC code

5af17ab

Removed indent for #ifdef and #ifndef ENALBE_OPENACC and respective #…

90e19b6

…endif loops

Removed one more indent

2acb0c3

Added job_gpu_levante to run jobs with OpenACC on levante

714a8ee

oops a trailing fortran comment bad for preprocessing

2a1f5b4

(cherry picked from commit 237434c)

Compiling and running base OpenACC code with nvhpc.

108b152

Removed the commented export in shell.nvhpc

5b7447f

basava70 force-pushed the openacc_test_main branch from f92e83e to 5b7447f Compare June 19, 2025 18:10

Merge branch 'main' into openacc_test_main

90e1cbc

JanStreffing requested a review from suvarchal June 27, 2025 09:40

suvarchal approved these changes Jun 27, 2025

View reviewed changes

suvarchal merged commit c3fdac7 into main Jun 30, 2025
7 checks passed

basava70 mentioned this pull request Jun 30, 2025

Phase 1 OpenACC #732

Merged

5 tasks

JanStreffing deleted the openacc_test_main branch June 30, 2025 08:08

basava70 mentioned this pull request Jul 1, 2025

Openacc phase 2 #734

Open

14 tasks

Openacc test main #693

Openacc test main #693

Uh oh!

Conversation

basava70 commented Feb 25, 2025

Uh oh!

JanStreffing commented Feb 25, 2025

Uh oh!

basava70 commented Feb 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

basava70 commented Feb 25, 2025

Uh oh!

JanStreffing commented Feb 25, 2025

Uh oh!

basava70 commented Feb 26, 2025

Uh oh!

JanStreffing commented Feb 26, 2025

Uh oh!

basava70 commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JanStreffing commented Feb 26, 2025

Uh oh!

basava70 commented Feb 26, 2025

Uh oh!

pgierz commented Feb 26, 2025

Uh oh!

basava70 commented Jun 19, 2025

Uh oh!

JanStreffing left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

suvarchal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

suvarchal Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

suvarchal Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

suvarchal Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

suvarchal commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mandresm commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

basava70 commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JanStreffing commented Jun 27, 2025

Uh oh!

suvarchal commented Jun 27, 2025

Uh oh!

basava70 commented Jun 27, 2025

Uh oh!

basava70 commented Jun 27, 2025

Uh oh!

JanStreffing commented Jun 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

basava70 commented Feb 26, 2025 •

edited

Loading

suvarchal commented Jun 19, 2025 •

edited

Loading

mandresm commented Jun 19, 2025 •

edited

Loading

basava70 commented Jun 27, 2025 •

edited

Loading