Resolve google cloud login

[rcannood]

See how we resolve google cloud authentication requirements in https://github.com/openproblems-bio/task_preprocessing_imagingbased_st/blob/aa0a15ab226c1deabe557b5a90b5fab540640f26/src/datasets/loaders/vizgen_merscope/config.vsh.yaml

[LouisK92]

I was looking quite a bit into this today since we were working on preparing the scRNAseq references for the vizgen datasets and I wanted to get the vizgen downloads running within docker, and especially on the cloud.

I think the way to go is with google cloud control service accounts and env vars. So, the procedure for running the vizgen download workflow would be

• create a google cloud service account (or use your existing one) (short description e.g. here) (is needed for generating a json, not possible with standard authentication afaik)

• get the service account key json

• convert the json into a str cat gcloud-key.json | base64

  • locally: set this as a global nextflow env variable GCLOUD_KEY_JSON
  • seqera: set a seqera env variable GCLOUD_KEY_JSON

• Then the login should work with these added lines in the vizgen download script:

    echo \$GCLOUD_KEY_JSON | base64 --decode > /gcloud-key.json
    export GOOGLE_APPLICATION_CREDENTIALS=/gcloud-key.json
    gcloud auth activate-service-account --key-file=/gcloud-key.json

I haven't tested that yet. And I'm not sure if it's straightforward to set the env variables such that they are also set in the docker container.

[rcannood]

Right. I'm wondering whether we can use rclone for this?

You just need to:

docker run --rm -it ubuntu 24.04

apt-get update && apt-get upgrade -y
apt-get install rclone ca-certificates

# add a google cloud storage to the rclone config
# unfortunately you need to do this before being able to run any of the next commands
rclone config create gs gcs anonymous=true

From that point on you can:

rclone ls "gs:$BUCKET_NAME/$DATASET/$FILE"
rclone sync "gs:$BUCKET_NAME/$DATASET/$FILE" "$TARGET_DIR/" -v

[rcannood]

Update: trying it out in the improvements branch

[rcannood]

Darn. I was hoping that it was possible to ls as an anonymous user...

2025/03/20 09:42:05 Failed to ls: googleapi: Error 401: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist)., required

[LouisK92]

just as a note in that context, the vizgen download needed small adjustments (I think due updates in common (?)), see here c2281fe

[LouisK92]

Was resolved with 1e45fb0 (solution: "let Nextflow download the directory from google cloud" | see message in discord ist-preprocessing from Robrecht 22.03.25, 07:46) (merged in PR #20)

[LouisK92]

@rcannood

It seems that for me the proposed solutions don't work, or maybe I am testing them in a wrong way.

I adjusted the test script to exchange the previous gcloud arguments with the new input argument: 741bc99 (ignore the dataset_id changes, this was because of an incorrectly resolved merge from my side).

Running the test script:

bash src/datasets/workflows/process_vizgen_merscope/test.sh

End up in the previous error:

Launching `target/nextflow/datasets/workflows/process_vizgen_merscope/main.nf` [sleepy_baekeland] DSL2 - revision: 5453c93eb0

[-        ] process_vizgen_merscope:run_wf:vizgen_merscope:processWf:vizgen_merscope_process -
[-        ] process_vizgen_merscope:run_wf:crop_region:processWf:crop_region_process         -
[-        ] process_vizgen_merscope:run_wf:vizgen_merscope:processWf:vizgen_merscope_process -
[-        ] process_vizgen_merscope:run_wf:crop_region:processWf:crop_region_process         -
[-        ] process_vizgen_merscope:publishFilesSimpleWf:publishFilesProc                    -
[-        ] process_vizgen_merscope:publishStatesSimpleWf:publishStatesProc                  -
ERROR ~ Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).

 -- Check script 'target/nextflow/datasets/workflows/process_vizgen_merscope/../../../../nextflow/datasets/loaders/vizgen_merscope/main.nf' at line: 3863 or see '.nextflow.log' file for more details

So, with a local nextflow run I can't get Nextflow to download the data automatically from gcloud.

Also, I am unsure how the code change

task_ist_preprocessing/src/datasets/loaders/vizgen_merscope/script.py

Line 78 in cff6309

RAW_DATA_DIR = Path(par["input_dir"])

works. We don't provide a par['input_dir'] argument. And the par['input'] argument is not a directory, but a gcloud link.

I'm not sure if the gcloud authentication can actually be solved within the container.
If not, maybe we can just keep the data loader as a script that downloads the files locally and then we upload them to aws?
For this I think we'd need to revert changes in ac0c0c0 (e.g. deletion of the download script)

[rcannood]

I think you might need to log into google cloud via gcloud auth application-default login so that Nextflow can fetch the files from GCloud before then passing the data to the component

-- more info: https://www.nextflow.io/docs/latest/google.html#credentials

Does this resolve the issue?