Missing python dependencies when using rna.h5ad as input

[ioansarr]

Thanks for developing this cool method!

I've encountered the following error at the "generate_atac_matrix.smk" step, which I think suggests that anndata is missing from the environment created by the snakemake workflow:

Error in (function (cond)  :
  error in evaluating the argument 'x' in selecting a method for function 't': ModuleNotFoundError: No module named 'anndata'
Run `reticulate::py_last_error()` for details.
Calls: t -> read_h5ad -> <Anonymous> -> py_module_import
Execution halted

I manually added anndata as a dependency to workflow/envs/sc_e2g.yml and this seems to resolve the issue

- pip
  - pip:
      - anndata==0.10.7

Presumably the anndata version will need to roughly match the one used to generate the rna.h5ad object.

I am now facing a different issue downstream, but will post this separately.

[613weilin]

Hi @ioansarr,

Thank you for reporting this issue and for providing a solution! I am unable to replicate this issue on my end, so I will not be updating the workflow/envs/sc_e2g.yml for now. However, your suggestion might be helpful to other users, and I appreciate your contribution. Please let us know if you encounter any other issues.

[ioansarr]

Hi Wei,

I wonder whether this is because you already have some of these modules installed centrally rather than in the conda environment.

In any case, still useful to have this here in case anyone else faces the same issue.

[613weilin]

I have checked the sc_e2g, mamba, and base environments using conda list, but I did not find the anndata package installed in any of them. It's indeed helpful to keep this discussion open for others who might face a similar issue.

[hmummey]

Hi all, thanks for the great package and useful discussion on this thread. I've been having the same problem with the generate_atac_matrix step of the snakemake pipeline. A few months ago I was able to fix it by manually installing the required package in the conda env snakemake uses for this step (conda install conda-forge::anndata), however I updated the repo and that hasn't been working anymore...

I have also tried the suggestion from @ioansarr and nothing changes for me, but perhaps I'm implementing it wrong. Do you mind sharing what your full yml file looks like?

Overall I think this is evidence that it might be worthwhile adding this important package into the relevant yaml files after all!

[ioansarr]

Hi @hmummey,

My previous solution also only worked with the initial release but no longer works in the updated version. It seems to be an issue with reticulate recognizing the correct conda environment.

At the moment, I am able to work around this by exporting the current conda environment as a snakemake parameter and then reading this info in the script:

In generate_atac_matrix.smk:

rule generate_atac_matrix:
	input:
		kendall_pairs_path = 
			os.path.join(
				RESULTS_DIR, 
				"{cluster}", 
				"Kendall", 
				"Pairs.tsv.gz"
			),
		atac_frag_path = 
			lambda wildcards: CELL_CLUSTER_DF.loc[wildcards.cluster, "atac_frag_file"],
		rna_matrix_path = 
			lambda wildcards: CELL_CLUSTER_DF.loc[wildcards.cluster, "rna_matrix_file"],
		cell_barcodes_path = get_cell_barcode_file(config["RNA_matrix_filtered"])
	output:
		atac_matrix_path = 
			os.path.join(
				RESULTS_DIR, 
				"{cluster}", 
				"Kendall", 
				"atac_matrix.rds"
			)
	resources:
		mem_mb=encode_e2g.ABC.determine_mem_mb
	conda:
		"../envs/sc_e2g.yml"
	params:
		python_bin = os.path.join(os.environ["CONDA_PREFIX"], "bin", "python")
	script:
		"../scripts/feature_computation/generate_atac_matrix.R"

Then in generate_atac_matrix.R:

suppressPackageStartupMessages({
  library(reticulate)
  use_python(snakemake@params$python_bin, required = TRUE)
  library(genomation)
  library(GenomicRanges)
  library(Signac)
  library(anndata)
  library(tools)
  library(Seurat)
})

After resolving this, I encountered the same problem with compute_kendall.smk/compute_kendall.R, where surprisingly the same fix doesn't work.

There, I had to directly hardcode the python binary inside the compute_kendall.R script (which is obviously much less flexible and needs to be adjusted with each installation):

suppressPackageStartupMessages({
  library(reticulate)
  use_python('path2conda/bin/python', required = TRUE)
  library(GenomicRanges)
  library(genomation)
  library(foreach)
  library(Signac)
  library(Seurat)
  library(Rcpp)
  library(data.table)
  library(Matrix)
  library(anndata)
  library(tools)
  library(dplyr)
  library(tibble)
})

This seems to allow the workflow to complete successfully.

I hope this helps.