Zarr support #190

keller-mark · 2024-11-05T16:09:31Z

Fixes #91

These changes are from both me and @Artur-man

The main public-facing changes here are:

The ZarrAnnData class
read_zarr and write_zarr top-level functions
Support for from_Seurat(output_class="ZarrAnnData")
Support for from_SingleCellExperiment(output_class="ZarrAnnData")

Internally:

read_zarr_helpers.R is the zarr analog of read_h5ad_helpers.R
write_zarr_helpers.R is the zarr analog of write_h5ad_helpers.R
Test fixtures within inst/extdata/example.zarr (this makes the diff noisy, apologies)
Lots of tests:
- test-Zarr-read.R (35 new tests)
- test-Zarr-write.R (70)
- test-ZarrAnnData.R (26)
- test-h5ad-zarr.R (17)

A number of these functions generate warnings in the R console that are intended to be followed up on to improve the code (and should probably be resolved as end users may not appreciate them), but the tests still pass despite these warnings.

Known things that are not implemented here:

support for recarrays
usage of mode = c("r", "r+", "a", "w", "w-", "x") parameter value

… internally.

…dataR into keller-mark/zarr

…erse#171)

…milar to HDF5AnnData

Simplify how obs and var names handled in ZarrAnnData

More zarr-related changes

Update comments

rcannood

Fantastic work @keller-mark and @Artur-man !

I went through the PR for a first time and left some minor comments. I will review the code by running it a couple of times next :)

R/read_zarr.R

rcannood · 2024-11-05T20:12:17Z

R/read_zarr_helpers.R

+  attrs <- g$get_attrs()$to_list()
+
+  if (!all(c("encoding-type", "encoding-version") %in% names(attrs))) {
+    path <- name


where are a lot of linting issues in this file -- could you run lintr::lint_package() and fix any issues that pop up?

done ... did a full lint_package check and corrected some R check issues too!

rcannood · 2024-11-05T20:15:30Z

inst/extdata/example2.zarr/.zgroup

This file should probably be removed

Bisaloo

FYI, the patterns I highlight here can also be detected via lintr if you use all_linters() and then selectively turn off the linters you don't want.

For example: https://github.com/Huber-group-EMBL/Rarr/blob/devel/.lintr

Bisaloo · 2025-11-25T16:12:07Z

R/Rarr_utils.R

+#' @param version zarr version
+#' @export
+create_zarr_group <- function(store, name, version = "v2"){
+  split.name <- strsplit(name, split = "\\/")[[1]]


It'll be slightly faster if regexes are not used:

Suggested change

split.name <- strsplit(name, split = "\\/")[[1]]

split.name <- strsplit(name, split = "/", fixed = TRUE)[[1]]

Bisaloo · 2025-11-25T16:13:00Z

R/Rarr_utils.R

+create_zarr_group <- function(store, name, version = "v2"){
+  split.name <- strsplit(name, split = "\\/")[[1]]
+  if(length(split.name) > 1){
+    split.name <- vapply(seq_len(length(split.name)),


Suggested change

split.name <- vapply(seq_len(length(split.name)),

split.name <- vapply(seq_along(split.name),

Bisaloo · 2025-12-04T15:43:34Z

R/write_zarr_helpers.R

+  if (!is.null(dim(value))) {
+    dims <- dim(value)
+  } else {
+    dims <- length(value)
+  }


If you're using a recent version of R:

Suggested change

if (!is.null(dim(value))) {

dims <- dim(value)

} else {

dims <- length(value)

}

dims <- dim(value) %||% length(value)

(you can also redefine the operator in this pkg for compatibility with older R versions)

oh I will change this directly, anndataR requires 4.5 and later anyways. Thanks Hugo.

lazappi · 2025-12-11T10:48:11Z

Some functions such as read_h5ad_data_frame and read_h5ad_collection would repeat themselves if we were to define a, e.g. read_zarr_data_frame/_collection since they do not directly call rhdf5 or Rarr methods.

I think they are slightly different because one will call read_h5ad_element() and the other will call read_zarr_element(). I'm open to reducing redundancy where it makes sense and is still easy to follow but maybe it's better to leave for a separate PR so it's easier to review the Zarr functionality without other changes.

Artur-man · 2025-12-11T11:01:33Z

Then @lazappi perhaps we can start your review, let me know if you guys want me to change anything (there are lots of changes!). Meanwhile, @Bisaloo will keep working on fixing a few small issues in Rarr.

lazappi

We will probably need some more rounds of reviews but hopefully I've caught a lot of stuff already.

Additional comments:

I commented on some style things the first time I saw them but please do the same wherever else it is relevant
The documentation in AnnData-usage needs to be updated to add ZarrAnnData
Round trip tests for Zarr need to be added
We should probably show Zarr usage somewhere in a main vignette
Please try to get the CI checks to pass

lazappi · 2025-12-12T08:52:46Z

inst/scripts/example_files.py

+# os.chdir("inst/extdata/")
+# zip = zipfile.ZipFile("example.zarr.zip", "w", zipfile.ZIP_DEFLATED)
+# zip.write("example.zarr")
+# shutil.rmtree("example.zarr")
+# zip.close()


Maybe this should be uncommented?

lazappi · 2025-12-12T08:53:36Z

inst/scripts/requirements.yml

If we are adding this we can probably remove the versions from the script.

I learned about this recently, which can be a good lightweight way to document versions without having to create a dedicated environment file: https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies

Note that it's not uv-specific but it's also documented in the official python docs.

lazappi · 2025-12-12T08:54:16Z

R/AbstractAnnData.R

      )
    },
    #' @description
+    #' Convert to an [`ZarrAnnData`]


Suggested change

#' Convert to an [`ZarrAnnData`]

#' Convert to a [`ZarrAnnData`]

lazappi · 2025-12-12T08:54:43Z

R/AbstractAnnData.R

+    #' @param compression See [as_ZarrAnnData()]
+    #' @param mode See [as_ZarrAnnData()]
+    #'
+    #' @return An [`ZarrAnnData`] object


Suggested change

#' @return An [`ZarrAnnData`] object

#' @return A [`ZarrAnnData`] object

lazappi · 2025-12-12T08:55:57Z

R/AbstractAnnData.R

      write_h5ad(object = self, path, compression = compression, mode = mode)
+    },
+    #' @description
+    #' Write the `AnnData` object to an H5AD file


Suggested change

#' Write the `AnnData` object to an H5AD file

#' Write the `AnnData` object to a Zarr file

lazappi · 2025-12-12T10:27:03Z

tests/testthat/test-h5ad-zarr.R

+})
+
+test_that("reading recarrays works", {
+  skip("read_zarr_rec_array is not implemented yet")


I think this is implemented now?

Hmm, ok the skip here is misleading, since Rarr currently doesn't read arbitrary structured data types, but my current implementation under read_zarr_rec_array only works for the example data. So it should stay like this until the following is fixed:
Huber-group-EMBL/Rarr#81

lazappi · 2025-12-12T10:28:56Z

tests/testthat/test-h5ad-zarr.R

+  expect_equal(mapping_h5ad, mapping_zarr)
+})
+
+test_that("reading dataframes works", {


Suggested change

test_that("reading dataframes works", {

test_that("reading dataframes is the same for h5ad and zarr", {

lazappi · 2025-12-12T10:29:52Z

tests/testthat/test-h5ad-zarr.R

+})
+
+test_that("reading H5AD as SingleCellExperiment is same for h5ad and zarr", {
+  skip("for now, example.zarr and example.h5ad are not identical!")


Why are they not identical?

some readers are not implemented yet, but will check again.
#190 (comment)

lazappi · 2025-12-12T10:30:17Z

tests/testthat/test-h5ad-zarr.R

+  expect_equal(df_h5ad, df_zarr)
+})
+
+test_that("reading H5AD as SingleCellExperiment is same for h5ad and zarr", {


Should probably add a test for Seurat

fixed, but skipping for now until identical.

lazappi · 2025-12-12T10:40:00Z

vignettes/articles/software_design.Rmd

 - The `ReticulateAnnData` access data stored in an `AnnData` object in a concurrent Python session.
  This comes with the overhead and complexity of using `r CRANpkg("reticulate")` but is sometimes useful to access functionality that has not yet been implemented in `r Biocpkg("anndataR")`.
- The planned `ZarrAnnData` will provide an interface to an `AnnData` Zarr store, similar to `HDF5AnnData`.
+- The `ZarrAnnData` provides an interface to a Zarr data (i.e. Zarr store) and, similar to a H5AD file, minimal data is stored in memory until it is requested by the user. It is as an intermediate object when reading/writing Zarr store but can be useful for accessing parts of large files.


Suggested change

- The `ZarrAnnData` provides an interface to a Zarr data (i.e. Zarr store) and, similar to a H5AD file, minimal data is stored in memory until it is requested by the user. It is as an intermediate object when reading/writing Zarr store but can be useful for accessing parts of large files.

- The `ZarrAnnData` provides an interface to a Zarr store and, similar to a H5AD file, minimal data is stored in memory until it is requested by the user. It is as an intermediate object when reading/writing Zarr stores but can be useful for accessing parts of large files.

Artur-man · 2025-12-12T11:14:36Z

We will probably need some more rounds of reviews but hopefully I've caught a lot of stuff already.

Additional comments:

I commented on some style things the first time I saw them but please do the same wherever else it is relevant

The documentation in AnnData-usage needs to be updated to add ZarrAnnData

Round trip tests for Zarr need to be added

We should probably show Zarr usage somewhere in a main vignette

Please try to get the CI checks to pass

awesome @lazappi I will get to it.

Bisaloo · 2025-12-12T12:41:52Z

inst/scripts/requirements.yml

I learned about this recently, which can be a good lightweight way to document versions without having to create a dedicated environment file: https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies

Note that it's not uv-specific but it's also documented in the official python docs.

Bisaloo · 2025-12-12T12:47:54Z

R/read_zarr_helpers.R

+  if (is.null(dim(data))) {
+    data <- as.vector(data)
+    dim(data) <- length(data)
+  }


Rarr::read_zarr_array() currently always returns arrays (i.e., dim() is not NULL). I don't expect this to change as I like the idea of a consistent output class.

Bisaloo · 2025-12-12T12:51:29Z

README.md

 - To convert to/from `Seurat` objects, install [SeuratObject](https://cran.r-project.org/package=SeuratObject):  
  `install.packages("SeuratObject")`
+- To read/write \*.zarr files, you need to install
+  [zarr](https://www.bioconductor.org/packages/release/bioc/html/Rarr.html):  


Suggested change

[zarr](https://www.bioconductor.org/packages/release/bioc/html/Rarr.html):

[Rarr](https://www.bioconductor.org/packages/release/bioc/html/Rarr.html):

Artur-man · 2025-12-21T16:08:39Z

Errors I am getting at lint and pkgdown GHAs look unrelated to the package:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 
    unable to load shared object '/home/runner/work/_temp/Library/Rhdf5lib/libs/Rhdf5lib.so':
    libsz.so.2: cannot open shared object file: No such file or directory

Artur-man · 2025-12-21T17:41:37Z

@Bisaloo also the tests that are passing for macos, ubuntu and windows are failing for macos (intel) and ubuntu-bioc-release. I understand the error in release (for obvious reasons), but macos-15 I don't know. Any ideas ?

Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
   5.         └─base::array(metadata$fill_value, dim = lengths(index))
  ── Error ('test-Zarr-read.R:87:3'): reading numeric scalars works ──────────────
  Error in `array(metadata$fill_value, dim = lengths(index))`: 'dim' cannot be of length 0
  Backtrace:
      ▆
   1. └─anndataR:::read_zarr_numeric_scalar(store, "uns/IntScalar") at test-Zarr-read.R:87:3
   2.   └─anndataR:::read_zarr_array(store, name)
   3.     └─Rarr::read_zarr_array(file.path(store, name))
   4.       └─Rarr:::read_data(...)
   5.         └─base::array(metadata$fill_value, dim = lengths(index))
  
  [ FAIL 4 | WARN 447 | SKIP 12 | PASS 1842 ]

https://github.com/scverse/anndataR/actions/runs/20411185095/job/58648259569?pr=190

Bisaloo · 2025-12-21T17:47:43Z

Yes, it looks that on this runner, Rarr 1.11.9 is installed, when support for reading scalars was added in Rarr 1.11.12. I think it's related to BioC devel not having more recent binaries for macOS x86 yet.

Maybe try setting:

Imports:
    Rarr (>= 1.11.12)

In DESCRIPTION to force installation from source?

This will not work for the runner using BioC release but it's already failing anyways.

Artur-man · 2025-12-21T18:05:58Z

I see. Then I will assume all checks are passing for now. thanks man

Artur-man · 2025-12-21T19:48:43Z

keller-mark and others added 28 commits May 6, 2024 16:25

WIP

19698c3

More tests passing

5100a40

Fix df read bug

4c9d01f

More tests passing after fixing zero-dimensional get bug in pizzarr

eb8b69c

WIP: writing

58f5ce4

Fix more tests

a15513a

Zarr df writing

1ef1be6

WIP: ZarrAnnData class

77a7ee2

Tests passing

27ce6c1

Tests that compare h5ad to zarr

8790715

Use Rarr to read full numeric arrays

bb0c6c7

Fix bugs. Add test for from_SingleCellExperiment with Zarr

0456ecd

Add a to_dense param to ZarrAnnData constructor. Add overwrite params…

acac772

… internally.

Update

30316ed

Backwards dense/sparse

c6b4d89

Merge branch 'keller-mark/zarr' of https://github.com/keller-mark/ann…

9cadc26

…dataR into keller-mark/zarr

Merge branch 'keller-mark/zarr' into zarr

a40a618

Simplify how obs and var names handled in ZarrAnnData (similar to scv…

1afd6eb

…erse#171)

update extdata and documentation

7f53049

fix set/get zarr _index, update text example.zarr and update tests si…

3800f38

…milar to HDF5AnnData

Merge pull request #5 from Artur-man/zarr

4a7d4c2

Simplify how obs and var names handled in ZarrAnnData

Merge

c881bb9

Fix test

4a1bbde

Revert unnecessary changes

15dfbde

Formatting

438809a

Merge pull request #6 from keller-mark/keller-mark/zarr-2

2215402

More zarr-related changes

Add comments

087ffb7

Merge pull request #7 from keller-mark/keller-mark/comments

37d1ae5

Update comments

rcannood requested changes Nov 5, 2024

View reviewed changes

remove unnecessary example zarr store

357a8d7

remove overwrite

2499d5c

Artur-man mentioned this pull request Dec 8, 2025

Reading structured data types Huber-group-EMBL/Rarr#81

Open

Bisaloo reviewed Dec 9, 2025

View reviewed changes

R code styling

bd6238a

lazappi requested changes Dec 12, 2025

View reviewed changes

Bisaloo reviewed Dec 12, 2025

View reviewed changes

Artur-man added 5 commits December 12, 2025 22:53

fixes from @lazappi

aaf9801

Merge branch 'main' into keller-mark/zarr

43d4f1f

air format

73ee0e3

update some documentation

d70a011

fix some tests

ccb0cdf

Bisaloo mentioned this pull request Dec 21, 2025

Bump GHA cache version to force reinstall #403

Open

6 tasks

Artur-man added 10 commits December 21, 2025 22:25

more fixes on anndata-zarr integration

fe8f196

update ZarrAnnData$initialize

e6efbf2

update zarr compression

8b0d1b0

fix column-order here, C based ordering for arrays

bafae8e

implement roundtrip tests for anndata-zarr

eead040

add zarr to vignettes

64e4289

update README and software_design.rmd

23f8ac5

update AnnData-usage

0852908

update write_zarr documentation

8677fad

update write_zarr_null

efa2ca0

	split.name <- strsplit(name, split = "\\/")[[1]]
	split.name <- strsplit(name, split = "/", fixed = TRUE)[[1]]

	split.name <- vapply(seq_len(length(split.name)),
	split.name <- vapply(seq_along(split.name),

	#' Convert to an [`ZarrAnnData`]
	#' Convert to a [`ZarrAnnData`]

	#' @return An [`ZarrAnnData`] object
	#' @return A [`ZarrAnnData`] object

	#' Write the `AnnData` object to an H5AD file
	#' Write the `AnnData` object to a Zarr file

	test_that("reading dataframes works", {
	test_that("reading dataframes is the same for h5ad and zarr", {

	- The `ZarrAnnData` provides an interface to a Zarr data (i.e. Zarr store) and, similar to a H5AD file, minimal data is stored in memory until it is requested by the user. It is as an intermediate object when reading/writing Zarr store but can be useful for accessing parts of large files.
	- The `ZarrAnnData` provides an interface to a Zarr store and, similar to a H5AD file, minimal data is stored in memory until it is requested by the user. It is as an intermediate object when reading/writing Zarr stores but can be useful for accessing parts of large files.

	[zarr](https://www.bioconductor.org/packages/release/bioc/html/Rarr.html):
	[Rarr](https://www.bioconductor.org/packages/release/bioc/html/Rarr.html):

Uh oh!

Zarr support #190

Are you sure you want to change the base?

Zarr support #190

Uh oh!

Conversation

keller-mark commented Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcannood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bisaloo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lazappi commented Dec 11, 2025

Uh oh!

Artur-man commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lazappi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Artur-man Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

keller-mark commented Nov 5, 2024 •

edited

Loading

Artur-man commented Dec 11, 2025 •

edited

Loading

lazappi left a comment •

edited

Loading

Artur-man Dec 21, 2025 •

edited

Loading

Artur-man commented Dec 21, 2025 •

edited

Loading

Bisaloo commented Dec 21, 2025 •

edited

Loading

Artur-man commented Dec 21, 2025 •

edited

Loading