ReflectionTable fails to save as H5 when a boolean column exists

[yash4karan]

I am having issues writing ReflectionTables as H5 files, but only when a boolean column exists in the ReflectionTable. For any other supported column data type, this error does not show up.

Here is a minimal code snippet that recreates this issue:

#include <dx2/reflection.hpp>
#include <vector>

int main() {
    ReflectionTable refl;
    refl.add_column("id", std::vector{0, 1, 2});
    std::vector<bool> bool_vec(3, true);
    refl.add_column("bool_column", bool_vec);
    refl.write("bool_write.refl");
    return 0;
}

The following is the code added to the corresponding CMakeFile:

add_executable(refl_bool_saving_issue
    refl_bool_saving_issue.cc
)
target_link_libraries(refl_bool_saving_issue
    PRIVATE
    dx2
)

Upon building the executable and running the code, I get the following error:

HDF5-DIAG: Error detected in HDF5 (1.14.6):
  #000: /tmp/hdf5-20250207-38588-gjrv3m/hdf5-1.14.6/src/H5T.c line 2063 in H5Tclose(): not a datatype
    major: Invalid arguments to routine
    minor: Inappropriate type

[jbeilstenedmands]

Hi Yash,
We encountered this issue before in ffs (the issue arises due to the specialisation of std::vector for bools), and our solution was to define a BoolEnum into the reflection table:

dx2/include/dx2/reflection.hpp

Line 370 in 82f0dc0

using BoolEnum = h5dispatch::BoolEnum;

You can see an example usage here of setting a bool column and then retrieving:

Setting a bool column:
https://github.com/DiamondLightSource/fast-feedback-service/blob/1a5ae57e613f54be25f6eb64c9f4b60ca305c9c7/baseline/indexer/indexer.cc#L234
https://github.com/DiamondLightSource/fast-feedback-service/blob/1a5ae57e613f54be25f6eb64c9f4b60ca305c9c7/baseline/indexer/indexer.cc#L263

Retrieving a bool column:
https://github.com/DiamondLightSource/fast-feedback-service/blob/1a5ae57e613f54be25f6eb64c9f4b60ca305c9c7/baseline/predictor/scan_static_predictor.cc#L54

Would this enable you to do what you need?

[jbeilstenedmands]

Sorry I was impatient, looks like you are following the same pattern so would have expected your code to work, having a further think....

[jbeilstenedmands]

Ok, I ran your code snippet and the error is printed, however the file is still written to disk and the expected data columns are there, including the bool_column.

[yash4karan]

Ah, I see. Yes, it outputs a valid file for me, too. Do you think the best way forward is to suppress this error?

[graeme-winter]

Does it output H5T_NATIVE_HBOOL?

https://support.hdfgroup.org/documentation/hdf5/latest/_h5_t__u_g.html

Checking, as it seems sane if it did.

[yash4karan]

It seems to output the following enum:

DATATYPE  H5T_ENUM {
  H5T_STD_U8LE;
  "FALSE"      0;
  "TRUE"       1;
}

[dimitrivlachos]

It does not use H5T_NATIVE_HBOOL I believe because we ran into issues with bit packing. I created that enum to get around it and handle vector<bool>'s in their own specialisation.

The error is probably coming from one of the h5 functions checking if a file/group/dataset exists, and it is not being silenced properly. Either that or something is trying to destroy the enum a second time, it is something that I need to look into, but for all intents and purposes it should still write out the data.