`DynamicTableRegion` written with `hdmf_experimental` instead of `hdmf_common` when using NWB extensions

[weiglszonja]

Hi all,

I ran into an issue when reading NWB files with MatNWB that was generated with extensions using DynamicTableRegion. MatNWB throws the following error:

Error using feval
Unrecognized function or variable 'types.hdmf_experimental.DynamicTableRegion'.

Error in io.createParsedType (line 28)
    typeInstance = feval(typeName, varargin{:});
Error in io.parseDataset (line 86)
    parsed = io.createParsedType(fullpath, Type.typename, kwargs{:});
Error in io.parseGroup (line 22)
    dataset = io.parseDataset(filename, datasetInfo, fullPath, Blacklist);
Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);
Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);
Error in nwbRead (line 99)
    nwb = io.parseGroup(filename, h5info(filename), blackList);
Error in <script> (line 17)
    nwbfile = nwbRead(nwbfile_path);

After debugging with @ehennestad, we found that the problem is that the namespace for DynamicTableRegion inside the file is set to hdmf_experimental instead of hdmf_common in /acquisition/electrical_series/electrodes.

This is a snippet from an NWB file generated with ndx-fscv:

This is another NWB file generated with ndx-anatomical-location:

I also managed to replicate this with ndx-fiber-photometry. In all cases, the written file incorrectly assigns the namespace hdmf_experimental to DynamicTableRegion.

PyNWB reads the file without any problems, however, MatNWB fails to resolve types.hdmf_experimental.DynamicTableRegion, which triggers the error.

Eivind is attempting to reproduce this on his side as well.

I'm not sure if this needs to be fixed in MatNWB or hdmf side so I'm opening this here for now and see what you think.

[bendichter]

@rly here's Opus 4.5's take:

Root Cause Analysis for Issue #1347

I've identified the root cause of the bug where DynamicTableRegion is incorrectly written with namespace hdmf_experimental instead of hdmf_common when using NWB extensions.

Problem Location

The bug is in TypeMap.merge() (lines 455-467) combined with how namespaces are added to __container_types during load_namespaces().

Root Cause

1. Namespace Registration Order: When load_namespaces() is called (lines 483-492), it processes namespace dependencies:

   for new_ns, ns_deps in deps.items():
       for src_ns, types in ns_deps.items():
           for dt in types:
               container_cls = self.get_dt_container_cls(dt, src_ns, autogen=False)
               if container_cls is None:
                   container_cls = TypeSource(src_ns, dt)
               self.register_container_type(new_ns, dt, container_cls)

2. hdmf-common has no dependencies, so nothing is registered for it in this loop.

3. hdmf-experimental depends on hdmf-common, so types like DynamicTableRegion are registered as TypeSource for hdmf-experimental. This makes hdmf-experimental the FIRST namespace added to __container_types (which is an OrderedDict).

4. Later, when get_class('DynamicTableRegion', CORE_NAMESPACE) is called, the class is generated and registered under hdmf-common. But now hdmf-common is SECOND in __container_types.

5. When merge() is called during deepcopy() (used when loading extensions):

   for namespace in type_map.__container_types:
       for data_type in type_map.__container_types[namespace]:
           container_cls = type_map.__container_types[namespace][data_type]
           self.register_container_type(namespace, data_type, container_cls)

   It iterates in insertion order:

   • First: hdmf-experimental → __data_types.setdefault(cls, ('hdmf-experimental', 'DynamicTableRegion')) SETS the value
   • Second: hdmf-common → setdefault doesn't overwrite because key already exists

6. During build, get_container_ns_dt() returns ('hdmf-experimental', 'DynamicTableRegion') from __data_types, so the wrong namespace is written to the file.

Proposed Fix

The merge() method should copy __data_types directly rather than re-registering types, which causes the wrong namespace to be stored when iterating in insertion order:

def merge(self, type_map, ns_catalog=False):
    if ns_catalog:
        self.namespace_catalog.merge(type_map.namespace_catalog)
    for namespace in type_map.__container_types:
        for data_type in type_map.__container_types[namespace]:
            container_cls = type_map.__container_types[namespace][data_type]
            self.register_container_type(namespace, data_type, container_cls)
    
    # Copy __data_types directly to preserve original namespace associations
    for container_cls, ns_dt in type_map._TypeMap__data_types.items():
        if container_cls not in self.__data_types:
            self.__data_types[container_cls] = ns_dt
    
    for container_cls in type_map.__mapper_cls:
        self.register_map(container_cls, type_map.__mapper_cls[container_cls])
    for custom_generators in reversed(type_map.__class_generator_manager.custom_generators):
        self.register_generator(custom_generators)

Alternatively, modify register_container_type() at line 752 to check if the class is already registered with a different namespace and prefer the "source" namespace (checking TypeSource relationships).

Impact

This bug affects any NWB file written with extensions that depend on hdmf-experimental. The DynamicTableRegion (and potentially other types from hdmf-common) will have incorrect namespace attributes, causing:

• MatNWB read failures (as reported)
• Potential validation issues
• Cross-language compatibility problems

[ehennestad]

Here is a reproducible example to create a simple file with this issue. If the import statement from ndx_fscv ... is commented out, the namespace of DynamicTableRegion is hdmf_common as should be expected, otherwise it is hdmf_experimental

from datetime import datetime
import numpy as np
from dateutil.tz import tzutc
from pynwb import NWBFile, NWBHDF5IO
from pynwb.ecephys import ElectricalSeries

# This import causes namespace of DynamicTableRegion to be hdmf_experimental instead of hdmf_common
from ndx_fscv import FSCVExcitationSeries, FSCVResponseSeries

# Create NWBFile
nwbfile = NWBFile(
    session_description="FSCV experiment",
    identifier="FSCV123",
    session_start_time=datetime(2024, 1, 1, tzinfo=tzutc()),
)

# Create device and electrode group
device = nwbfile.create_device(
    name="fscv_device",
    description="A 4-channel FSCV device.",
)
electrode_group = nwbfile.create_electrode_group(
    name="fscv_electrode_group",
    description="The electrode group for the FSCV electrodes.",
    device=device,
    location="brain region",
)

# Add the electrodes to the NWBFile
for _ in range(4):
    nwbfile.add_electrode(group=electrode_group, location="brain region")

# Create an electrode table region referencing the FSCV electrodes
electrodes = nwbfile.create_electrode_table_region(region=[0, 1, 2, 3], description="FSCV electrodes")

raw_data = np.random.randn(50, 12)
response_series = ElectricalSeries(
    name="ElectricalSeries",
    description="Raw acquisition traces",
    data=raw_data,
    electrodes=electrodes,
    starting_time=0.0,  # timestamp of the first sample in seconds relative to the session start time
    rate=20000.0,  # in Hz
)

# # Add to NWBFile
nwbfile.add_acquisition(response_series)

# Write to file
with NWBHDF5IO("example_fscv.nwb", "w") as io:
    io.write(nwbfile)

[rly]

Thanks for the detailed bug report and analysis @weiglszonja @ehennestad and @bendichter (and claude)!

What puzzles me is why we haven't seen this before. At least some NWB files with extensions have been written before without this issue. The issue does not arise when importing ndx-franklab-novela or ndx-optogenetics. And it happens in @ehennestad 's example with hdmf==4.0.0 as well before we changed some of the logic in NamespaceCatalog.load_namespaces. The fix looks reasonable though it seems kind of hacky to fix there. Would it be all right for me to dig a little deeper and resolve this when I have more time on Tuesday?

[rly]

Update: The issue stems from ndx-fscv calling get_class on a spec that has a field of type DynamicTableRegion (the "electrodes" attribute). When the class generator tries to find the class based on the name "DynamicTableRegion" alone, it encounters the data type in the "hdmf-experimental" namespace first, sees that it points to a TypeSource instead of a container class, finds the correct container class associated with the "hdmf-common" namespace, and associates the "DynamicTableRegion" data type in the "hdmf-experimental" namespace with this class. During build/write, the type map is deep-copied, which calls merge, which processes the hdmf-experimental namespace first, sees the class, and associates it with the "hdmf-experimental" namespace since that was processed first. This is the namespace that is written. Claude's root cause analysis is partially correct.

The issue appears to be specific to extensions that call get_class on a data type that includes a field with data type in the hdmf-common namespace, which is usually limited to DynamicTable or DynamicTableRegion. Even though ndx-optogenetics defines a subtype of TimeIntervals with a custom DynamicTableRegion column, because get_class is not called on this type (a written, not generated Python class is provided instead), the issue does not appear when importing that extension.

I identified two working fixes for this but both feel like bandaids. Execution is not quite right in a couple of places. I need to think through more what would be the cleanest approach. I'll push a PR on Thursday.

[weiglszonja]

Thanks @rly for looking into this. Let me know how it goes.

[weiglszonja]

@rly just checking in, are there any updates on this? This issue is blocking me at the moment to be able to read the files with MatNWB. Will this fix require me to re-convert any NWB files, or can this be fixed on read?

[rly]

Hi @weiglszonja I'm working on a fix for this today. Sorry for the delay. The fix will only affect the writing of new files.

Any previously written NWB files have the incorrect namespace for the DynamicTableRegion types so they cannot be fixed on read, unless we add specific code to handle this in MatNWB and PyNWB.

You can correct these NWB files with a script that uses h5py to search for those attributes and correct them (inexpensive). This script, from, I believe, an earlier instance of encountering this bug, does that: https://gist.github.com/rly/7b3365d157ebb7d0a565bbd48eb12fdd#file-fix_dandiset_000114-py . You can run the script on a directory of NWB files.

[bendichter]

@rly have you integrated a fix for write yet?

[ehennestad]

@bendichter For MatNWB, Do you think I should add specific code to catch this and correct the namespace on read?

[bendichter]

Yeah I think that's probably smart