Error when reading NWB file in MatNWB, but not PyNWB

[amadaabrego]

Hi!

I have a dataset from a collaborator in NWB format. I can upload it using pynwb, but when I try to do it in matlab, I get the following error

read_nwbfile = nwbRead('sub-TA525_task-places_ieeg.nwb')

Error using types.core.ElectricalSeries/validate_data_unit_
Unable to set the 'data_unit' property of class 'ElectricalSeries' because it is read-only.

Error in types.core.TimeSeries/set.data_unit (line 94)
obj.data_unit = obj.validate_data_unit(val);

Error in types.core.TimeSeries (line 61)
obj.data_unit = p.Results.data_unit;

Error in types.core.ElectricalSeries (line 22)
obj = obj@types.core.TimeSeries(varargin{:});

Error in io.parseGroup (line 85)
parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 79)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

For what it's worth, the dataset was written in Matlab. And it was generated years ago. Collaborator never had any issues with it, so they think it might be something in a newer update.

I'm not sure what is the best way to address this. I have gone through all the functions, and have not found a way to fix it. Any ideas?

[CodyCBakerPhD]

attn @lawrence-mbf or @bendichter

[lawrence-mbf]

@amadaabrego what is the version of the schema for the file? You can run util.getSchemaVersion(<filename>) to get a file's schema version. FWIW, ElectricalSeries in 2.6.0 does not have the data_unit validation defined, just a default setter in the constructor.

[ehennestad]

If you change the commit you are using, you might also need to run generateCore again before opening the file.

[ehennestad]

@amadaabrego Just to confirm, the value for the unit is different than volts, right? Is it a question of casing being different, or is it another unit altogether like millivolts?

[amadaabrego]

@ehennestad Our collaborator shared with me the script they use to make the NWB files - our lab is going to adapt it, so I'm using that to base ours, but I'm trying to get familiarized with the dataset right now. In their script, they have their units as 'V'. I'm not sure if that might be the issue

I did not have any issues with reading the file into Python. I was able to extract the data matrix and everything. And, I'm trying to make this work with Matlab because we will be using Brainstorm for some ECoG data visualization

[ehennestad]

Yes, the issue is setting the value to "V".

The NWB schemas specify that the unit for ElectricalSeries data should be "volts".
https://github.com/NeurodataWithoutBorders/nwb-schema/blob/d65d42257003543c569ea7ac0cd6d7aee01c88d6/core/nwb.ecephys.yaml#L35-L42

I was using matnwb and I realised that I could set the unit to whatever and I therefore proposed a fix to make this property read-only in matnwb to adhere to the schema. Apparently such a limitation does not exist in pynwb.

Maybe @oruebel @rly or @bendichter could comment on this, whether this is a bug in pynwb or if I am misunderstanding the schema definition?

[ehennestad]

@amadaabrego

I would suggest that you do not fill in the value of units, i.e skip setting unit to "V". It should by default be set to volts. If you still need to read in files in MATLAB which are created earlier with python and has the unit set to "V", let me know, and I will propose a fix where instead of erroring, there will be a warning instead.

[ehennestad]

@ehennestad if there is a better way to resolve this, that would be good too.

@lawrence-mbf
It might be better to throw a warning rather than an error.

And I guess it is best to throw a warning and keep the value from the file, not enforce volts.

Another question is then whether this deviation from schema is also present in pynwb?

[oruebel]

Another question is then whether this deviation from schema is also present in pynwb?

In PyNWB, the unit for ElectricalSeries is set to a fixed value here:

https://github.com/NeurodataWithoutBorders/pynwb/blob/65b2a7d0ea642132a7a5fcb5a5199b675f092e0b/src/pynwb/ecephys.py#L93

[ehennestad]

This makes it fixed in the constructor, but it does not stop anyone from overwriting the value after an electrical series is created, like in the example below:

raw_data = np.random.randn(50, 12)
raw_electrical_series = ElectricalSeries(
    name="ElectricalSeries",
    data=raw_data,
    electrodes=all_table_region,
    starting_time=0.0,  # timestamp of the first sample in seconds relative to the session start time
    rate=20000.0,  # in Hz
)

# Set the unit for the electrical series
raw_electrical_series.unit = "V"

[adredish]

Did anyone solve this for matnwb?
I'm getting the same error.

[adredish]

Something is very weird in this code. I found where the error is happening, changed the code, but it keeps getting changed back. Is this code regenerating classes?

This code seems to also be trying to handle and rethrow errors, which means that you can't use the debugger to get the actual debugging stack.

[adredish]

This code is definitely regenerating classes.

[bendichter]

Yes, MatNWB generates the MATLAB code

[ehennestad]

@adredish Can you share a sample file where you experience this issue? Is this a file you have created yourself, or a file from a collaborator / online repository (i.e dandi)?

The problem is most likely that a value in the file is not according to the NWB schema, and this causes a read error in MatNWB. As discussed above, it might be better to show a warning for a situation like this. I created an issue on MatNWB to follow up on this

[adredish]

It's from the following DANDI collection: [https://dandiarchive.org/dandiset/000978/0.240511.0307].

I'm a pretty sophisticated Matlab programmer but the structure of the MatNWB system --- try/catch and rethrowing errors without actually solving them (which precludes using the debugger correctly), and generating code each time (which precludes implementing trivial correction changes for problems that don't affect your actual data) --- has made this impossible to fix, and made it impossible to access the data.

One solution would be to add a check for if the classdef file already exists. I could then hack the generated classdef file so I could read the data.

[ehennestad]

has made this impossible to fix, and made it impossible to access the data.

Yes, you have discovered a issue with the software that we need to fix centrally. I am sorry for the delay this is causing you, I will work on getting a patch ready for you soon.

One solution would be to add a check for if the classdef file already exists. I could then hack the generated classdef file so I could read the data.

I think it is better to fix this once and for all, so that this problem will not affect other users in the future.

[ehennestad]

@adredish

I made a temporary fix for your particular issue on this branch:
https://github.com/NeurodataWithoutBorders/matnwb/tree/support-reading-files-with-invalid-values

Please try it out and let me know if that helps you. We will continue working towards implementing this into the main branch and add it to a future release.

[adredish]

Thanks. I was able to hack the main to work. It turns out that the "ignorecache" really means "don't regenerate the classdefs".

I have also tested your version, which works. It provides a warning for the invalid value, but allows it to proceed. It looks like the data is loading correctly.

Thank you for responding so quickly.