returned dtype is not always consistent.

[ceesem]

In v1dd, I am getting some situations where pt_position_x/y/z is being returned as a string[pyarrow] instead of an int64[pyarrow] with the new typing.

For example,

pf_df = client.materialize.tables.cell_class_coarse.get_all.tables.proofreading_status_and_strategy.get_all(desired_resolution=[1,1,1], split_positions=True)

gives (using pf_df.dtypes)

id                                       int64[pyarrow]
created                  timestamp[ns, tz=UTC][pyarrow]
superceded_id                           string[pyarrow]
valid                                     bool[pyarrow]
classification_system                   string[pyarrow]
cell_type                               string[pyarrow]
pt_position_x                           string[pyarrow]
pt_position_y                           string[pyarrow]
pt_position_z                           string[pyarrow]
pt_supervoxel_id                         int64[pyarrow]
pt_root_id                               int64[pyarrow]

However, other tables work fine.

nuc_df = client.materialize.tables.cell_class_coarse.get_all(desired_resolution=[1,1,1], split_positions=True)

correctly returns (using nuc_df.dtypes)

id                                       int64[pyarrow]
created                  timestamp[ns, tz=UTC][pyarrow]
valid                                     bool[pyarrow]
volume                                  double[pyarrow]
pt_position_x                            int64[pyarrow]
pt_position_y                            int64[pyarrow]
pt_position_z                            int64[pyarrow]
bb_start_position_x                      int64[pyarrow]
bb_start_position_y                      int64[pyarrow]
bb_start_position_z                      int64[pyarrow]
bb_end_position_x                        int64[pyarrow]
bb_end_position_y                        int64[pyarrow]
bb_end_position_z                        int64[pyarrow]
pt_supervoxel_id                         int64[pyarrow]
pt_root_id                               int64[pyarrow]
id_ref                                   int64[pyarrow]
created_ref              timestamp[ns, tz=UTC][pyarrow]
valid_ref                                 bool[pyarrow]
target_id                                int64[pyarrow]
classification_system                   string[pyarrow]
cell_type                               string[pyarrow]
dtype: object

[ceesem]

As an additional data point, it seems to depend on setting desired_resolution.

pf_df = client.materialize.tables.cell_class_coarse.get_all.tables.proofreading_status_and_strategy.get_all(split_positions=True)

returns the expected types.

[ceesem]

Ah, I think I understand the conceptual issue. I think that the columns are somehow expected to be integers, but the resolution conversion is making the values become floats. For whatever reason, pyarrow is falling back to handling these as strings rather than converting to floats.