[SYNPY-1700] Updating recordset to include the validation_file_handle_id

[BryanFauble]

Problem:

• There is a new validation_file_handle_id field for the RecordSet that points to a validation CSV file denoting the specific issues with the validation rules.
• Static type checking had issues with the TypeVar strings that we were using before, and not actually rendering out to Pandas types

Solution:

• Adding the new field, and allowing the detailed results to come back via a Pandas DataFrame for analysis

Testing:

• Integration testing to verify logic, I also verified that suggestions in the how-to guide for curator as well

[BryanFauble]

I created https://sagebionetworks.jira.com/browse/SYNPY-1712 so we can revisit this and make doing this work a bit easier for folks

[BryanFauble]

I made this change because if you do have a previous session open, and you are not aware that this defaults to True it could drop you into a session you did not intend to and overwrite, or delete data in the RecordSet.

[thomasyu888]

I see, so the default behavior is always going to be creation of a new session? Which means data within the working session previously won't be included?

[BryanFauble]

In this case that is correct unless they change that boolean to True.

Users may also use the list method as well to get a list of their active grid sessions and select the correct one from there.

[thomasyu888]

This is fantastic

[thomasyu888]

🔥 Thanks for the great work here. I tagged @aditigopalan for a review as well, but I'll defer a final review to another person on the team.

[linglp]

Hi @BryanFauble and @thomasyu888 ! The code looks good to me, but I am confused when I tested out the functionality. I started by creating a record set, a task, and a grid by using a test data schema:

record_set, task, grid = create_record_based_metadata_task(
    synapse_client=syn,
    project_id="syn71816385",
    folder_id="syn71816388",
    record_set_name="BiospecimenMetadata_RecordSet",
    record_set_description="RecordSet for biospecimen metadata curation",
    curation_task_name="BiospecimenMetadataTemplate",
    upsert_keys=["specimenID"],
    instructions="Please curate this metadata according to the schema requirements",
    schema_uri="sage.schemas.v2571-nf.BiospecimenTemplate.schema-9.14.0"
)

I entered some data in the grid on the UI, but when I tried to download the validation result:

async def main():
    syn = Synapse()
    syn.login()
    # Assuming you have a RecordSet with a bound schema
    record_set = await RecordSet(id="syn71816405").get_async()
    # Create and export Grid session to generate validation results
    grid = await Grid(record_set_id=record_set.id).create_async()
    await grid.export_to_record_set_async()
    await grid.delete_async()
    # Re-fetch the RecordSet to get updated validation_file_handle_id
    record_set = await record_set.get_async()
    # Get the detailed validation results
    results_df = await record_set.get_detailed_validation_results_async(download_location="/Users/lpeng/code/synapsePythonClient")
    print('result df', results_df)
    # Analyze the results
    print(f"Total rows: {len(results_df)}")
    print(f"Columns: {results_df.columns.tolist()}")
    # Filter for valid and invalid rows
    # Note: is_valid is boolean (True/False) for validated rows
    valid_rows = results_df[results_df['is_valid'] == True]  # noqa: E712
    invalid_rows = results_df[results_df['is_valid'] == False]  # noqa: E712
    print(f"Valid rows: {len(valid_rows)}")
    print(f"Invalid rows: {len(invalid_rows)}")
    # View invalid rows with their error messages
    if len(invalid_rows) > 0:
        print(invalid_rows[['row_index', 'validation_error_message']])
asyncio.run(main())

Without modifying anything in the grid, I got different validation results:
The first time running the code:

"row_index","is_valid","validation_error_message","all_validation_messages"
"0","true",,

The second time running the code:

"row_index","is_valid","validation_error_message","all_validation_messages"
"0",,,

Any ideas why this is happening?

[linglp]

Nit: the test name is misleading. The name suggests that an error is raised, but in this case, it only raises a warning?

[SageGJ]

would it be helpful for this id to link to an actual record set? right now it leads to a Page Unavailable message

[SageGJ]

Would we want to display a message in the cases of ImportErrors or not type checking to notify users that the specific types weren't available?