[SYNPY-1589] Implement "Evaluation" OOP model (#1244)

* TDD: initial proof of concept integration tests (async)

* [SYNPY-1589] Implement core functionality of Evaluation OOP model (#1245)

* Initialize protocol class + Evaluation dataclass

* exposing Evaluation API services. dataclass updates. syntax updates.

* dataclass methods are added which wrap API service functions

* fixed missing imports and syntax failures

* Evaluations can be imported like from synapseclient.models import Evaluation

* correct ACL inputs and output types. first pass at fixing async integration tests.

* fix import issues

* fix type errors. api module and its functions can now be imported individually. test improvements.

* add etag to prevent issues from OCC when updating an evaluation

* some formatting

* refactor: update and create will rely on the attributes changing in the class object itself rather than feeding it into the method. new to_synapse_request

* remove principal_id param. cannot be used. adding examples to docstrings

* patch all async tests

* style

* remove unnecessary imports

* complete refactor of store/update. store is now used for updating too

* add example to Evaluation dataclass

* new evaluation protocol class. moved out to protocols subfolder

* add synchronous integration tests

* new unit tests. updated integration tests. content_source is an immutable field. style

* show available access types

* remove todo

* get logger instance

* no need for request_type param in store_evaluation_async. removing CASE 2 for handling ID of new evals. remove integration test

* docstring example formatting

Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>

* update examples in docstrings

* the logger called in merge_dataclass_entities is now retrieved from active client

* remove _async suffix from evaluation_services functions

* style

* updated docstrings

* store -> create_or_update, updated tests

* style

* refactor update_acl dataclass method for user qol

* give users option to remove principals from ACL

* convert sync test methods to async so it calls schedule_for_cleanup fixture properly

* style

* if/elif -> if/else

* iteratively grab the attributes instead

* request_type is now a CREATE/UPDATE enum

* feed query params directly into client httpx response call so it can parse params with urlencode

* style

* grab the logger outside of the for-loop

---------

Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>

* [SYNPY-1589] Implement documentation of Evaluation OOP model (#1247)

* first draft of docs

* style

* update tutorial to reflect new code in tutorial script. update formatting of model examples.

* add more examples to API reference docs

* update tutorial source code

* enforce principal_id is not none

* move PRINCIPAL_ID assert. update tutorial .md

* Update docs/tutorials/python/evaluation.md

Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>

* force uppercase

* fix uppercase force

* additional prereq

* fix indentation. fix example

* change eval names

---------

Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com>

* no colons.

* always rely on client logger

* document refresh

* update sync update_acl examples

* fix vulnerability in assert statement

Author: jaymedina

docs/reference/experimental/async/evaluation.md

@@ -0,0 +1,21 @@
+# Evaluation
+
+Contained within this file are experimental interfaces for working with the Synapse Python
+Client. Unless otherwise noted these interfaces are subject to change at any time. Use
+at your own risk.
+
+## API reference
+
+::: synapseclient.models.Evaluation
+    options:
+        inherited_members: true
+        members:
+            - store_async
+            - get_async
+            - delete_async
+            - get_all_evaluations_async
+            - get_available_evaluations_async
+            - get_evaluations_by_project_async
+            - get_acl_async
+            - update_acl_async
+            - get_permissions_async

docs/reference/experimental/sync/evaluation.md

@@ -0,0 +1,21 @@
+# Evaluation
+
+Contained within this file are experimental interfaces for working with the Synapse Python
+Client. Unless otherwise noted these interfaces are subject to change at any time. Use
+at your own risk.
+
+## API reference
+
+::: synapseclient.models.Evaluation
+    options:
+        inherited_members: true
+        members:
+            - store
+            - get
+            - delete
+            - get_all_evaluations
+            - get_available_evaluations
+            - get_evaluations_by_project
+            - get_acl
+            - update_acl
+            - get_permissions

docs/tutorials/python/evaluation.md

@@ -0,0 +1,61 @@
+# Evaluations
+An Evaluation is essentially a container that organizes and manages the submission, assessment, and scoring of data, models, or other research artifacts.
+It allows teams to set up challenges where participants contribute their work, and those contributions can be systematically reviewed or scored.
+
+This tutorial will walk you through the basics of working with Evaluations using the Synapse Python client.
+
+## Tutorial Purpose
+In this tutorial you will:
+
+1. Create and update an Evaluation on Synapse
+1. Update the ACL (Access Control List) of an Evaluation on Synapse
+1. Retrieve and delete all Evaluations from a given Project
+
+## Prerequisites
+* You have completed the [Project](./project.md) tutorial, or have an existing Project on Synapse to work from
+* You have a Synapse user or team ID to share the evaluation with
+
+## 1. Create and update an Evaluation on Synapse
+
+In this first part, we'll be showing you how to interact with an Evaluation object as well as introducing you to its two core functionalities `store()` and `get()`.
+
+```python
+{!docs/tutorials/python/tutorial_scripts/evaluation.py!lines=5-46}
+```
+
+## 2. Update the ACL of an Evaluation on Synapse
+
+Like Synapse entities, Evaluations have ACLs that can be used to control who has access to your evaluations and what level of access they have. Updating the ACL of an Evaluation object is slightly different from updating other Evaluation components, because the ACL is not an attribute of the Evaluation object. Let's see an example of how this looks:
+
+```python
+{!docs/tutorials/python/tutorial_scripts/evaluation.py!lines=54-64}
+```
+
+You can also remove principals from an ACL by simply feeding `update_acl` an empty list for the `access_type` argument, like so:
+
+```python
+{!docs/tutorials/python/tutorial_scripts/evaluation.py!lines=66-67}
+```
+
+## 3. Retrieve and delete all Evaluations from a given Project
+
+Now we will show how you can retrieve lists of Evaluation objects, rather than retrieving them one-by-one with `get()`. This is a powerful tool if you want to perform the same action on all the evaluations in a given project, for example, like what we're about to do here:
+
+```python
+{!docs/tutorials/python/tutorial_scripts/evaluation.py!lines=69-75}
+```
+
+## Source code for this tutorial
+
+<details class="quote">
+  <summary>Click to show me</summary>
+
+```python
+{!docs/tutorials/python/tutorial_scripts/evaluation.py!}
+```
+</details>
+
+## References
+- [Evaluation][synapseclient.models.Evaluation]
+- [Project][synapseclient.models.Project]
+- [syn.login][synapseclient.Synapse.login]

docs/tutorials/python/tutorial_scripts/evaluation.py

@@ -0,0 +1,75 @@
+"""
+Here is where you'll find the code for the Evaluation tutorial.
+"""
+
+from synapseclient import Synapse
+from synapseclient.models import Evaluation, Project
+
+syn = Synapse()
+syn.login()
+
+# REQUIRED: Set this to the Synapse user ID or team ID you want to grant permissions to
+# Do NOT leave this as None - the script will not work properly
+PRINCIPAL_ID = None  # Replace with actual user/team ID
+
+# Retrieve the Project where your Evaluation will be stored
+project = Project(name="My uniquely named project about Alzheimer's Disease").get()
+project_id = project.id
+
+print(f"Working within Project: {project_id}")
+
+# Create a new Evaluation object
+evaluation = Evaluation(
+    name="My Challenge Evaluation for Study ABC - Round 1",
+    description="Evaluation for my data challenge",
+    content_source=project_id,
+    submission_instructions_message="Submit CSV files only",
+    submission_receipt_message="Thank you for your submission!",
+)
+
+# Create the Evaluation on Synapse
+evaluation.store()
+
+print("Evaluation has been created with the following name and description:")
+print(evaluation.name)
+print(evaluation.description)
+
+# Update the Evaluation object's name and description
+evaluation.name = "My Challenge Evaluation for Study XYZ - Round 1"
+evaluation.description = "Updated description for my evaluation"
+
+# Update the Evaluation on Synapse
+evaluation.store()
+
+print("Evaluation has been updated with the following name and description:")
+print(evaluation.name)
+print(evaluation.description)
+
+# Confirm what's in Synapse matches the evaluation stored
+from_synapse = Evaluation(id=evaluation.id).get()
+
+print("The following evaluation has been retrieved from Synapse:")
+print(from_synapse)
+
+# Update the Evaluation's ACL on Synapse by adding a new user
+assert (
+    PRINCIPAL_ID is not None
+), "PRINCIPAL_ID must be set to the Synapse user ID or team ID you want to grant permissions to."
+
+evaluation.update_acl(principal_id=PRINCIPAL_ID, access_type=["READ", "SUBMIT"])
+
+# Get the Evaluation's ACL to confirm the update
+acl = evaluation.get_acl()
+print("The following ACL has been retrieved from Synapse:")
+print(acl)
+
+# Now let's remove the user we just added from the Evaluation's ACL
+evaluation.update_acl(principal_id=PRINCIPAL_ID, access_type=[])
+
+# Finally let's retrieve all Evaluations stored within this project, including the one we just created
+evaluations_list = Evaluation.get_evaluations_by_project(project_id)
+
+# Let's delete the evaluation we created for this tutorial, and any other evaluations in this project (uncomment below to enable deletion)
+# for evaluation_to_delete in evaluations_list:
+#     print(f"Deleting evaluation: {evaluation_to_delete.name}")
+#     evaluation_to_delete.delete()

mkdocs.yml

@@ -27,6 +27,7 @@ nav:
           - Project: tutorials/python/project.md
           - Folder: tutorials/python/folder.md
           - File: tutorials/python/file.md
+          - Evaluation: tutorials/python/evaluation.md
           - Annotation: tutorials/python/annotation.md
           # - Versions: tutorials/python/versions.md
           # - Activity/Provenance: tutorials/python/activity.md
@@ -85,6 +86,7 @@ nav:
           - Project: reference/experimental/sync/project.md
           - Folder: reference/experimental/sync/folder.md
           - File: reference/experimental/sync/file.md
+          - Evaluation: reference/experimental/sync/evaluation.md
           - Table: reference/experimental/sync/table.md
           - VirtualTable: reference/experimental/sync/virtualtable.md
           - Dataset: reference/experimental/sync/dataset.md
@@ -104,6 +106,7 @@ nav:
               - Project: reference/experimental/async/project.md
               - Folder: reference/experimental/async/folder.md
               - File: reference/experimental/async/file.md
+              - Evaluation: reference/experimental/async/evaluation.md
               - Table: reference/experimental/async/table.md
               - VirtualTable: reference/experimental/async/virtualtable.md
               - Dataset: reference/experimental/async/dataset.md

synapseclient/api/__init__.py

@@ -63,6 +63,17 @@
     update_activity,
     update_entity_acl,
 )
+from .evaluation_services import (
+    create_or_update_evaluation,
+    delete_evaluation,
+    get_all_evaluations,
+    get_available_evaluations,
+    get_evaluation,
+    get_evaluation_acl,
+    get_evaluation_permissions,
+    get_evaluations_by_project,
+    update_evaluation_acl,
+)
 from .file_services import (
     AddPartResponse,
     get_file_handle,
@@ -261,4 +272,14 @@
     "is_user_certified",
     # table_services
     "create_table_snapshot",
+    # evaluation_services
+    "create_or_update_evaluation",
+    "get_evaluation",
+    "get_evaluations_by_project",
+    "get_all_evaluations",
+    "get_available_evaluations",
+    "delete_evaluation",
+    "get_evaluation_acl",
+    "update_evaluation_acl",
+    "get_evaluation_permissions",
 ]