-
Notifications
You must be signed in to change notification settings - Fork 73
[SYNPY-1590] Implement documentation of Submission(+Status, +Bundle) OOP model #1252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jaymedina
wants to merge
46
commits into
synpy-1590-submission-model-functionality
Choose a base branch
from
synpy-1590-submission-model-docs
base: synpy-1590-submission-model-functionality
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
a061fb4
tdd initial tests
jaymedina d85a9d7
initial intro of dataclasses
jaymedina 50498b7
expose api services for submission object
jaymedina 94cedc6
style and update docstring
jaymedina cd19e0c
add submission and submissionstatus models
jaymedina 99cba20
add submission status retrieval and update methods; remove empty subm…
jaymedina 9ca2448
pipe query params directly into restAPI httpx requests
jaymedina f0d7ecc
new dataclass object submission_bundle
jaymedina 91b947b
move submission services functions to evaluation_services.py
jaymedina a6b46d9
renaming imports, to_synapse_request, request body refactor
jaymedina b68bd8e
patching up store method signature
jaymedina 5a44f07
update docs
jaymedina 65664d1
new suite of tests
jaymedina d214673
submissionstatus rework as a mutable object
jaymedina 2781797
bug fix for Statuses: updated to_synapse_request to follow same patte…
jaymedina 3230b1f
replace != with is not for full object comparison (not just keys)
jaymedina 2d43e5d
expose the is_private arg for to_submission_status_annotations ONLY F…
jaymedina d4d63ba
fixed submission status/submission annotations
jaymedina 8b9f9c9
add support for legacy annotations
jaymedina 6aa914a
remove debug prints
jaymedina 2d67fcc
get_all_submission_statuses now returns a list of substat objects
jaymedina c368e49
docstring updates
jaymedina fbd43f6
update submissionbundle docstrings, add more examples
jaymedina d5fac7e
initial sync test for status. moved evaluation_id retrieval to fill_f…
jaymedina 47797c0
update submissionBundle submissionstatus with evaluation_id
jaymedina e8e5361
patch sync substatus integ tests. style.
jaymedina 58ffe42
fix submissionStatus integ tests and has_changed attribute
jaymedina f34667d
new substatus async integ tests. can_cancel can now be modified by an…
jaymedina 5e762f0
new test class for submission cancel functionality
jaymedina 1bd7f4b
substatus async unit tests
jaymedina 4c8904d
remove compare=false for some attributes. update sync unit tests
jaymedina a223320
add submissionBundle integration tests
jaymedina f059751
add submissionBundle unit tests
jaymedina 3cb1915
remove unnecessary imports and add style
jaymedina 1fb0fbf
get_evaluation_submissions returns generator object
jaymedina 281ed3d
get_user_submissions returns generator object
jaymedina 1168cae
submissionBundle methods return generators
jaymedina a9977b2
address final todo: implement docker_tag, and note in docs that versi…
jaymedina 689fcb1
add async page in api references
jaymedina c54609b
add initial submission, status, bundle docs
jaymedina e0a79cc
updated tutorial purpose. add api reference to navbar
jaymedina 35efed7
new tutorial scripts
jaymedina e07206d
remove try/excepts. add line references. add resources and source cod…
jaymedina e193753
add reference to File model
jaymedina b710da0
deprecate old submission model
jaymedina 7917011
style
jaymedina File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Submission | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.Submission | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - store_async | ||
| - get_async | ||
| - delete_async | ||
| - cancel_async | ||
| - get_evaluation_submissions_async | ||
| - get_user_submissions_async | ||
| - get_submission_count_async |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Submission Bundle | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionBundle | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get_evaluation_submission_bundles_async | ||
| - get_user_submission_bundles_async |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Submission Status | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionStatus | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get_async | ||
| - store_async | ||
| - get_all_submission_statuses_async | ||
| - batch_update_submission_statuses_async |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Submission | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.Submission | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - store | ||
| - get | ||
| - delete | ||
| - cancel | ||
| - get_evaluation_submissions | ||
| - get_user_submissions | ||
| - get_submission_count |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Submission Bundle | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionBundle | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get_evaluation_submission_bundles | ||
| - get_user_submission_bundles |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Submission Status | ||
|
|
||
| Contained within this file are experimental interfaces for working with the Synapse Python | ||
| Client. Unless otherwise noted these interfaces are subject to change at any time. Use | ||
| at your own risk. | ||
|
|
||
| ## API Reference | ||
|
|
||
| ::: synapseclient.models.SubmissionStatus | ||
| options: | ||
| inherited_members: true | ||
| members: | ||
| - get | ||
| - store | ||
| - get_all_submission_statuses | ||
| - batch_update_submission_statuses |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| # Submissions, SubmissionStatuses, SubmissionBundles | ||
| Users can work with Submissions on the python client, since these objects are part of the Evaluation API data model. A user in a Synapse Evaluation can submit a Synapse Entity as a Submission to that Evaluation. Submission data is owned by the parent Evaluation, and is immutable. | ||
|
|
||
| The data model includes additional objects to support scoring of Submissions and convenient data access: | ||
|
|
||
| - **SubmissionStatus**: An object used to track scoring information for a single Submission. This object is intended to be modified by the users (or test harnesses) managing the Evaluation. | ||
| - **SubmissionBundle**: A convenience object to transport a Submission and its accompanying SubmissionStatus in a single web service call. | ||
|
|
||
| This tutorial will demonstrate how to work with all 3 object types using the python client for 2 different use-cases: | ||
|
|
||
| 1. Participating in a Synapse challenge | ||
| 1. Organizing a Synapse challenge | ||
|
|
||
|
|
||
| ## Tutorial Purpose | ||
| In this tutorial: | ||
|
|
||
| As a participant of a Synapse challenge, you will | ||
|
|
||
| 1. Make a submission to an existing evaluation queue on Synapse | ||
| 1. Fetch your existing submission | ||
| 1. Count your submissions | ||
| 1. Fetch all of your submissions from an existing evaluation queue on Synapse | ||
| 1. Check the status of your submission | ||
| 1. Cancel your submission | ||
|
|
||
| As an organizer of a Synapse challenge, you will | ||
|
|
||
| 1. Annotate a submission to score it | ||
| 1. Batch-update submission statuses | ||
| 1. Fetch the submission bundle for a given submission | ||
| 1. Allow cancellation of submissions | ||
| 1. Delete submissions | ||
|
|
||
| ## Prerequisites | ||
| * You have completed the [Evaluation](./evaluation.md) tutorial, or have an existing Evaluation on Synapse to work from | ||
| * You have an existing entity with which to make a submission (can be a [File](./file.md) or Docker Repository) | ||
| * You have the correct permissions on the Evaluation queue for your desired tutorial section (participant or organizer) | ||
|
|
||
| ## 1. Participating in a Synapse challenge | ||
|
|
||
| ### 1. Make a submission to an existing evaluation queue on Synapse | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=30-51} | ||
| ``` | ||
|
|
||
| ### 2. Fetch your existing submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=56-71} | ||
| ``` | ||
|
|
||
| ### 3. Count your submissions | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=72-92} | ||
| ``` | ||
|
|
||
| ### 4. Fetch all of your submissions from an existing evaluation queue on Synapse | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=93-107} | ||
| ``` | ||
|
|
||
| ### 5. Check the status of your submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=108-130} | ||
| ``` | ||
|
|
||
| ### 6. Cancel your submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!lines=131-157} | ||
| ``` | ||
|
|
||
| ## 2. Organizing a Synapse challenge | ||
|
|
||
| ### 1. Annotate a submission to score it | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=29-57} | ||
| ``` | ||
|
|
||
| ### 2. Batch-update submission statuses | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=58-101} | ||
| ``` | ||
|
|
||
| ### 3. Fetch the submission bundle for a given submission | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=102-142} | ||
| ``` | ||
|
|
||
| ### 4. Allow cancellation of submissions | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=143-195} | ||
| ``` | ||
|
|
||
| ### 5. Delete submissions | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!lines=196-229} | ||
| ``` | ||
|
|
||
| ## Source code for this tutorial | ||
|
|
||
| <details class="quote"> | ||
| <summary>Click to show me (source code for Participant)</summary> | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_participant.py!} | ||
| ``` | ||
| </details> | ||
|
|
||
| <details class="quote"> | ||
| <summary>Click to show me (source code for Organizer)</summary> | ||
|
|
||
| ```python | ||
| {!docs/tutorials/python/tutorial_scripts/submission_organizer.py!} | ||
| ``` | ||
| </details> | ||
|
|
||
| ## References | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing submission? |
||
| - [Evaluation][synapseclient.models.Evaluation] | ||
| - [File][synapseclient.models.File] | ||
| - [syn.login][synapseclient.Synapse.login] | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not seeing this in the index page of tutorial: