Creation of pipeline to publish dts python package to pypi #43

RyanLettieri · 2025-03-05T00:59:51Z

This PR adds in a new workflow yaml that will be responsible for publishing the durabletask scheduler package to pypi. It also adds in e2e tests for DTS

berndverst

A few more changes required to ensure we produce correct releases.

Note that we use tags of form azuremanaged-v* instead of v* -- this is because the latter should be reserved for the main durabletask python sdk which can have a separate and independent release cadence and cycle. We want to decouple the publishing of SDKs.

berndverst · 2025-03-06T02:15:21Z

.github/workflows/publish-dts-sdk.yml

+        uses: actions/checkout@v4
+
+      - name: Extract version from tag
+        run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_ENV  # Extract version from the tag


Suggested change

run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_ENV # Extract version from the tag

run: echo "VERSION=${GITHUB_REF#refs/tags/azuremanaged-v}" >> $GITHUB_ENV # Extract version from the tag

FYI - right now we are not doing anything with the version. However, if we wanted to - we could use this version from the tag and DYNAMICALLY replace the value in pyproject.toml BEFORE running the build and publish commands. This way releasing a new SDK only requires tagging... otherwise what would be required is also a manual PR to change the version in pyproject -- and in fact what can happen is that the pyproject file version and the tag version are not aligned.

Consider having a script here that uses sed or similar to replace the version within the pyproject.toml

.github/workflows/publish-dts-sdk.yml

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

berndverst · 2025-03-19T23:52:23Z

.github/workflows/pr-validation.yml

+        pip install -r requirements.txt
+    - name: Lint with flake8
+      run: |
+        flake8 . --count --show-source --statistics --exit-zero


Suggested change

flake8 . --count --show-source --statistics --exit-zero

flake8 . --count --show-source --statistics

We want this step to fail if there are linter issues, so let's not use --exit-zero

I see that was there before - but let's actually change this!

This leads to a lot of lint errors that were being ignored. I'll create an additional PR afterwards that removes this flag and addresses the linting problems so I don't clutter this PR too much

Ok, please do that! Just because existing linter errors exist doesn't mean we should leave them in place.

.github/workflows/pr-validation.yml

.github/workflows/publish-dts-sdk.yml

examples/dts/requirements.txt

.github/workflows/publish-dts-sdk.yml

berndverst

Left some comments - just some pipeline changes and dependency changes. Nothing major.

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

.github/workflows/publish-dts-sdk.yml

) * Creating of pipeline to publish dts python package to pypi Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Upgrading version of durabletask-azuremanaged from 0.1b1 to 0.1 Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Updating versioning on packages Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Incrementing version to allign with pypi Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Adressing majority of first round of feedback Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Updating pipeline to have linting Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Updating versions in pyproject.toml Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Updating working dirs in yml Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Adding requirements.txt Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Moving durabletask tests into specific dir and more Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Fixing more paths Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * ATtemptign to ignore durabletask-azuremanaged folder Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * installing dts dependencies Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Changing path for requirements.txt Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Moving init.py Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Updating readme and some tests Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Running all dts tests in publish pipeline Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Removing PYTHONPATH and installing regular deps Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Adding timeout to dts orchestration e2e test Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Removing suspend and continue as new tests from dts Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Removing raise event timeout tests Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Only runnign publish on tag push Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Changing dts action to run on tag creation Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Updating tag name Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Adressing review feedback Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Fixing run requirements in actions and adding exit-zero Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> * Update .github/workflows/publish-dts-sdk.yml --------- Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com> Co-authored-by: Bernd Verst <github@bernd.dev> Signed-off-by: Albert Callarisa <albert@diagrid.io>

RyanLettieri changed the title ~~Creating of pipeline to publish dts python package to pypi~~ Creation of pipeline to publish dts python package to pypi Mar 5, 2025

berndverst requested changes Mar 6, 2025

View reviewed changes

berndverst reviewed Mar 6, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Outdated Show resolved Hide resolved

berndverst reviewed Mar 6, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Show resolved Hide resolved

berndverst reviewed Mar 6, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Outdated Show resolved Hide resolved

berndverst reviewed Mar 6, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Show resolved Hide resolved

RyanLettieri marked this pull request as ready for review March 6, 2025 16:49

RyanLettieri added 23 commits March 10, 2025 15:09

Creating of pipeline to publish dts python package to pypi

9a7569a

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Upgrading version of durabletask-azuremanaged from 0.1b1 to 0.1

2431d2f

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Updating versioning on packages

67fe8c1

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Incrementing version to allign with pypi

58f0f36

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Adressing majority of first round of feedback

4f7615a

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Updating pipeline to have linting

46364b6

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Updating versions in pyproject.toml

4c4e6bf

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Updating working dirs in yml

717ab88

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Adding requirements.txt

e46d90a

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Moving durabletask tests into specific dir and more

30bce57

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Fixing more paths

59b21ce

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

ATtemptign to ignore durabletask-azuremanaged folder

29ddd2f

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

installing dts dependencies

8716a9e

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Changing path for requirements.txt

406d83c

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Moving init.py

a654c0d

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Updating readme and some tests

fe7e102

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Running all dts tests in publish pipeline

9628875

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Removing PYTHONPATH and installing regular deps

7337186

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Adding timeout to dts orchestration e2e test

cb824d8

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Removing suspend and continue as new tests from dts

76307eb

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Removing raise event timeout tests

3c065ae

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Only runnign publish on tag push

faa3d91

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Changing dts action to run on tag creation

89373e2

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

RyanLettieri force-pushed the lettieri/pypi-pipeline branch from 1428556 to 89373e2 Compare March 19, 2025 23:39

Updating tag name

aa1a8aa

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

berndverst reviewed Mar 19, 2025

View reviewed changes

.github/workflows/pr-validation.yml Show resolved Hide resolved

berndverst reviewed Mar 19, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Outdated Show resolved Hide resolved

berndverst reviewed Mar 19, 2025

View reviewed changes

examples/dts/requirements.txt Outdated Show resolved Hide resolved

berndverst reviewed Mar 20, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Outdated Show resolved Hide resolved

berndverst requested changes Mar 20, 2025

View reviewed changes

RyanLettieri added 2 commits March 20, 2025 10:58

Adressing review feedback

7cda76c

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

Fixing run requirements in actions and adding exit-zero

b53adda

Signed-off-by: Ryan Lettieri <ryanLettieri@microsoft.com>

berndverst reviewed Mar 21, 2025

View reviewed changes

.github/workflows/publish-dts-sdk.yml Outdated Show resolved Hide resolved

Update .github/workflows/publish-dts-sdk.yml

88e86d1

berndverst approved these changes Mar 21, 2025

View reviewed changes

berndverst merged commit 8ad3849 into main Mar 21, 2025
9 checks passed

berndverst deleted the lettieri/pypi-pipeline branch March 21, 2025 18:22

	run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_ENV # Extract version from the tag
	run: echo "VERSION=${GITHUB_REF#refs/tags/azuremanaged-v}" >> $GITHUB_ENV # Extract version from the tag

	flake8 . --count --show-source --statistics --exit-zero
	flake8 . --count --show-source --statistics

Creation of pipeline to publish dts python package to pypi #43

Creation of pipeline to publish dts python package to pypi #43

Uh oh!

Conversation

RyanLettieri commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

berndverst left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

berndverst Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

berndverst Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

berndverst Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

berndverst Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

RyanLettieri Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

berndverst Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

berndverst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RyanLettieri commented Mar 5, 2025 •

edited

Loading

berndverst left a comment •

edited

Loading