APP-9587 : Using Pubsub for message processors #898
+828
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
📜 Docstring Coverage ReportRESULT: PASSED (minimum: 30.0%, actual: 76.6%) Detailed Coverage Report |
📦 Trivy Vulnerability Scan Results
Report Summary
Scan Result Details✅ No vulnerabilities found during the scan for |
📦 Trivy Secret Scan Results
Report Summary
Scan Result Details✅ No secrets found during the scan for |
|
🛠 Docs available at: https://k.atlan.dev/application-sdk/add-message-processor |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Collaborator
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
|
🛠 Full Test Coverage Report: https://k.atlan.dev/coverage/application-sdk/pr/898 |
niteesh-atlan
changed the title
niteesh-atlan
requested review from
AtMrun,
firecast and
inishchith
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces Pub/Sub messaging support for message processors using Dapr, enabling applications to subscribe to message topics and process incoming messages through configurable handlers.
Key changes:
- Added Dapr pubsub component configuration with support for both in-memory and Kafka backends
- Introduced
PubSubSubscriptionandBulkSubscribemodels for configuring message subscriptions - Implemented automatic registration of message handler endpoints and Dapr subscription generation
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| components/messaging.yaml | New Dapr pubsub component configuration file with in-memory and commented Kafka configurations |
| components/eventstore.yaml | Minor formatting fix to ensure both webhook URLs remain commented out |
| application_sdk/server/fastapi/models.py | Added BulkSubscribe and PubSubSubscription models to support message subscription configuration |
| application_sdk/server/fastapi/init.py | Integrated messaging subscriptions into server initialization, router registration, and Dapr subscription endpoint |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| pubsub_component_name: str | ||
| topic: str | ||
| route: str | ||
| message_handler: Callable[[Any], Any] # Required callback function |
There was a problem hiding this comment.
The type hint for message_handler should be more specific to indicate it can be an async callable (coroutine). The current type hint Callable[[Any], Any] doesn't accurately reflect that FastAPI routes added via add_api_route typically expect async functions. Consider using Callable[[Any], Coroutine[Any, Any, Any]] or Union[Callable[[Any], Any], Callable[[Any], Coroutine[Any, Any, Any]]] to support both sync and async handlers.
components/messaging.yaml
Outdated
| type: pubsub.in-memory | ||
| version: v1 | ||
| # Bulk subscribe for in-memory pubsub has a batch size of 1 as it is not supported. | ||
| # B |
There was a problem hiding this comment.
Incomplete comment: Line 9 contains only "# B" which appears to be an incomplete or orphaned comment. This should either be completed with meaningful content or removed.
Suggested change
| # B |
components/messaging.yaml
Outdated
| # B | ||
|
|
||
| ## Use the below component for Kafka. Kafka supports bulk subscribe. | ||
| # Bulk subscribe is |
There was a problem hiding this comment.
Incomplete comment: Line 12 states "# Bulk subscribe is" but doesn't complete the sentence. This comment should be completed to explain what bulk subscribe is or how it works with Kafka.
Suggested change
| # Bulk subscribe is | |
| # Bulk subscribe is enabled for Kafka via the bulkSubscribe metadata below, allowing messages to be delivered in batches instead of individually. |
niteesh-atlan
changed the title
3 tasks
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: niteesh-atlan <niteesh.hegde@atlan.com>
firecast
reviewed
Jan 6, 2026
Member
firecast
left a comment
firecast
left a comment
There was a problem hiding this comment.
We also need docs on how can the handlers be written and a typedefinition defined for it as arguments so that handlers are specific
| subscription.message_handler, | ||
| methods=["POST"], | ||
| ) | ||
| self.app.include_router(messaging_router, prefix="/message-processor") |
Member
There was a problem hiding this comment.
Lets have the prefix as subscriptions
Member
There was a problem hiding this comment.
Do we also need to version this path? Check how events are being handled
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
APP-9587 : Using Pubsub for message processors
Scope
Sample app Pr - atlanhq/atlan-sample-apps#119
Additional context (e.g. screenshots, logs, links)
Checklist
Copyleft License Compliance