feat: integration service sync to repositories #3722
Conversation
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
2 similar comments
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
1 similar comment
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| }), | ||
| }, | ||
| segmentOptions, | ||
| PlatformType.GITHUB, |
There was a problem hiding this comment.
Missing repository sync for GitHub and GitLab integrations
The PR adds PlatformType.GITHUB and PlatformType.GITLAB parameters to gitConnectOrUpdate calls, which causes the internal mapUnifiedRepositories call to be skipped (due to the if (!sourcePlatform) check at line 1422). However, unlike githubNangoConnect (which adds an external call at line 961) and gerritConnectOrUpdate (which adds an external call at line 1789), neither mapGithubRepos nor mapGitlabRepos add the required external mapUnifiedRepositories call. This means when these methods are called directly via API endpoints (githubMapRepos.ts, gitlabMapRepos.ts) or from updateGithubIntegrationSettings, repositories won't be synced to public.repositories, breaking the PR's stated goal of consistent repository mapping across all platforms.
Additional Locations (1)
|
|
||
| const txOptions = { ...this.options, transaction } | ||
| const txService = new IntegrationService(txOptions) | ||
| await txService.mapUnifiedRepositories(PlatformType.GERRIT, integration.id, mapping) |
There was a problem hiding this comment.
Gerrit repository sync fails when Git disabled
The mapUnifiedRepositories call for Gerrit is placed outside the if (integrationData.remote.enableGit) block, meaning it executes unconditionally. However, buildRepositoryPayloads requires a Git integration to exist - it calls IntegrationRepository.findByPlatform(PlatformType.GIT, ...) and throws Error400 if not found. When enableGit is false, no Git integration is created by gitConnectOrUpdate, so the subsequent mapUnifiedRepositories call will fail with "Git integration not found for segment". The mapUnifiedRepositories call should be inside the if (enableGit) block.
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
| sourceIntegrationId: string, | ||
| mapping: { [url: string]: string }, | ||
| ) { | ||
| const transaction = await SequelizeRepository.createTransaction(this.options) |
There was a problem hiding this comment.
Nested transaction breaks transactional integrity across platforms
The mapUnifiedRepositories method always creates a new transaction instead of joining an existing one. Callers like gitConnectOrUpdate, gerritConnectOrUpdate, and gitlabConnectOrUpdate pass transaction context via new IntegrationService(txOptions), expecting the method to participate in the outer transaction. However, line 3148 unconditionally calls createTransaction(this.options), ignoring any existing transaction. If the outer transaction fails after mapUnifiedRepositories commits its own transaction, repository records persist while parent operations roll back, causing data inconsistency. The fix requires checking for an existing transaction first, similar to the pattern used in githubNangoConnect and gitConnectOrUpdate with SequelizeRepository.getTransaction().
Additional Locations (2)
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
| const [gitRepoIdMap, sourceIntegration] = await Promise.all([ | ||
| // TODO: after migration, generate UUIDs instead of fetching from git.repositories | ||
| getGitRepositoryIdsByUrl(qx, urls), | ||
| isGitHubPlatform ? IntegrationRepository.findById(sourceIntegrationId, txOptions) : null, |
There was a problem hiding this comment.
GitLab forkedFrom data not synced to public.repositories
In buildRepositoryPayloads, the sourceIntegration is only fetched for GitHub platforms (line 3118 uses a conditional that returns null for non-GitHub). The forkedFromMap at lines 3157-3166 is populated exclusively from sourceIntegration?.settings?.orgs, which is GitHub-specific. For GitLab repositories, forkedFromMap remains empty, causing all GitLab repos to have forkedFrom: null in public.repositories. However, GitLab's forkedFrom data IS available and passed to gitConnectOrUpdate (lines 2873-2875, 2884-2886), so the information exists but isn't propagated to the unified repositories table.
Additional Locations (1)
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
| }), | ||
| }, | ||
| segmentOptions, | ||
| PlatformType.GITHUB, |
There was a problem hiding this comment.
Missing repository sync in GitHub repos mapping function
The mapGithubRepos function now passes PlatformType.GITHUB to gitConnectOrUpdate which skips the internal mapUnifiedRepositories call (per the JSDoc comment "caller handles it"). However, the function never calls mapUnifiedRepositories itself. This means when GitHub repos are mapped via this function, git.repositories is updated but public.repositories is not synced, causing data inconsistency between the two tables.
Additional Locations (1)
| remotes: repositories.map((url) => ({ url, forkedFrom: null })), | ||
| }, | ||
| txOptions, | ||
| platform, |
There was a problem hiding this comment.
Missing repository sync in integration update method
The update method now passes platform to gitConnectOrUpdate which skips the internal mapUnifiedRepositories call. However, the update method doesn't add its own mapUnifiedRepositories call afterward. This creates a regression where updating code platform integrations syncs to git.repositories but no longer syncs to public.repositories, breaking the unified repository table consistency.
This pull request introduces significant improvements to the repository integration and synchronization logic within the
IntegrationServiceinbackend/src/services/integrationService.ts. The main focus is on ensuring that repository mappings between source integrations and the unifiedpublic.repositoriestable are consistent, robust, and transactional across different platforms (GitHub, GitLab, Gerrit, and direct Git). The changes add validation, transactional safety, and better handling of edge cases when mapping, restoring, or soft-deleting repositories.Key changes include:
Repository Mapping and Synchronization Enhancements
mapUnifiedRepositories, which handles inserting, restoring, and soft-deleting repositories in the unifiedpublic.repositoriestable in a transactional manner. This includes robust validation to prevent remapping repositories across different integrations and ensures consistency across platforms.validateRepoIntegrationMappingandbuildRepositoryPayloadsto support the main mapping logic, including building payloads with correct associations (segment, integration, insights project, forkedFrom, etc.).Platform-Aware Repository Synchronization
gitConnectOrUpdatemethod and all relevant integration flows (GitHub, GitLab, Gerrit, direct Git) to callmapUnifiedRepositorieswith the correct platform and integration context, ensuring that repository synchronization is handled appropriately for each platform. [1] [2] [3]sourcePlatformparameter togitConnectOrUpdateto control when unified repository mapping should be triggered, allowing for more flexible integration flows.Integration with Data Access Layer
Improved Transaction Management
Platform-Specific Handling in Integration Flows
These changes make the repository integration process more reliable, maintainable, and scalable across different source platforms.
Note
Strengthens repository synchronization across code platforms with a unified, transactional mapping flow.
mapUnifiedRepositoriesto insert/restore/soft-delete inpublic.repositorieswith validation against cross-integration remaps; builds payloads (segment, insights project, git integration, fork lineage), using existing Git IDs or generated UUIDsgitConnectOrUpdatewith optionalsourcePlatform; now syncs to unified repos for directGITand lets callers (GitHub/GitLab/Gerrit) handle mapping; validates GIT deletion safetysourcePlatform, merge existing remotes, and invoke unified repo mapping; ensures operations run inside transactionsIRepository,ICreateRepository,insertRepositories,restoreRepositories,softDeleteRepositories,getRepositoriesByUrl,getRepositoriesBySourceIntegrationId,getGitRepositoryIdsByUrl)enableGitto true and hide the toggleWritten by Cursor Bugbot for commit 9df6d82. This will update automatically on new commits. Configure here.