Build: disable builds after 25 consecutive failed builds on default version (#12624)

Copilot · humitos · ericholscher · web-flow · commit 19f8bfea7e7a · 2025-12-09T10:36:17.000+01:00
- [x] Explore and understand the codebase - [x] Add a new notification message for consecutive failed builds (`MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES`) in `readthedocs/projects/notifications.py` - [x] Create a signal receiver to check for consecutive failed builds after build completion in `readthedocs/builds/signals_receivers.py` - [x] Add a helper method to count consecutive failed builds for the default version - [x] When 50+ consecutive failures occur: attach notification to Project and disable builds (`skip=True`) - [x] Add documentation explaining this behavior in `docs/user/builds.rst` - [x] Add tests for the new functionality in `readthedocs/builds/tests/test_signals_receivers.py` - [x] Run code review and fix any issues - [x] Run CodeQL security check - no vulnerabilities found - [x] Address review feedback: - [x] Move "Automatic disabling of builds" section before "Build resources" - [x] Simplify documentation text - [x] Move RTD_BUILDS_MAX_CONSECUTIVE_FAILURES setting to base.py - [x] Update docstring to reference the setting name - [x] Keep only the main test for disabling at threshold - [x] Convert signal handler to trigger a Celery task (builders don't have DB access) - [x] Add comment explaining why project_slug/version_slug check is needed - [x] Revert task logic to use groupby solution (the simplified version didn't work correctly)  <details> <summary>Original prompt</summary> ---- *This section details on the original issue you should resolve* <issue_title>Build: disable webhooks after N failed builds in a row</issue_title> <issue_description>Analyzing the data we have, I've found that we have a bunch of projects that have been triggering builds where _all of them_ have failed. During months. This consumes resources in our side, but also degrade the UX for valid projects since they have to wait for a build that _we already know_ it's gonna fail, before their project's build is taken. So, to reduce projects in this scenario, we are thinking about "auto-disable webhooks on projects that have N failed builds in a row" and communicate their maintainers. To re-enable the webhook, they would have to go to the project's admin page and re-activate/re-configure/re-add the webhook. This is the metabase question for future reference: https://ethicalads.metabaseapp.com/question/309-project-with-all-failed-builds-in-the-last-3-months. It shows the amount of builds and minutes consumed per project.</issue_description> <agent_instructions>We want to attach a notification to the `Project` when the active default version fails 50+ times to build. Besides, we want to disable the builds for this project and tell the user to contact support if it was a mistake. Also, adds documentation about this behavior.</agent_instructions> ## Comments on the Issue (you are @copilot in this section) <comments> <comment_new><author>@agjohnson</author><body> Also, it would be great to have a maintainers site notification specific to this that goes out, so we can re-use this pattern when soft-disabling a project. A hard disable (setting `Project.skip = True`) can be reserved for abusive/spam projects, but that "notification" pattern is quite broken for normal users. Or perhaps I'm describing two potential notifications -- a specific notification and a generic notification? - "Your project hasn't built successfully for 50 consecutive builds, we've disabled your project from automatically building" - "Your project has a problem, we've disabled your project from automatically building"</body></comment_new> <comment_new><author>@humitos</author><body> Yeah, I'm fine with better notifications. Maybe we want to make that part of the work on #9279 and #3399 We need to probably set two different thresholds here: 1. N consecutive builds on branch/tags (lower, maybe 25) and disable the webhook completely 2. N consecutive builds on PR builds (upper, maybe 50) and only disable PR building I'm trying to say that "builds failing on PRs are _more acceptable_ than failing on branch/tags and should be handled differently"</body></comment_new> <comment_new><author>@agjohnson</author><body> Yeah, great point on PR builds. Maybe we are strictly only concerned with failed active versions? I agree we can rethink notifications in a larger PR. If we add something here it will be using the new notification pattern anyways. Some of the old patterns can eventually be culled, and just adding a new mechanism for disabling projects here would be enough to start that process.</body></comment_new> <comment_new><author>@humitos</author><body> @agjohnson > Maybe we are strictly only concerned with failed active versions? It makes sense to me, yeah.</body></comment_new> <comment_new><author>@humitos</author><body> I'd like to think about prioritizing this feature if possible because this will help us a lot to know "how many active/valid non-spam projects do we have in our platform" in an easier way. This data will be useful when migrating/deprecating things as we are doing with "building without a config file" since we will know "how many of the active/valid projects have already migrated". Many non-spam projects will not migrate to the new configuration file just because they don't need it or have to. Mainly projects that are archived or where created just for testing/development/educational purposes, for example. I assume we have _a lot of them_. The work required that I see here is: - [ ] check for N failed builds on the default version and delete/disable the webhook integrated with that project - [ ] send an onsite and email notification to all the maintainers explaining the situation - [ ] write some documentation explaining this behavior (probably in https://docs.readthedocs.io/en/stable/guides/connecting-git-account.html)</body></comment_new> <comment_new><author>@agjohnson</author><body> Because taking automated actions on projects has been hard to get right -- spam banning, build retry -- maybe we should start fairly conservatively here. I think I'm comfortable saying that any project who has a *default* version that has 25 consecutive failed builds is probably not monitoring their project. We also don't have to take any automated action yet, we could start with just a notification or drip campaign. At least disabling the webho... </details> - Fixes #9690  --- 💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: humitos <244656+humitos@users.noreply.github.com> Co-authored-by: Manuel Kaufmann <humitos@gmail.com> Co-authored-by: ericholscher <25510+ericholscher@users.noreply.github.com>
diff --git a/docs/user/builds.rst b/docs/user/builds.rst
@@ -112,6 +112,25 @@ Read the Docs supports three different mechanisms to cancel a running build:
 
       Take a look at :ref:`build-customization:cancel build based on a condition` section for some examples.
 
+Automatic disabling of builds
+-----------------------------
+
+To reduce resource consumption and improve build queue times for all users,
+Read the Docs will automatically disable builds for projects that have too many consecutive failed builds on their default version.
+
+When a project has **25 consecutive failed builds** on its default version,
+we will disable builds for the project.
+
+This helps ensure that projects with persistent build issues don't consume resources that could be used by active projects.
+
+.. note::
+
+   This only applies to the default version of a project.
+   Builds on other versions (branches, tags, pull requests) are not counted towards this limit.
+
+If your project has been disabled due to consecutive build failures, you'll need to re-enable from your project settings.
+Make sure to fix the underlying issue to avoid being disabled again.
+
 Build resources
 ---------------
 
diff --git a/readthedocs/builds/signals_receivers.py b/readthedocs/builds/signals_receivers.py
@@ -4,13 +4,17 @@
 NOTE: Done in a separate file to avoid circular imports.
 """
 
+import structlog
 from django.db.models.signals import post_save
 from django.dispatch import receiver
 
 from readthedocs.builds.models import Build
 from readthedocs.projects.models import Project
 
 
+log = structlog.get_logger(__name__)
+
+
 @receiver(post_save, sender=Build)
 def update_latest_build_for_project(sender, instance, created, **kwargs):
     """When a build is created, update the latest build for the project."""
diff --git a/readthedocs/builds/tasks.py b/readthedocs/builds/tasks.py
@@ -667,3 +667,64 @@ def send_webhook(self, webhook):
                 webhook_id=webhook.id,
                 webhook_url=webhook.url,
             )
+
+
+@app.task(queue="web")
+def check_and_disable_project_for_consecutive_failed_builds(project_slug, version_slug):
+    """
+    Check if a project has too many consecutive failed builds and disable it.
+
+    When a project has more than RTD_BUILDS_MAX_CONSECUTIVE_FAILURES consecutive failed builds
+    on the default version, we attach a notification to the project and disable builds (skip=True).
+    This helps reduce resource consumption from projects that are not being monitored.
+    """
+    from readthedocs.builds.constants import BUILD_STATE_FINISHED
+    from readthedocs.projects.notifications import (
+        MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+    )
+
+    try:
+        project = Project.objects.get(slug=project_slug)
+    except Project.DoesNotExist:
+        return
+
+    # Only check for the default version
+    if version_slug != project.get_default_version():
+        return
+
+    # Skip if the project is already disabled
+    if project.skip or project.n_consecutive_failed_builds:
+        return
+
+    # Count consecutive failed builds on the default version
+    builds = list(
+        Build.objects.filter(
+            project=project,
+            version_slug=version_slug,
+            state=BUILD_STATE_FINISHED,
+        )
+        .order_by("-date")
+        .values_list("success", flat=True)[: settings.RTD_BUILDS_MAX_CONSECUTIVE_FAILURES]
+    )
+    if not any(builds) and len(builds) >= settings.RTD_BUILDS_MAX_CONSECUTIVE_FAILURES:
+        consecutive_failed_builds = builds.count(False)
+        log.info(
+            "Disabling project due to consecutive failed builds.",
+            project_slug=project.slug,
+            version_slug=version_slug,
+            consecutive_failed_builds=consecutive_failed_builds,
+        )
+
+        # Disable the project
+        project.n_consecutive_failed_builds = True
+        project.save()
+
+        # Attach notification to the project
+        Notification.objects.add(
+            message_id=MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+            attached_to=project,
+            dismissable=False,
+            format_values={
+                "consecutive_failed_builds": consecutive_failed_builds,
+            },
+        )
diff --git a/readthedocs/builds/tests/test_tasks.py b/readthedocs/builds/tests/test_tasks.py
@@ -2,6 +2,7 @@
 from textwrap import dedent
 from unittest import mock
 
+from django.conf import settings
 from django.contrib.auth.models import User
 from django.test import TestCase, override_settings
 from django.utils import timezone
@@ -19,10 +20,12 @@
 from readthedocs.builds.models import Build, BuildCommandResult, Version
 from readthedocs.builds.tasks import (
     archive_builds_task,
+    check_and_disable_project_for_consecutive_failed_builds,
     delete_closed_external_versions,
     post_build_overview,
 )
 from readthedocs.filetreediff.dataclasses import FileTreeDiff, FileTreeDiffFileStatus
+from readthedocs.notifications.models import Notification
 from readthedocs.oauth.constants import GITHUB_APP
 from readthedocs.oauth.models import (
     GitHubAccountType,
@@ -31,6 +34,9 @@
 )
 from readthedocs.oauth.services import GitHubAppService
 from readthedocs.projects.models import Project
+from readthedocs.projects.notifications import (
+    MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+)
 
 
 class TestTasks(TestCase):
@@ -124,6 +130,76 @@ def test_archive_builds(self, build_commands_storage):
         self.assertEqual(Build.objects.filter(cold_storage=True).count(), 5)
         self.assertEqual(BuildCommandResult.objects.count(), 50)
 
+    def _create_builds(self, project, version, count, success=False):
+        """Helper to create a series of builds."""
+        builds = []
+        for _ in range(count):
+            build = get(
+                Build,
+                project=project,
+                version=version,
+                success=success,
+                state=BUILD_STATE_FINISHED,
+            )
+            builds.append(build)
+        return builds
+
+    @override_settings(RTD_BUILDS_MAX_CONSECUTIVE_FAILURES=50)
+    def test_task_disables_project_at_max_consecutive_failed_builds(self):
+        """Test that the project is disabled at the failure threshold."""
+        project = get(Project, slug="test-project", n_consecutive_failed_builds=False)
+        version = project.versions.get(slug=LATEST)
+        version.active = True
+        version.save()
+
+        # Create failures at the threshold
+        self._create_builds(project, version, settings.RTD_BUILDS_MAX_CONSECUTIVE_FAILURES + 1, success=False)
+
+        # Call the Celery task directly
+        check_and_disable_project_for_consecutive_failed_builds(
+            project_slug=project.slug,
+            version_slug=version.slug,
+        )
+
+        project.refresh_from_db()
+        self.assertTrue(project.n_consecutive_failed_builds)
+
+        # Verify notification was added
+        notification = Notification.objects.filter(
+            message_id=MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES
+        ).first()
+        self.assertIsNotNone(notification)
+        self.assertEqual(notification.attached_to, project)
+
+    @override_settings(RTD_BUILDS_MAX_CONSECUTIVE_FAILURES=50)
+    def test_task_does_not_disable_project_with_successful_build(self):
+        """Test that the project is NOT disabled when there's at least one successful build."""
+        project = get(Project, slug="test-project-success", n_consecutive_failed_builds=False)
+        version = project.versions.get(slug=LATEST)
+        version.active = True
+        version.save()
+
+        # Create failures below the threshold with one successful build
+        self._create_builds(project, version, settings.RTD_BUILDS_MAX_CONSECUTIVE_FAILURES - 1, success=False)
+        self._create_builds(project, version, 1, success=True)  # One successful build
+        self._create_builds(project, version, 1, success=False)  # One more failure
+
+        # Call the Celery task directly
+        check_and_disable_project_for_consecutive_failed_builds(
+            project_slug=project.slug,
+            version_slug=version.slug,
+        )
+
+        project.refresh_from_db()
+        self.assertFalse(project.n_consecutive_failed_builds)
+
+        # Verify notification was NOT added
+        self.assertFalse(
+            Notification.objects.filter(
+                message_id=MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+            ).exists()
+        )
+
 
 @override_settings(
     PRODUCTION_DOMAIN="readthedocs.org",
diff --git a/readthedocs/notifications/signals.py b/readthedocs/notifications/signals.py
@@ -9,6 +9,9 @@
 from readthedocs.notifications.models import Notification
 from readthedocs.organizations.models import Organization
 from readthedocs.projects.models import Project
+from readthedocs.projects.notifications import (
+    MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+)
 from readthedocs.projects.notifications import MESSAGE_PROJECT_SKIP_BUILDS
 from readthedocs.subscriptions.notifications import MESSAGE_ORGANIZATION_DISABLED
 
@@ -32,6 +35,16 @@ def project_skip_builds(instance, *args, **kwargs):
         )
 
 
+@receiver(post_save, sender=Project)
+def project_n_consecutive_failed_builds(instance, *args, **kwargs):
+    """Check if the project has not N+ consecutive failed builds anymore and cancel the notification."""
+    if not instance.n_consecutive_failed_builds:
+        Notification.objects.cancel(
+            message_id=MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+            attached_to=instance,
+        )
+
+
 @receiver(post_save, sender=Organization)
 def organization_disabled(instance, *args, **kwargs):
     """Check if the organization is ``disabled`` and add/cancel the notification."""
diff --git a/readthedocs/projects/forms.py b/readthedocs/projects/forms.py
@@ -435,6 +435,7 @@ class Meta:
             "default_branch",
             "readthedocs_yaml_path",
             "search_indexing_enabled",
+            "n_consecutive_failed_builds",
             # Meta data
             "programming_language",
             "project_url",
@@ -478,6 +479,11 @@ def __init__(self, *args, **kwargs):
         if self.instance.search_indexing_enabled:
             self.fields.pop("search_indexing_enabled")
 
+        # Only show this field if building for this project is disabled due to N+ consecutive builds failing
+        # We allow disabling it from the form, but not enabling it.
+        if not self.instance.n_consecutive_failed_builds:
+            self.fields.pop("n_consecutive_failed_builds")
+
         # NOTE: we are deprecating this feature.
         # However, we will keep it available for projects that already using it.
         # Old projects not using it already or new projects won't be able to enable.
diff --git a/readthedocs/projects/migrations/0157_disable_builds_after_n_failed.py b/readthedocs/projects/migrations/0157_disable_builds_after_n_failed.py
@@ -0,0 +1,36 @@
+# Generated by Django 5.2.7 on 2025-12-02 09:19
+
+from django.db import migrations
+from django.db import models
+from django_safemigrate import Safe
+
+
+class Migration(migrations.Migration):
+    safe = Safe.before_deploy()
+
+    dependencies = [
+        ("projects", "0156_project_search_indexing_enabled"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="historicalproject",
+            name="n_consecutive_failed_builds",
+            field=models.BooleanField(
+                db_default=False,
+                default=False,
+                help_text="Builds on this project were automatically disabled due to many consecutive failures. Uncheck this field to re-enable building.",
+                verbose_name="Disable builds for this project",
+            ),
+        ),
+        migrations.AddField(
+            model_name="project",
+            name="n_consecutive_failed_builds",
+            field=models.BooleanField(
+                db_default=False,
+                default=False,
+                help_text="Builds on this project were automatically disabled due to many consecutive failures. Uncheck this field to re-enable building.",
+                verbose_name="Disable builds for this project",
+            ),
+        ),
+    ]
diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py
@@ -524,6 +524,14 @@ class Project(models.Model):
     featured = models.BooleanField(_("Featured"), default=False)
 
     skip = models.BooleanField(_("Skip (disable) building this project"), default=False)
+    n_consecutive_failed_builds = models.BooleanField(
+        _("Disable builds for this project"),
+        default=False,
+        db_default=False,
+        help_text=_(
+            "Builds on this project were automatically disabled due to many consecutive failures. Uncheck this field to re-enable building."
+        ),
+    )
 
     # null=True can be removed in a later migration
     # be careful if adding new queries on this, .filter(delisted=False) does not work
diff --git a/readthedocs/projects/notifications.py b/readthedocs/projects/notifications.py
@@ -20,6 +20,9 @@
 MESSAGE_PROJECT_SSH_KEY_WITH_WRITE_ACCESS = "project:ssh-key-with-write-access"
 MESSAGE_PROJECT_DEPRECATED_WEBHOOK = "project:webhooks:deprecated"
 MESSAGE_PROJECT_SEARCH_INDEXING_DISABLED = "project:search:indexing-disabled"
+MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES = (
+    "project:builds:disabled-due-to-consecutive-failures"
+)
 
 messages = [
     Message(
@@ -223,5 +226,18 @@
         ),
         type=INFO,
     ),
+    Message(
+        id=MESSAGE_PROJECT_BUILDS_DISABLED_DUE_TO_CONSECUTIVE_FAILURES,
+        header=_("Builds disabled due to consecutive failures"),
+        body=_(
+            textwrap.dedent(
+                """
+                Your project has been automatically disabled because the default version has failed to build {{consecutive_failed_builds}} times in a row.
+                Please fix the build issues and re-enable builds by unchecking "Disable builds for this project" option from <a href="{% url 'projects_edit' instance.slug %}">the project settings</a>.
+                """
+            ).strip(),
+        ),
+        type=WARNING,
+    ),
 ]
 registry.add(messages)
diff --git a/readthedocs/projects/querysets.py b/readthedocs/projects/querysets.py
@@ -95,6 +95,7 @@ def is_active(self, project):
 
         if (
             project.skip
+            or project.n_consecutive_failed_builds
             or any_owner_banned
             or (organization and organization.disabled)
             or spam_project
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
@@ -43,6 +43,7 @@
 from readthedocs.builds.models import APIVersion
 from readthedocs.builds.models import Build
 from readthedocs.builds.signals import build_complete
+from readthedocs.builds.tasks import check_and_disable_project_for_consecutive_failed_builds
 from readthedocs.builds.utils import memcache_lock
 from readthedocs.config.config import BuildConfigV2
 from readthedocs.config.exceptions import ConfigError
@@ -572,6 +573,13 @@ def on_failure(self, exc, task_id, args, kwargs, einfo):
                 status=status,
             )
 
+        # Trigger task to check number of failed builds and disable the project if needed (only for community)
+        if not settings.ALLOW_PRIVATE_REPOS:
+            check_and_disable_project_for_consecutive_failed_builds.delay(
+                project_slug=self.data.project.slug,
+                version_slug=self.data.version.slug,
+            )
+
         # Update build object
         self.data.build["success"] = False
 
diff --git a/readthedocs/settings/base.py b/readthedocs/settings/base.py
@@ -176,6 +176,7 @@ def RTD_RESTRICTED_DOMAINS(self):
     RTD_MAX_CONCURRENT_BUILDS = 4
     RTD_BUILDS_MAX_RETRIES = 25
     RTD_BUILDS_RETRY_DELAY = 5 * 60  # seconds
+    RTD_BUILDS_MAX_CONSECUTIVE_FAILURES = 25  # The project is disabled when hitting this limit on the default version
     RTD_BUILD_STATUS_API_NAME = "docs/readthedocs"
     RTD_ANALYTICS_DEFAULT_RETENTION_DAYS = 30 * 3
     RTD_AUDITLOGS_DEFAULT_RETENTION_DAYS = 30 * 3
