Integration framework API

[delner]

Integration framework API

Purpose & Benefits

This PR lays the foundation for auto-instrumentation of AI libraries (OpenAI, Anthropic, etc.) by introducing an integration framework that provides a consistent, scalable pattern for integrations.

This PR introduces the framework only (registry, base classes, generator). No actual library integrations yet—those come later.

Adding New Integrations

Contributors can now scaffold new integrations with a single command:

rake contrib:generate NAME=anthropic AUTO_REGISTER=true

This generates:

• The basic files needed for an integration, along with tests
• Optional auto-registration so that the integration is available in the Registry

See CONTRIBUTING.md for full documentation.

---

Design & Key Principles

Architecture (at a glance)

  Braintrust::Contrib
  ├── Registry (singleton)           # Contains all known integrations
  ├── Integration (base module)      # Schema all integrations implement
  ├── Patcher (base class)          # Base class integrations implement to apply instrumentation
  └── {integration_name}/
      ├── integration.rb            # An integration's schema
      └── patcher.rb               # An integration's patching code

Key Design Decisions

1. Lazy loading of integration code limits performance impact on startup, especially as the SDK grows in the size and complexity of its integration offerings.
2. Integration schema acts as a single source of metadata for the integration, suitable for programmatic usage (to be leveraged by auto instrumentation in the near future.)
3. Thread safety: It is possible multiple threads may load the integrations (again, through auto-instrumentation later), so there are safeguards in place to prevent double patching and other conflicts:
   • Registry uses double-checked locking for cache invalidation
   • Patcher uses mutex with double-check for idempotent patching
   • Tested with 100 concurrent threads
4. Module per integration
   • Each integration is a module (e.g., Braintrust::Contrib::OpenAI)
   • Self-contained, can be extracted to separate gems later (if needed)
5. Fail-Safe Design
   • All patching wrapped in rescue blocks
   • Errors logged, never raised
   • Apps continue running even if instrumentation fails
6. Integration generator uses ERB templates for consistent integration formats.
7. No breaking changes: new functionality only, existing tracing APIs (Braintrust::Trace::OpenAI.wrap) are unaffected.

[clutchski]

IMO we should keep github in the path for new stuff.

[clutchski]

I think this should include moving the instrumentation to this method so we can verify it actually works.

my question is how does it handle the muitple libraries that use require 'openai'

[clutchski]

Can you also show exactly how users will call this?

[realark]

I'd second Matt's feedback about moving the existing instrumentation over to call this feature done, but if you'd like to get this on to main first then follow-up that SGTM

[delner]

No problem: I can hold off onto merging this until we can prove one integration works well on top of it. (This was my intent anyway!)

---

I'd like to clarify on the integration paths/require behavior though. As these are two separate issues.

Integrations can "subscribe" to a require event and get a callback to patch based on the path required. In the case of require 'openai' both the official and Alexrudall versions listen to this (because both gems decided to use this path.) Obviously only one of these will be the correct one... each can do a check like we already do in trace.rb and skip if conditions are not met:

# After triggered by `require 'openai'`...

# Check which OpenAI gem's code is actually loaded by inspecting $LOADED_FEATURES
# (both gems can be in Gem.loaded_specs, but only one's code can be loaded)
openai_load_path = $LOADED_FEATURES.find { |f| f.end_with?("/openai.rb") }

return unless Gem.loaded_specs["ruby-openai"] && openai_load_path&.include?("ruby-openai")

---

In regards to directory names, embedding repository metadata is not idiomatic to Ruby. I would suggest two options:

Option 1: Name integrations by rubygems.org name (simplest)

└── braintrust/
    └── contrib/
        ├── openai/        # Offical OpenAI gem
        └── ruby-openai/   # Alexrudall gem

• There's a clean mapping of gem --> integration. Since virtually all users will be use rubygems.org as a source for their gems, this canonical registry gives sufficient delineation between conflicting packages. It's most idiomatic to refer to packages by this gem name.
• This keeps the structure relatively flat (requires are shorter, easier.)
• Because GitHub repo ownership can change (e.g. `alexrudall/ruby-openai`` could be moved under an organization... common as OSS libs become more mature, popular) this avoids having a mismatching directory path embedded in our library. Avoids a big headache, possible breaking change.

Option 2: Subdivide by source

└── braintrust/
    └── contrib/
        ├── rubygems/
        │   └── openai/        # Offical OpenAI gem
        └── github/
            └── alexrudall/
                └── ruby-openai/   # Alexrudall gem

• Would help avoid naming conflicts if alternative canonical sources are developed: e.g. RubyGems.org is abandoned for a new registry and openai is registered in each for different packages.
• Still suffers from Github volatility issues (see above)

My recommendation: Option 1 is the simplest, most idiomatic solution, as Option 2 only adds benefit in a low-probability hypothetical scenario.

---

Let me know your thoughts!

[delner]

Closing this while I reformulate PRs.