Skip to content

Conversation

@pyramation
Copy link
Contributor

@pyramation pyramation commented Dec 31, 2025

feat(cdn): add BUCKET_PROVIDER env var for explicit storage provider selection

Summary

Fixes an issue where isMinio was always true because minioEndpoint: 'http://localhost:9000' was hardcoded in pgpmDefaults. When deepmerge combined defaults with env vars, the default minioEndpoint persisted even when not explicitly set, causing S3 to be misconfigured in production.

This PR adds a BUCKET_PROVIDER env var (s3 | minio | gcs) as the single source of truth for provider selection. The provider is propagated through the entire upload chain:

pgpm/typespgpm/envs3-streamergraphile-upload-plugingraphile-settingsexplorer

Key changes:

  • New BucketProvider type and provider field in CDNOptions
  • Default provider is 'minio' for local dev (preserves existing behavior)
  • Setting BUCKET_PROVIDER=s3 in production correctly configures for S3
  • Breaking: Removed IS_MINIO env var and hostname heuristics entirely
  • Breaking: createS3Bucket now requires explicit { provider } options parameter

Updates since last revision

  • Refactored to ensure process.env is only read in env packages (pgpm/env)
  • createS3Bucket(client, bucket)createS3Bucket(client, bucket, { provider })
  • Removed all process.env.BUCKET_PROVIDER reads from s3-utils
  • Provider must now be passed explicitly from getEnvOptions() at call sites
  • Updated all tests to pass { provider: 'minio' } instead of setting env vars
  • Replaced console.log/console.error with Logger in create-bucket.ts

Review & Testing Checklist for Human

  • Breaking API change: createS3Bucket signature changed - verify no external callers exist outside this repo
  • Check fallback logic in s3.ts: Line has opts.provider === 'minio' || (opts.provider === undefined && Boolean(opts.minioEndpoint)) - verify this backwards-compat fallback is intentional or should be removed
  • Test S3 production config: Set BUCKET_PROVIDER=s3 and verify MinIO-specific config (endpoint, forcePathStyle) is NOT applied
  • Test MinIO local dev: Verify BUCKET_PROVIDER=minio (or unset, which defaults to minio) works with local MinIO

Recommended test plan:

  1. Run existing upload tests to verify no regression
  2. Manually test file upload with BUCKET_PROVIDER=s3 against real S3 bucket
  3. Manually test file upload with BUCKET_PROVIDER=minio against local MinIO
  4. Grep for IS_MINIO and process.env.BUCKET_PROVIDER to verify cleanup is complete

Notes

  • The BUCKET_PROVIDER env var is not validated at runtime - invalid values will be passed through as-is
  • The pnpm-lock.yaml changes are mostly formatting; the only functional change is adding @pgpmjs/types as a dependency to graphile-upload-plugin
  • Future work: Could add gcs (Google Cloud Storage) support using the same provider pattern

Link to Devin run: https://app.devin.ai/sessions/786904f315914f5c9d9c45e6707d7048
Requested by: Dan Lynch (@pyramation)

…selection

- Add BucketProvider type ('s3' | 'minio' | 'gcs') to CDNOptions interface
- Add provider field to pgpmDefaults (defaults to 'minio' for local dev)
- Add BUCKET_PROVIDER env var parsing in pgpm/env
- Update s3-streamer to use provider instead of minioEndpoint presence
- Update s3-utils/policies.ts to check BUCKET_PROVIDER first, then fall back to heuristics
- Pass provider through graphile-upload-plugin, graphile-settings, and explorer
- Update create-bucket.ts to use provider for S3 client configuration

This fixes the issue where isMinio was always true because minioEndpoint
was hardcoded in defaults. Now setting BUCKET_PROVIDER=s3 in production
will correctly configure for S3 regardless of default minioEndpoint.
@devin-ai-integration
Copy link
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

…icy helper

- Extract provider detection into dedicated shouldUseMinioPolicy() function
- Document priority order: BUCKET_PROVIDER > IS_MINIO (deprecated) > hostname heuristics
- Rename isMinio to useMinioPolicy for clarity
- Keep IS_MINIO for backwards compatibility but mark as deprecated
- Simplify shouldUseMinioPolicy() to only check BUCKET_PROVIDER env var
- Remove IS_MINIO from create-bucket.ts
- Update tests to use BUCKET_PROVIDER=minio instead of IS_MINIO=true
- Remove hostname heuristics - explicit provider only
- Add CreateS3BucketOptions interface with required provider field
- Update createS3Bucket signature to accept options parameter
- Remove process.env.BUCKET_PROVIDER read from s3-utils
- Update all call sites to pass provider explicitly
- Env vars are now only read in env packages (pgpm/env)
@pyramation pyramation merged commit 11fe190 into main Dec 31, 2025
35 checks passed
@pyramation pyramation deleted the devin/1767218073-bucket-provider branch December 31, 2025 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants