feat(grpcclients): add ZSTD compression support for gRPC CAS client

[tyler-french]

This PR adds a new compression control to the grpc client configuration that wraps the existing gRPC client with ZSTD compression support for ByteStream operations.

This is a pre-requisite to supporting compressed bb-clientd.

Example Configuration

Frontend:

{
  blobstore: {
    contentAddressableStorage: {
      sharding: {
        shards: {
          "0": {
            backend: { grpc: {
              client: { address: 'storage-0:8981' },
              enableZstdCompression: true,
            } },
            weight: 1,
          },
          "1": {
            backend: { grpc: {
              client: { address: 'storage-1:8981' },
              enableZstdCompression: true,
            } },
            weight: 1,
          },
        },
      },
    },
  },
}

Storage backend:

{
  supportedCompressors: ['ZSTD'],
}

[aspect-workflows]

Test

2 test targets passed

Targets

//pkg/blobstore/grpcclients:grpcclients_test [k8-fastbuild]                          245ms
//pkg/blobstore/sharding/integration:integration_test [k8-fastbuild]                 178ms

Total test execution time was 423ms. 28 tests (93.3%) were fully cached saving 5s.

[EdSchouten]

I'd say, don't make this a separate storage backend. Just make it so that the existing CAS client caches the supported compressors returned by the last call to GetCapabilities() and uses that to determine whether to send requests that use Zstandard.

[tyler-french]

I'd say, don't make this a separate storage backend. Just make it so that the existing CAS client caches the supported compressors returned by the last call to GetCapabilities() and uses that to determine whether to send requests that use Zstandard.

I think the client needs to be able to make the decision on whether to send the request compressed or not, similar to how Bazel does. We also probably want to define a compression threshold similar to Bazel, so clients can optimize based on their network properties.

Maybe it'd be better to have the compression configuration separated a little bit. I updated the PR.

[tyler-french]

Friendly bump, @EdSchouten if you have time to take another look

[EdSchouten]

I'm not a big fan of how this is named. Compression shouldn't be treated as a different kind of backend. It's merely a behavioral aspect of the gRPC CAS client. Right now the problem is that we don't have a dedicated message for that. For all storage backends (AC, CAS, ...) we only provide a buildbarn.configuration.grpc.ClientConfiguration. My recommendation would be to do one of the following things instead:

• Don't make any changes to the configuration schema, and simply respect what the server announces via GetCapabilities().
• Add a GrpcBlobAccessConfiguration and change the type of field grpc to that. This means that options pertaining to compression and such will be available for all backends, but they will be no-ops for !CAS. Though slightly confusing, it does have the advantage that setups that merely want to forward all CAS/AC/... traffic can continue to write something like this:

{
  local backend = ...,
  contentAddressableStorage: backend,
  actionCache: backend,
  initialSizeClassCache: backend,
  ...
}

• Rename CompressedClientConfiguration to GrpcContentAddressableStorageBlobAccessConfiguration and this compressed field to grpc_content_addressable_storage.

[tyler-french]

Added GrpcBlobAccessConfiguration with enable_zstd_compression field that's a no-op for non-CAS backends.

[aron-muon]

Hey @tyler-french - do you have time to pick this back up?

[aron-muon]

Does this close #158?

[Copilot]

Pull request overview

This PR adds ZSTD compression support to the gRPC CAS (Content Addressable Storage) client to optimize network transfer of large blobs. The implementation introduces a new GrpcBlobAccessConfiguration protobuf message that wraps the existing gRPC client configuration with an optional compression flag.

Key changes:

• New GrpcBlobAccessConfiguration protobuf message with enable_zstd_compression boolean flag
• Compression negotiation via GetCapabilities() to check server support for ZSTD
• Compression/decompression readers and writers for ByteStream operations with a hardcoded 100-byte threshold

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
pkg/proto/configuration/blobstore/blobstore.proto	Defines new GrpcBlobAccessConfiguration message with compression flag
pkg/proto/configuration/blobstore/blobstore.pb.go	Generated protobuf Go code for the new configuration message
pkg/blobstore/grpcclients/cas_blob_access.go	Implements ZSTD compression/decompression for ByteStream Read/Write operations
pkg/blobstore/grpcclients/cas_blob_access_test.go	Adds test coverage for compressed and uncompressed blob operations
pkg/blobstore/grpcclients/BUILD.bazel	Adds zstd library dependency
pkg/blobstore/configuration/cas_blob_access_creator.go	Configures CAS client with 100-byte compression threshold when enabled
pkg/blobstore/configuration/ac_blob_access_creator.go	Updates to extract client from new wrapper configuration
pkg/blobstore/configuration/icas_blob_access_creator.go	Updates to extract client from new wrapper configuration
pkg/blobstore/configuration/iscc_blob_access_creator.go	Updates to extract client from new wrapper configuration
pkg/blobstore/configuration/fsac_blob_access_creator.go	Updates to extract client from new wrapper configuration

Comments suppressed due to low confidence (1)

pkg/blobstore/grpcclients/cas_blob_access.go:402

• The SupportedCompressors field is stored internally but is not included in the returned ServerCapabilities. This means callers of GetCapabilities won't see the compression support information, which could be needed for debugging or client decision-making. Consider adding SupportedCompressors to the returned CacheCapabilities structure.

	return &remoteexecution.ServerCapabilities{
		CacheCapabilities: &remoteexecution.CacheCapabilities{
			DigestFunctions: digest.RemoveUnsupportedDigestFunctions(cacheCapabilities.DigestFunctions),
		},

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.