Skip to content

Conversation

@tomgr
Copy link
Contributor

@tomgr tomgr commented Aug 22, 2025

Whilst writing the integration test for buildbarn in bb-deployments, I noticed that bb_storage took a long time to shutdown on Windows. When looking at the system monitor, bb_storage was saturating the disk writing out data on shutdown. This was unexpected, since the test only wrote a small amount of data to the cache, so only a small amount of the block device should have been used (hundreds of MBs out of the GBs in the block device).

It seems that when using SetEndOfFile, Windows will write out the entire file's contents when the process gets shutdown. This is a problem if buildbarn is started with a large block device but only uses a small part of it, like in the deployment tests.

Following the suggestions at https://devblogs.microsoft.com/oldnewthing/20110922-00/?p=9573 we now create a sparse file and instantly zero the entire file's contents (if it's new or is resized). This completely fixes the performance issues.

@aspect-workflows
Copy link

aspect-workflows bot commented Aug 22, 2025

Test

31 test targets passed

Targets
//pkg/auth:auth_test [k8-fastbuild]                                           902ms
//pkg/blobstore/buffer:buffer_test [k8-fastbuild]                             48ms
//pkg/blobstore/completenesschecking:completenesschecking_test [k8-fastbuild] 65ms
//pkg/blobstore/grpcclients:grpcclients_test [k8-fastbuild]                   68ms
//pkg/blobstore/grpcservers:grpcservers_test [k8-fastbuild]                   124ms
//pkg/blobstore/local:local_test [k8-fastbuild]                               76ms
//pkg/blobstore/mirrored:mirrored_test [k8-fastbuild]                         42ms
//pkg/blobstore/readcaching:readcaching_test [k8-fastbuild]                   44ms
//pkg/blobstore/readfallback:readfallback_test [k8-fastbuild]                 41ms
//pkg/blobstore/replication:replication_test [k8-fastbuild]                   42ms
//pkg/blobstore/sharding/integration:integration [k8-fastbuild]               98ms
//pkg/blobstore/sharding/integration:integration_test [k8-fastbuild]          35ms
//pkg/blobstore/sharding/legacy:legacy_test [k8-fastbuild]                    477ms
//pkg/blobstore/sharding:sharding_test [k8-fastbuild]                         818ms
//pkg/blobstore:blobstore_test [k8-fastbuild]                                 150ms
//pkg/blockdevice:blockdevice_test [k8-fastbuild]                             199ms
//pkg/builder:builder_test [k8-fastbuild]                                     129ms
//pkg/capabilities:capabilities_test [k8-fastbuild]                           66ms
//pkg/digest:digest_test [k8-fastbuild]                                       107ms
//pkg/eviction:eviction_test [k8-fastbuild]                                   79ms
//pkg/filesystem/path:path_test [k8-fastbuild]                                68ms
//pkg/filesystem:filesystem_test [k8-fastbuild]                               100ms
//pkg/grpc:grpc_test [k8-fastbuild]                                           261ms
//pkg/http:http_test [k8-fastbuild]                                           144ms
//pkg/jmespath:jmespath_test [k8-fastbuild]                                   421ms
//pkg/jwt:jwt_test [k8-fastbuild]                                             219ms
//pkg/otel:otel_test [k8-fastbuild]                                           136ms
//pkg/prometheus:prometheus_test [k8-fastbuild]                               42ms
//pkg/random:random_test [k8-fastbuild]                                       114ms
//pkg/util:util_test [k8-fastbuild]                                           73ms
//pkg/x509:x509_test [k8-fastbuild]                                           44ms

Whilst writing the integration test for buildbarn in bb-deployments,
I noticed that bb_storage took a long time to shutdown on Windows. When
looking at the system monitor, bb_storage was saturating the disk
writing out data on shutdown. This was unexpected, since the test only
wrote a small amount of data to the cache, so only a small amount of
the block device should have been used (hundreds of MBs out of the GBs
in the block device).

It seems that when using SetEndOfFile, Windows will write out the entire
file's contents when the process gets shutdown. This is a problem if
buildbarn is started with a large block device but only uses a small
part of it, like in the deployment tests.

Following the suggestions at
https://devblogs.microsoft.com/oldnewthing/20110922-00/?p=9573
we now create a sparse file and instantly zero the entire file's
contents (if it's new or is resized). This completely fixes the
performance issues.
@tomgr tomgr force-pushed the windows-fix-block-device-performance branch from 9a4633d to c9ad39f Compare August 22, 2025 11:56
@tomgr tomgr requested a review from EdSchouten August 22, 2025 12:54
@EdSchouten EdSchouten merged commit 8af6436 into buildbarn:master Aug 26, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants