Skip to content

Conversation

@bb7133
Copy link
Member

@bb7133 bb7133 commented Sep 16, 2025

This PR extends our logging library to support buffered log writing by wrapping the underlying WriteSyncer with zapcore.BufferedWriteSyncer.

Motivation

Previously, logs were written synchronously on every call, which can cause unnecessary I/O overhead under high log volume. With buffering enabled, log entries are accumulated in memory and only flushed to disk:

  • Periodically (on a configurable flush interval), or
  • When the buffer reaches a specified size.

This reduces the frequency of disk writes and improves performance, while still guaranteeing that logs are eventually persisted.

Implementation

  • Update InitLogger() to introduce BufferedWriteSyncer.
  • Configurable parameters:
    • Buffer size: maximum number of bytes held in memory before a flush is triggered.
    • Flush interval: maximum duration logs may remain in memory before being flushed.
  • We did not add a Close() method for now, because:
    • zap.Logger itself does not expose Close(), only Sync().
    • Our current use cases rely on application-wide or test-level lifecycle management, where calling Sync() at shutdown is sufficient.
    • Introducing Close() would require additional lifecycle handling (e.g. stopping background flush goroutines), which we may revisit if needed in the future.

Testing

  • Added unit tests that verify:
    • Logs are not written to disk before flush is triggered.
    • Flush occurs after the configured time interval.
    • Flush occurs immediately once the buffer exceeds the configured size.
    • Log rotation still works correctly when combined with buffered flush.
  • Benchmark on my laptop:
goos: darwin
goarch: arm64
pkg: github.com/pingcap/log
cpu: Apple M2 Pro
BenchmarkLogger/unbuffered-10         	  132064	      9130 ns/op	     368 B/op	       5 allocs/op
BenchmarkLogger/buffered-4KB-10       	  182089	      6272 ns/op	     368 B/op	       5 allocs/op
BenchmarkLogger/buffered-64KB-10      	  308062	      3941 ns/op	     369 B/op	       5 allocs/op
BenchmarkLogger/buffered-256KB-10     	  291272	      3868 ns/op	     370 B/op	       5 allocs/op
BenchmarkLogger/buffered-1MB-10       	  296786	      3929 ns/op	     372 B/op	       5 allocs/op
PASS
ok  	github.com/pingcap/log	7.394s

@bb7133 bb7133 merged commit 19901e0 into pingcap:master Sep 17, 2025
2 checks passed
@bb7133 bb7133 deleted the bb7133/buffered branch September 17, 2025 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants