Why rotate log files?
If you log to a file, it’s a good idea to rotate logs to limit their size.
After rotation you can back them up to online storage or delete them.
Rotating daily has a good balance of how often to rotate vs. how large the log file becomes.
You can write logs to stdout and let your service manager or container runtime manage them. If you write directly to files, an external rotation tool must coordinate reopening those files with the application.
I prefer the simplicity of handling that in my own code, and Go’s io.Writer interface makes it easy to implement a reusable file rotation package.
Add the dependency with go get github.com/kjk/common/filerotate. Here is a complete example:
package main
import (
"fmt"
"log"
"github.com/kjk/common/filerotate"
)
func main() {
didClose := func(path string, didRotate bool) {
fmt.Printf("closed file '%s', didRotate: %v\n", path, didRotate)
if didRotate {
// Queue this path for a worker that compresses and uploads
// the closed file; keep this callback short.
}
}
f, err := filerotate.NewDaily(".", "log.txt", didClose)
if err != nil {
log.Fatalf("filerotate.NewDaily() failed with '%s'\n", err)
}
_, err = f.Write([]byte("hello"))
if err != nil {
log.Fatalf("f.Write() failed with '%s'\n", err)
}
err = f.Close()
if err != nil {
log.Fatalf("f.Close() failed with '%s'\n", err)
}
}
filerotate.NewDaily() creates an io.Writer that rotates on the first write after the local calendar date changes. It does not run a midnight timer. You provide a directory where files will be stored and a file suffix.
The files will be named YYYY-MM-DD-${suffix}. For example, the suffix log.txt produces 2024-06-14-log.txt.
*filerotate.File implements io.Writer and serializes writes with a mutex. Use one owner per output file; that mutex does not coordinate separate processes or separate File instances.
The current
implementation also provides
Write2(d []byte, sync bool) (int64, int, error): it returns the write offset, byte count, and error, and can sync the file to disk. The path is available separately as
f.Path; reading it while another goroutine may rotate the file needs coordination. Older examples with a four-result
Write2 signature describe a different API.
The rotation callback runs while the file’s mutex is held. Keep it short, avoid calling methods on the same writer, and hand slow uploads to a separate worker. Close the writer after logging has stopped.
Use it with structured logging
Go 1.21 added
log/slog. Its handlers accept an
io.Writer, so the same rotating file can store JSON logs:
logger := slog.New(slog.NewJSONHandler(f, nil))
logger.Info("request completed", "method", "GET", "path", "/", "status", 200)
This fragment imports log/slog and uses the file f opened above, before it is closed. slog formats the records; the writer controls rotation. Daily rotation does not cap a day’s file size or remove old files, so add retention or size limits when needed.
Other real-world uses
Rotation is not limited to log files. I use it as part of a small web server analytics system.
I log info about web requests to
filerotate.File using my
siser simple serialization format.
When a file is rotated, I compress it, upload it to
Backblaze for backup and delete local files older than 7 days to free up space.
I also calculate basic daily statistics and e-mail a summary to myself.
I know, I could just use Google Analytics. The advantage of my little system is that I can tailor it exactly to my needs.
For example, I don’t need 99% of the functionality of Google Analytics. I just want a daily email with a summary, to keep track of things with minimal effort.
