Why rotate log files?

If you log to a file, it’s a good idea to rotate logs to limit their size.

After rotation you can backup them up to online storage or delete them.

Rotating daily has a good balance of how often to rotate vs. how large the log file becomes.

You can write logs to stdout and use external program, like logrotate, to do the rotation.

I prefer the simplicity of handling that in my own code and Go’s io.Writer interface makes it easy to implement a re-usable file rotation package.

I did just that in filerotate (documentation).

Here’s basic use (the code):

package main

import (
	"fmt"
	"log"

	"github.com/kjk/common/filerotate"
)

func main() {
	didClose := func(path string, didRotate bool) {
		fmt.Printf("closed file '%s', didRotate: %v\n", path, didRotate)
		if didRotate {
			// here you can implement e.g. compressing a log file and
			// uploading them to s3 for long-term storage
		}
	}
	f, err := filerotate.NewDaily(".", "log.txt", didClose)
	if err != nil {
		log.Fatalf("filerotate.NewDaily() failed with '%s'\n", err)
	}
	_, err = f.Write([]byte("hello"))
	if err != nil {
		log.Fatalf("f.Write() failed with '%s'\n", err)
	}
	err = f.Close()
	if err != nil {
		log.Fatalf("f.Close() failed with '%s'\n", err)
	}
}

filerotate.NewDaily() creates a io.Writer which will rotatate the file daily (using UTC time). You provide a directory where files will be stored and a file suffix.

The files will be named YYYY-MM-DD-${suffix}. For exmple: log.txt => 2024-06-14-log.txt.txt etc.

filerotate.File is io.Writer and safe to use from multiple goroutines.

In addition to Write(d []byte) it also implements a Write2(d []byte, flush bool) (string, int64, int, error). It has 2 improvements over Write:

allows to flush in a single call. Flushing after each write is slower but less likely to loose data or corrupt the file when program crashes
it returns file path and offset in the file at which the data was written. This is important for building a random-access index to records written to filerotate.File

Other real-world uses

Rotation is not limited to log files. I use it as part of poor-man’s web server analytics system.

I log info about web requests to filerotate.File using my siser simple serialization format.

When a file is rotated, I compress it, upload to backblaze for backup and delete local files older than 7 days to free up space.

I also calculate basic daily statistics and e-mail a summary to myself.

I know, I could just use Google Analytics. Advantage of my little system is that I can tailor it exactly to my needs.

For example, I don’t need 99% of functionality of Google Analytics. I just want a daily email with summary, to keep track of things with minimal effort.