What is filerotate?
filerotate is a Go library for rotating files e.g. daily, hourly.
References:
Imagine you’re writing a backend for a web app in Go.
Logging is important for observability and debugging.
Web backends are long-running and you don’t want your log files to grow infinitely.
Log rotation solves the issue: when the log file grows beyond a certain size or at certain time, a new log file is created.
To implement log rotation you can use external programs like logrotate but I prefer simpler solution and thanks to io.Writer interface, Go makes implementing log rotation simple.
filerotate is a Go library that implements log rotation.
It provides easy to use daily (at UTC midnight) and hourly rotation as well as flexibility to choose your own schedule.
How to use filerotate
Open log file
Logging happens everywhere in the code so typically we would have a global variable for the log file and open the log at program start
var (
logFile *filerotate.File
)
func openLogFile(dir string, fileNameSuffix string, onClose func(string, bool)) error {
w, err := fileraote.NewDaily(dir, fileNameSuffix, onLogClose)
if err != nil {
return err
}
logFile = w
return nil
}
func main() {
logDir := "logs"
// we have to ensure the directory we want to write to
// already exists
err := os.MkdirAll(logDir, 0755)
if err != nil {
log.Fatalf("os.MkdirAll()(")
}
fileNameSuffix := ".txt" // files will be "2024-06-10.txt" etc.
err = openLogFile(logDir, fileNameSuffix, onLogClose)
if err != nil {
log.Fatalf("openLogFile failed with '%s'\n", err)
}
// ... the rest of your program
}
Just like for regular os.Create() we have to ensure that the directory for log files exists with os.MkdirAll(dir, 0755).
Rotating files implies that the name of the file changes.
Daily rotation strikes the right balance between frequency of rotation and the sizes of files.
Write to log file
filerotate.File implements io.Writer so to write we uses the standard f.Write(d []byte) (int, error).
func writeToLog(msg string) error {
_, err := logFile.Write([]byte(msg))
return err
}
Close log file
filerotate.File implements io.Closer so to close call Close() error.
It’s safe to call Close multiple times.
func closeLogFile() error {
return logFile.Close()
}
Do something after log rotation
Imagine that when a log rotates you want to backup the file to S3.
filerotate.NewDaily takes an optional function that will be called when the log file is closed. A skeleton of a callback:
func onLogClose(path string, didRotate bool) {
fmt.Printf("we just closed a file '%s', didRotate: %v\n", path, didRotate)
if !didRotate {
return
}
// process just closed file e.g. upload to S3 for backup
go func() {
// if processing takes a long time, do it in background
}()
}
The file can be closed either because you called Close explicitly (e.g. because the program is exiting) or implicitly due to rotation.
You can distinguish the 2 cases with didRotate argument.
Rotate at a different time
filerotate.NewDaily() does daily rotation.
filerotate.NewHourly() does hourly rotation.
You can create your own schedule by providing PathIfShouldRotate member of filerotate.Config and use filerotate.New(onfig *Config).
PathIfShouldRotate is func(creationTime time.Time, now time.Time) string.
It receives the creation time of the current log file, current time and returns a path of log file if rotation should happen or "" if no rotation should happen.
This is called on every write so should be efficient.
The logic for daily rotation checks if year/month/day in both times is the same. If yes, returns “” (no rotation). If not, returns a path of new daily file which causes rotation (closing of the old file and creation of new file).
