What is filerotate?
filerotate is a Go library for rotating files e.g. daily, hourly.
References:
Imagine you’re writing a backend for a web app in Go.
Logging is important for observability and debugging.
Web backends are long-running and you don’t want your log files to grow infinitely.
Log rotation solves the issue: when the log file grows beyond a certain size or at certain time, a new log file is created.
To implement log rotation you can use external programs like logrotate but I prefer simpler solution and thanks to io.Writer interface, Go makes implementing log rotation simple.
filerotate is a Go library that implement log rotation.
By default logs rotate daily, at midnight
UTC time.
How to use filerotate
Open log file
Logging happens everywhere in the code so typically we would have a global variable for the log file and open the log at program start
var (
logFile *filerotate.File
)
func openLogFile(pathFormat string, onClose func(string, bool)) error {
w, err := dailyrotate.NewFile(pathFormat, onLogClose)
if err != nil {
return err
}
logFile = w
return nil
}
func main() {
logDir := "logs"
// we have to ensure the directory we want to write to
// already exists
err := os.MkdirAll(logDir, 0755)
if err != nil {
log.Fatalf("os.MkdirAll()(")
}
pathFormat := filepath.Join(logDir, "2006-01-02.txt")
err = openLogFile(pathFormat, onLogClose)
if err != nil {
log.Fatalf("openLogFile failed with '%s'\n", err)
}
// ... the rest of your program
}
Just like for regular os.Create() we have to ensure that the directory for log files exists with os.MkdirAll(dir, 0755).
Rotating files implies that the name of the file changes.
I like the convention of using date in the name of the file so dailyrotate uses [time.Format](https://golang.org/pkg/time/#Time.Format) formatting layout for file paths.
I use 2006-01-02.txt for the file format (which is YYYY-MM-DD + .txt).
It’s easy to locate a log for a given day and the names sort by day.
Log file is opened in append mode.
Write to log file
dailyrotate.File implements io.Writer so to write we uses the standard f.Write(d []byte) (int, error).
func writeToLog(msg string) error {
_, err := logFile.Write([]byte(msg))
return err
}
Close log file
dailyrotate.File implements io.Close so to close we use Close() error.
It’s safe to call Close multiple times.
func closeLogFile() error {
return logFile.Close()
}
Do something after log rotation
Imagine that when a log rotates you want to immediately backup the file to online storage like S3.
dailyrotate.NewFile takes an optional function that will be called when the log file is closed. A skeleton of a callback:
func onLogClose(path string, didRotate bool) {
fmt.Printf("we just closed a file '%s', didRotate: %v\n", path, didRotate)
if !didRotate {
return
}
// process just closed file e.g. upload to S3 for backup
go func() {
// if processing takes a long time, do it in background
}()
}
The file can be closed either because you called Close explicitly (e.g. because the program is exiting) or implicitly due to rotation.
You can distinguish the 2 cases with didRotate argument.
Rotate at a different time
By default logs rotate at midnight UTC time. We use UTC because “midnight” means different time in different timezones. In California, it’s 5 PM and in Paris it’s 2 AM.
You can change UTC to a different time zone by setting File.Location to a timezone Location (which you can load with [time.LoadLocation](https://golang.org/pkg/time/#LoadLocation)):
loc, errr := time.LoadLocation("America/Los_Angeles")
if err != nil {
log.Fatalf("time.LoadLocation() failed with '%s'\n", err)
}
f, err := dailyrotate.NewFile("2006-01-02.txt", nil)
if err != nil {
log.Fatalf("dailyrotate.NewFile() failed with '%s'\n", err)
}
f.Location = loc
