What is filerotate?
filerotate is a Go library for rotating files e.g. daily, hourly.
References:
- api docs: https://pkg.go.dev/github.com/kjk/common/filerotate
- runnable example: https://replit.com/@kjk1/filerotate-example#main.go
- code: https://github.com/kjk/common/tree/main/filerotate
Imagine you're writing a backend for a web app in Go.
Logging is important for observability and debugging.
Web backends are long-running and you don't want your log files to grow infinitely.
Log rotation solves the issue: when the log file grows beyond a certain size or at certain time, a new log file is created.
To implement log rotation you can use external programs like
logrotate but I prefer simpler solution and thanks to io.Writer interface, Go makes implementing log rotation simple.
filerotate is a Go library that implement log rotation.
By default logs rotate daily, at midnight UTC time.
How to use filerotate
Runnable code: https://replit.com/@kjk1/filerotate-example#main.go
Open log file
Logging happens everywhere in the code so typically we would have a global variable for the log file and open the log at program start
var (
logFile *filerotate.File
)
func openLogFile(pathFormat string, onClose func(string, bool)) error {
w, err := dailyrotate.NewFile(pathFormat, onLogClose)
if err != nil {
return err
}
logFile = w
return nil
}
func main() {
logDir := "logs"
// we have to ensure the directory we want to write to
// already exists
err := os.MkdirAll(logDir, 0755)
if err != nil {
log.Fatalf("os.MkdirAll()(")
}
pathFormat := filepath.Join(logDir, "2006-01-02.txt")
err = openLogFile(pathFormat, onLogClose)
if err != nil {
log.Fatalf("openLogFile failed with '%s'\n", err)
}
// ... the rest of your program
}Just like for regular
os.Create() we have to ensure that the directory for log files exists with os.MkdirAll(dir, 0755).
Rotating files implies that the name of the file changes.
I like the convention of using date in the name of the file so
dailyrotate uses time.Format formatting layout for file paths.
I use
2006-01-02.txt for the file format (which is YYYY-MM-DD + .txt).
It's easy to locate a log for a given day and the names sort by day.
Log file is opened in append mode.
Write to log file
dailyrotate.File implements io.Writer so to write we uses the standard f.Write(d []byte) (int, error).
func writeToLog(msg string) error {
_, err := logFile.Write([]byte(msg))
return err
}Close log file
dailyrotate.File implements io.Close so to close we use Close() error.
It's safe to call
Close multiple times.
func closeLogFile() error {
return logFile.Close()
}Do something after log rotation
Imagine that when a log rotates you want to immediately backup the file to online storage like S3.
dailyrotate.NewFile takes an optional function that will be called when the log file is closed. A skeleton of a callback:
func onLogClose(path string, didRotate bool) {
fmt.Printf("we just closed a file '%s', didRotate: %v\n", path, didRotate)
if !didRotate {
return
}
// process just closed file e.g. upload to S3 for backup
go func() {
// if processing takes a long time, do it in background
}()
}The file can be closed either because you called
Close explicitly (e.g. because the program is exiting) or implicitly due to rotation.
You can distinguish the 2 cases with
didRotate argument.
Rotate at a different time
By default logs rotate at midnight UTC time. We use UTC because "midnight" means different time in different timezones. In California, it's 5 PM and in Paris it's 2 AM.
You can change UTC to a different time zone by setting
File.Location to a timezone Location (which you can load with time.LoadLocation):
loc, errr := time.LoadLocation("America/Los_Angeles")
if err != nil {
log.Fatalf("time.LoadLocation() failed with '%s'\n", err)
}
f, err := dailyrotate.NewFile("2006-01-02.txt", nil)
if err != nil {
log.Fatalf("dailyrotate.NewFile() failed with '%s'\n", err)
}
f.Location = loc