What is filerotate?
filerotate is a Go library for rotating files e.g. daily, hourly.
References:
Imagine you’re writing a backend for a web app in Go.
Logging is important for observability and debugging.
Web backends are long-running and you don’t want your log files to grow infinitely.
Log rotation solves the issue: when the log file grows beyond a certain size or at certain time, a new log file is created.
To implement log rotation you can use external programs like logrotate
but I prefer simpler solution and thanks to io.Writer
interface, Go makes implementing log rotation simple.
filerotate is a Go library that implements log rotation.
It provides easy to use daily (at UTC midnight) and hourly rotation as well as flexibility to choose your own schedule.
How to use filerotate
Open log file
Logging happens everywhere in the code so typically we would have a global variable for the log file and open the log at program start
var (
logFile *filerotate.File
)
func openLogFile(dir string, fileNameSuffix string, onClose func(string, bool)) error {
w, err := fileraote.NewDaily(dir, fileNameSuffix, onLogClose)
if err != nil {
return err
}
logFile = w
return nil
}
func main() {
logDir := "logs"
// we have to ensure the directory we want to write to
// already exists
err := os.MkdirAll(logDir, 0755)
if err != nil {
log.Fatalf("os.MkdirAll()(")
}
fileNameSuffix := ".txt" // files will be "2024-06-10.txt" etc.
err = openLogFile(logDir, fileNameSuffix, onLogClose)
if err != nil {
log.Fatalf("openLogFile failed with '%s'\n", err)
}
// ... the rest of your program
}
Just like for regular os.Create()
we have to ensure that the directory for log files exists with os.MkdirAll(dir, 0755)
.
Rotating files implies that the name of the file changes.
Daily rotation strikes the right balance between frequency of rotation and the sizes of files.
Write to log file
filerotate.File
implements io.Writer
so to write we uses the standard f.Write(d []byte) (int, error)
.
func writeToLog(msg string) error {
_, err := logFile.Write([]byte(msg))
return err
}
Close log file
filerotate.File
implements io.Closer
so to close call Close() error
.
It’s safe to call Close
multiple times.
func closeLogFile() error {
return logFile.Close()
}
Do something after log rotation
Imagine that when a log rotates you want to backup the file to S3.
filerotate.NewDaily
takes an optional function that will be called when the log file is closed. A skeleton of a callback:
func onLogClose(path string, didRotate bool) {
fmt.Printf("we just closed a file '%s', didRotate: %v\n", path, didRotate)
if !didRotate {
return
}
// process just closed file e.g. upload to S3 for backup
go func() {
// if processing takes a long time, do it in background
}()
}
The file can be closed either because you called Close
explicitly (e.g. because the program is exiting) or implicitly due to rotation.
You can distinguish the 2 cases with didRotate
argument.
Rotate at a different time
filerotate.NewDaily()
does daily rotation.
filerotate.NewHourly()
does hourly rotation.
You can create your own schedule by providing PathIfShouldRotate
member of filerotate.Config
and use filerotate.New(onfig *Config)
.
PathIfShouldRotate
is func(creationTime time.Time, now time.Time) string
.
It receives the creation time of the current log file, current time and returns a path of log file if rotation should happen or ""
if no rotation should happen.
This is called on every write so should be efficient.
The logic for daily rotation checks if year/month/day in both times is the same. If yes, returns “” (no rotation). If not, returns a path of new daily file which causes rotation (closing of the old file and creation of new file).