Writing to a file robustly is tricky:
- we need to check errors from
Write()
- we need to
Sync()
beforeClose()
, and check the error - we need to check error from
Close()
- if any of the above returned an error, the file is corrupted so we should remove it
Go package github.com/kjk/atomicfile (godoc) makes it easy to get this logic right
To open a file for writing use
atomicfile.New()
instead of os.Create()
or os.OpenFile()
.
It returns an
atomicfile.File
struct which implements io.Reader
, io.Closer
and io.ReadCloser
interfaces so it can be used by most code that accepted os.File
.
An example (full code):
func writeToFileAtomically(filePath string, data []byte) error {
f, err := atomicfile.New(filePath)
if err != nil {
return err
}
// ensure that if there's a panic or early exit due to
// error, the file will not be created
// RemoveIfNotClosed() after Close() is a no-op
defer f.RemoveIfNotClosed()
_, err = f.Write(data)
if err != nil {
return err
}
return f.Close()
}
For proper cleanup, we do
defer f.RemoveIfNotClose()
.
It'll ensure that temporary file is removed if we never call
Close()
(e.g. due to a panic or early exit due to error).
Why it's tricky
Here's file writing code that has a subtle bug:
func writeToFileAtomically(filePath string, data []byte) error {
f, err := os.Create(filePath)
if err != nil {
return err
}
defer f.Close()
_, err = f.Write(data)
if err != nil {
return err
}
return nil
}
There are problems with this code:
- if
Write()
fails, we create a corrupted (partially written) file. We shouldos.Remove()
it in case of errors Close()
can return an error. Writes to files are buffered soClose()
might need to write buffered data which can fail. Forgetting to handle an error fromClose()
is a common mistake when writing to files- even if
Close()
returns no error, the filesystem cache might not be written to disk, so we need to callSync()
beforeClose()
- it anything fails, we should delete the partially written file
- ideally we shouldn't expose partially written file until it's fully written
How atomicfile works
To avoid exposing partially written file,
atomicfile
writes to a temporary file.
Only after we finish writing, call
Close()
and there were no errors, we rename temporary file to final name.
The limitation of this approach is that it only works for new files. It doesn't support appending.
Calling
Close()
multiple times is ok. You should check the error returned by Close()
but you can also use defer f.Close()
to ensure it's closed on error conditions and panics.
References
Those issues are surprisingly tricky. Some people investigated them in depth:
More Go resources
- Essential Go is a free, comprehensive book on Go