atomicfile - robustly writing to a file in Go

Home / Software written by me / atomicfile - robustly writing to a file in Go edit

Try Documentalist, my app that offers fast, offline access to 190+ programmer API docs.

Writing to a file robustly is tricky:

we need to check errors from Write()
we need to Sync() before Close(), and check the error
we need to check error from Close()
if any of the above returned an error, the file is corrupted so we should remove it

Go package github.com/kjk/atomicfile (godoc) makes it easy to get this logic right

To open a file for writing use atomicfile.New() instead of os.Create() or os.OpenFile().

It returns an atomicfile.File struct which implements io.Reader, io.Closer and io.ReadCloser interfaces so it can be used by most code that accepted os.File.

An example (full code):

func writeToFileAtomically(filePath string, data []byte) error {
	f, err := atomicfile.New(filePath)
	if err != nil {
		return err
	}

	// ensure that if there's a panic or early exit due to
	// error, the file will not be created
	// RemoveIfNotClosed() after Close() is a no-op
	defer f.RemoveIfNotClosed()

	_, err = f.Write(data)
	if err != nil {
		return err
	}
	return f.Close()
}

For proper cleanup, we do defer f.RemoveIfNotClose().

It'll ensure that temporary file is removed if we never call Close() (e.g. due to a panic or early exit due to error).

Why it's tricky

Here's file writing code that has a subtle bug:

func writeToFileAtomically(filePath string, data []byte) error {
	f, err := os.Create(filePath)
	if err != nil {
		return err
	}

	defer f.Close()

	_, err = f.Write(data)
	if err != nil {
		return err
	}

	return nil
}

There are problems with this code:

if Write() fails, we create a corrupted (partially written) file. We should os.Remove() it in case of errors
Close() can return an error. Writes to files are buffered so Close() might need to write buffered data which can fail. Forgetting to handle an error from Close() is a common mistake when writing to files
even if Close() returns no error, the filesystem cache might not be written to disk, so we need to call Sync() before Close()
it anything fails, we should delete the partially written file
ideally we shouldn't expose partially written file until it's fully written

How atomicfile works

To avoid exposing partially written file, atomicfile writes to a temporary file.

Only after we finish writing, call Close() and there were no errors, we rename temporary file to final name.

The limitation of this approach is that it only works for new files. It doesn't support appending.

Calling Close() multiple times is ok. You should check the error returned by Close() but you can also use defer f.Close() to ensure it's closed on error conditions and panics.

References

Those issues are surprisingly tricky. Some people investigated them in depth:

More Go resources

Essential Go is a free, comprehensive book on Go

Written on Aug 16 2021.

home

Found a mistake, have a comment? Let me know.