I wanted to save records with a flexible set of fields to a file. The records needed to be easy to inspect with tools like grep and tail, and the implementation needed to be simple.
I designed
siser, short for Simple Serialization. This article describes the current
github.com/kjk/common/siser API; the original version used different reader methods and record framing.
For ordinary structured logging, Go 1.21’s
log/slog is a convenient starting point. It has text and JSON handlers:
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Info("request completed", "url", "/", "status", 200)
This fragment imports log/slog and os. JSON records work with many existing analysis tools. CSV is compact, but fields depend on an agreed column layout; Go’s CSV reader can accept variable field counts by setting FieldsPerRecord to a negative value. Protocol Buffers provides a schema and compact binary encoding when human readability is less important.
Go 1.27 also added
encoding/json/v2 and encoding/json/jsontext, and moved
encoding/json onto the new implementation while preserving its API behavior. The v2 API has different defaults, so review compatibility before migrating. Performance comparisons against older JSON implementations should be rerun.
Siser is useful when I control both ends of the format and want readable key/value data with length-prefixed values that can contain arbitrary bytes.
Write and read records
Add the dependency to your module:
go get github.com/kjk/common/siser
Here is a complete round-trip example:
package main
import (
"bufio"
"bytes"
"fmt"
"log"
"github.com/kjk/common/siser"
)
func main() {
var data bytes.Buffer
writer := siser.NewWriter(&data)
record := siser.Record{Name: "http"}
if err := record.Write("url", "/", "ipaddr", "127.0.0.1", "code", 200); err != nil {
log.Fatal(err)
}
if _, err := writer.WriteRecord(&record); err != nil {
log.Fatal(err)
}
reader := siser.NewReader(bufio.NewReader(&data))
for reader.ReadNextRecord() {
code, ok := reader.Record.Get("code")
if ok {
fmt.Println(code)
}
}
if err := reader.Err(); err != nil {
log.Fatal(err)
}
}
For a log file, pass the open file to NewWriter or wrap it with bufio.NewReader for NewReader. Check write and close errors, and flush any buffering writer before closing its file.
Record.Write accepts alternating keys and values, including integers. Values are stored as text, so the reader returns strings. Use fixed application-defined field names, and parse numeric or time fields explicitly when analyzing them.
A short value uses key: value followed by a newline. Long, empty, or non-line-safe values use key:+N, a newline, and exactly N bytes of value data, with a trailing newline added for readability when needed.
Record.Marshal returns the key/value payload.
Writer.WriteRecord wraps it in a header containing the payload’s byte length, timestamp, and optional record name. A standalone
--- separator from the original format is not the current framing. See
record.go and
writer.go for the encoding details.
Reuse and ownership
Writer.WriteRecord resets the record after writing, including when the write returns an error; preserve the data separately if you need to retry. Record.Marshal returns bytes backed by the record’s buffer. Copy them before resetting or modifying the record if they need to outlive it.
The reader reuses its record on the next read. Process each record immediately or copy the values you need to retain. Do not share a mutable record between goroutines without synchronization. The
reader source documents these lifetimes.
The original implementation favored slices over maps partly to reuse storage. Maps can also be reused: delete their entries, or use the clear built-in added in Go 1.21. Slices and maps have different lookup costs and memory behavior; reallocation is not an unavoidable cost of using a map.
In an earlier benchmark, writing was roughly comparable to json.Marshal, while reading was about eight times faster:
$ go test -bench=.
BenchmarkSiserMarshalWriteMany-12 1329409 871.1 ns/op
BenchmarkSiserMarshalWriteSingle-12 1000000 1088 ns/op
BenchmarkJSONMarshal-12 1340370 767.9 ns/op
BenchmarkSiserUnmarshal-12 4720020 251.5 ns/op
BenchmarkJSONUnmarshal-12 603591 2036 ns/op
These are the original measurements, not results from current Go. Compare representative records with the same framing and error handling on both sides, using repeated runs and benchstat. The JSON implementation has changed substantially since these numbers were collected.
Siser trades ecosystem support and automatic struct mapping for a small format I can control. That suits my own analytics logs; standard JSON is often easier when other systems need to consume the records.
