Generating good unique ids in Go

Home / Go Cookbook / Generating good unique ids in Go edit

Try Documentalist, my app that offers fast, offline access to 190+ programmer API docs.

Imagine you’re writing a note taking application.

Each note needs a unique id.

Generating unique ids is easy if you can coordinate.

The simplest way is to get database to do it: use AUTOINCREMENT column and the database will generate unique id when you insert a new note row into a table.

What if you can’t coordinate?

For example, you want the app to also generate unique note id when offline, when it cannot talk to the database.

The requirement of non-coordinated generation of unique ids comes up often in distributed systems.

A simple solution is to generate a random id. If you give it 16 bytes of randomness, the chances of generating the same random number are non-existent.

It’s such a common problem that over 30 years ago we created a standard for this called UUID/GUID.

We can do better than GUID. A good random unique id:

is unique; we can’t skip the basics
can be sorted by its string representation
is time-clustered i.e. ids generated at the same time are close to each other when sorted
string representation can be used as part of URL without escaping
the shorter, the better

There are few Go implementation of such id, following the same basic idea:

use time as part of the id to achieve time-clustering
fill rest of the id with random data
encode as a string in a way that allows lexicographic sorting and is url-safe

Here are Go packages for generating unique id and how their ids look like in string format:

GUID libraries

Package	Id	Format
github.com/segmentio/ksuid	`0pPKHjWprnVxGH7dEsAoXX2YQvU`	4 bytes of time (seconds) + 16 random bytes
github.com/rs/xid	`b50vl5e54p1000fo3gh0`	4 bytes of time (seconds) + 3 byte machine id + 2 byte process id + 3 bytes random
github.com/kjk/betterguid	`-Kmdih_fs4ZZccpx2Hl1`	8 bytes of time (milliseconds) + 9 random bytes
github.com/sony/sonyflake	`20f8707d6000108`	~6 bytes of time (10 ms) + 1 byte sequence + 2 bytes machine id
github.com/oklog/ulid	`01BJMVNPBBZC3E36FJTGVF0C4S`	6 bytes of time (milliseconds) + 8 bytes random
github.com/chilts/sid	`1JADkqpWxPx-4qaWY47~FqI`	8 bytes of time (ns) + 8 random bytes
https://github.com/lithammer/shortuuid	`dwRQAc68PhHQh4BUnrNsoS`	UUIDv4 or v5, encoded in a more compact way
github.com/satori/go.uuid	`5b52d72c-82b3-4f8e-beb5-437a974842c`	UUIDv4 from RFC 4112 for comparison
https://github.com/google/uuid	`c01d7cf6-ec3f-47f0-9556-a5d6e9009a43`	UUIDv4

You can see how the values change over time by refreshing this test page a couple of times.

How to generate unique ids using different libraries:

import (
	"fmt"
	"log"
	"math/rand"
	"time"

	"github.com/chilts/sid"
	guuid "github.com/google/uuid"
	"github.com/kjk/betterguid"
	"github.com/lithammer/shortuuid"
	"github.com/oklog/ulid"
	"github.com/rs/xid"
	"github.com/satori/go.uuid"
	"github.com/segmentio/ksuid"
	"github.com/sony/sonyflake"
)

func genShortUUID() {
	id := shortuuid.New()
	fmt.Printf("github.com/lithammer/shortuuid: %s\n", id)
}

func genUUID() {
	id := guuid.New()
	fmt.Printf("github.com/google/uuid:         %s\n", id.String())
}

func genXid() {
	id := xid.New()
	fmt.Printf("github.com/rs/xid:              %s\n", id.String())
}

func genKsuid() {
	id := ksuid.New()
	fmt.Printf("github.com/segmentio/ksuid:     %s\n", id.String())
}

func genBetterGUID() {
	id := betterguid.New()
	fmt.Printf("github.com/kjk/betterguid:      %s\n", id)
}

func genUlid() {
	t := time.Now().UTC()
	entropy := rand.New(rand.NewSource(t.UnixNano()))
	id := ulid.MustNew(ulid.Timestamp(t), entropy)
	fmt.Printf("github.com/oklog/ulid:          %s\n", id.String())
}

func genSonyflake() {
	flake := sonyflake.NewSonyflake(sonyflake.Settings{})
	id, err := flake.NextID()
	if err != nil {
		log.Fatalf("flake.NextID() failed with %s\n", err)
	}
	// Note: this is base16, could shorten by encoding as base62 string
	fmt.Printf("github.com/sony/sonyflake:      %x\n", id)
}

func genSid() {
	id := sid.Id()
	fmt.Printf("github.com/chilts/sid:          %s\n", id)
}

func genUUIDv4() {
	id, err := uuid.NewV4()
	if err != nil {
		log.Fatalf("uuid.NewV4() failed with %s\n", err)
	}
	fmt.Printf("github.com/satori/go.uuid:      %s\n", id)
}

Full example: generate-unique-id/main.go

Which one to use?

All of them are good.

I would pick either rs/xid or segmentio/ksuid.

oklog/ulid allows custom entropy (randomness) source but pays for that with complex API.

sony/sonyflake is the smallest but also the least random. It’s based on Twitter’s design for generating IDs for tweets.

For simplicity the example code serializes sony/snoflake in base16. It would be even shorter in base62 encoding used by other libraries, but other libraries provide that out-of-the-box and for sony/snoflake I would have to implement it myself.

The last one is UUID v4 from RFC 4112, for comparison.

To learn more:

https://segment.com/blog/a-brief-history-of-the-uuid/ : the history of ksuid
https://firebase.googleblog.com/2015/02/the-2120-ways-to-ensure-unique_68.html : inspiration for betterguid design
http://antoniomo.com/blog/2017/05/21/unique-ids-in-golang-part-1/
http://antoniomo.com/blog/2017/05/28/unique-ids-in-golang-part-2/
http://antoniomo.com/blog/2017/06/03/unique-ids-in-golang-part-3/
https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake.html : details of Twitter’s snowflake on which sonyflake is based

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/generate-unique-id

Go Cookbook

Found a mistake, have a comment? Let me know.