Imagine you’re writing a note taking application.

Each note needs a unique id.

Generating unique ids is easy if you can coordinate.

The simplest way is to get database to do it: use AUTOINCREMENT column and the database will generate unique id when you insert a new note row into a table.

What if you can’t coordinate?

For example, you want the app to also generate unique note id when offline, when it cannot talk to the database.

The requirement of non-coordinated generation of unique ids comes up often in distributed systems.

A simple solution is to generate a random id. If you give it 16 bytes of randomness, the chances of generating the same random number are non-existent.

It’s such a common problem that over 30 years ago we created a standard for this called UUID/GUID.

We can do better than GUID. A good random unique id:

• is unique; we can’t skip the basics
• can be sorted by its string representation
• is time-clustered i.e. ids generated at the same time are close to each other when sorted
• string representation can be used as part of URL without escaping
• the shorter, the better

There are few Go implementation of such id, following the same basic idea:

• use time as part of the id to achieve time-clustering
• fill rest of the id with random data
• encode as a string in a way that allows lexicographic sorting and is url-safe

Here are Go packages for generating unique id and how their ids look like in string format:

You can see how the values change over time by refreshing this test page a couple of times.

How to generate unique ids using different libraries:

import (
    "github.com/chilts/sid"
    "github.com/kjk/betterguid"
    "github.com/oklog/ulid"
    "github.com/rs/xid"
    "github.com/satori/go.uuid"
    "github.com/segmentio/ksuid"
    "github.com/sony/sonyflake"
)

// To run:
// go run main.go

func genXid() {
    id := xid.New()
    fmt.Printf("github.com/rs/xid:           %s\n", id.String())
}

func genKsuid() {
    id := ksuid.New()
    fmt.Printf("github.com/segmentio/ksuid:  %s\n", id.String())
}

func genBetterGUID() {
    id := betterguid.New()
    fmt.Printf("github.com/kjk/betterguid:   %s\n", id)
}

func genUlid() {
    t := time.Now().UTC()
    entropy := rand.New(rand.NewSource(t.UnixNano()))
    id := ulid.MustNew(ulid.Timestamp(t), entropy)
    fmt.Printf("github.com/oklog/ulid:       %s\n", id.String())
}

func genSonyflake() {
    flake := sonyflake.NewSonyflake(sonyflake.Settings{})
    id, err := flake.NextID()
    if err != nil {
        log.Fatalf("flake.NextID() failed with %s\n", err)
    }
    // Note: this is base16, could shorten by encoding as base62 string
    fmt.Printf("github.com/sony/sonyflake:   %x\n", id)
}

func genSid() {
    id := sid.Id()
    fmt.Printf("github.com/chilts/sid:       %s\n", id)
}

func genUUIDv4() {
    id := uuid.NewV4()
    fmt.Printf("github.com/satori/go.uuid:   %s\n", id)
}

func main() {
    genXid()
    genKsuid()
    genBetterGUID()
    genUlid()
    genSonyflake()
    genSid()
    genUUIDv4()
}

Full example: generate-unique-id/main.go

Which one to use?

All of them are good.

I would pick either rs/xid or segmentio/ksuid.

oklog/ulid allows custom entropy (randomness) source but pays for that with complex API.

sony/sonyflake is the smallest but also the least random. It’s based on Twitter’s design for generating IDs for tweets.

For simplicity the example code serializes sony/snoflake in base16. It would be even shorter in base62 encoding used by other libraries, but other libraries provide that out-of-the-box and for sony/snoflake I would have to implement it myself.

The last one is UUID v4 from RFC 4112, for comparison.

To learn more:

• https://segment.com/blog/a-brief-history-of-the-uuid/ : the history of ksuid
• https://firebase.googleblog.com/2015/02/the-2120-ways-to-ensure-unique_68.html : inspiration for betterguid design
• http://antoniomo.com/blog/2017/05/21/unique-ids-in-golang-part-1/
• http://antoniomo.com/blog/2017/05/28/unique-ids-in-golang-part-2/
• http://antoniomo.com/blog/2017/06/03/unique-ids-in-golang-part-3/
• https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake.html : details of Twitter’s snowflake on which sonyflake is based

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/generate-unique-id