Krzysztof Kowalczyk blog

@levelsio and survivorship bias

2021-10-20T00:00:00Z

Pieter Levels is a prolific maker of software.

He's also very successful maker of software: he's close to making $1.5 million a year from his business, almost all of it profit.

Almost all of it is his profit since for most of the time he was a sole developer / marketer / copy writer, with a part-time sysadmin for ensuring the server stays alive. Recently he hired customer support and chat moderator.

This level of success attracts attention

It also attracts inevitable claims of survivorship bias. The gist of it is: if you do things he did, you won't have the same level of success.

I think it's a very defeatist attitude. Life is hard, there is no step-by-step guide to $1.5 million a year business.

You can, however, do things right or you can do them badly.

To create a successful business you need to do more things right than badly.

Let's call it Do Things Right bias.

Let's dissect Peter Level's journey to see what he did right and how that differed from doing things badly.

I've been following Pieter for a long time and he created a large body of tweets and blog posts, a book, a few youtube talks, most of it related to his journey from making $0 a year to $1.5 million a year.

People say you can learn from other people's mistakes. It's even better to learn from other people's successes.

What did Pieter do right?

Pieter ships products

Here's a list of his projects going back 11 years. Limiting the list to only software products:

2010: Uber clone
2012 dating site for college campuses, YouTube network
2013: YouTube analytics
2014: Slack community for digital nomads, GIF Book, Nomad Jobs, Nomad List 1.0, Go Fucking Do It, Play My Inbox
2015: Startup Retreats, Taylor Bot (a Telegram bot), Nomad List 2.0, Remote OK
2016: Virtual Reality, Places to Work, learning 3D modelling
2017: Nomad List 3.0, Nomad Gear, Mute, Hoodmaps
2018: Nomad List FIRE Calculator, MAKE Book, Maker Rank, No More Google
2019: Bali Sea Cable, Nomad List 5, How Much Is My Side Project Worth
2020: QR Menu Creator, IdeasAI, Remote OK Workers, Airline List, Nomad List Climate Finder
2021: Rebase, Inflation Chart, MAKE Book NFT,

You can see him building a product from first line of code to frontpage of Reddit

Doing it badly

Not shipping products.

Commenting on Hacker News how you couldn't possibly take away from your precious time commenting on Hacker News and write some code instead.

Pieter is persistent

He's clearly persistent.

His overnight success was 11 years in the making.

A lot of his projects failed but he didn't give up.

He kept making more.

He creates new things even when he doesn't have to because Nomad List and RemoteOk bring more money that he needs.

Doing it badly

Abandoning all work after first failure and using survivorship bias as a convenient excuse.

Pieter understands compund interest

If you have a profitable business, there are 2 ways to make even more money:

start a new business
sell more stuff to your existing customers

The second option tends to be easier and more reliable way to increate revenue and profits.

The hardest thing in business is getting attention of potential customers.

It's much easier to get attention of customers you already have.

While Pieter starts new projects, he's smart enough to mostly do things related to hist most successful business: Nomad List.

He keeps improving Nomad List.

He built jobs website for remote workers and does other related projects like Nomad List Climate Finder.

This is similar to the magic of compounded interest in investing.

Doing it badly

Not doubling down on things that work. Chasing distractions.

Pieter abandons projects that are not working

You need to be persistent, but in a smart way.

You'll know when you have a product market fit.

Nomad List started as a Google Spreadsheets that Pieter created for himself and accidently left unprotected.

Within days people were adding more information to it. That was a clear sign of interest so he turned it into a website.

All projects start as weakling babies that need to be initially nurtured to health. But if they never manage to get up on their feet, you have to throw them in the river.

Ok, that was exceedingly bad analogy.

Doing it badly

Sticking with a product that doesn't perform instead of creating a new project that just might.

Pieter chargers money

Freemium is a popular tactic for getting users.

Many people find it psychologically hard to charge for software.

Those are potential reasons why people don't charge for their products.

Not Pieter. He asks for money early and he asks for a lot. His book is $49 (and I bought it). Nomad List is $199.

The only real validation is people paying for your product

Doing it badly

Not charging at all, not charging early, not charging enough.

Pieter ships quickly

NomadList started as a Google Docs Spreadsheet.

Pieter turned it into a simple website and made that simple website available immediately.

He then kept working on it and improving it.

This is true of all his projects: ship the smallest thing that provides value, promote it, double-down on things that are working.

Doing it badly

Spending a year writing that web app without showing it to anyone.

Pieter does a lot of marketing and promotion

From early days he was promoting his stuff on his blog, Hacker News, Product Hunt, Reddit, Twitter etc.

There's a thin line between promotion and spamming. Don't cross it.

Most people, especially people good at programming, neglect promotion and marketing.

Some say that you should spend as much time on marketing as you do on coding the product.

I don't know what the right amount is but it's not zero.

Doing it badly

Not spending any time on promotion and marketing.

Not writing blog posts related to your product.

Not tweeting, not trying to promote your writing on Hacker News or reddit or quora.

If a tree falls in the forest and there's no one to see it, write a blog post about it and post it on Hacker News.

Pieter is good at promotion and marketing

I'm not sure if this can be thought but Pieter seems to have a natural gift for story telling and promotion.

Consider those 2 examples:

2014 I'm launching 12 Startups in 12 Months.
- "12 startups in 12 months" is a much more interesting story than "I'm starting a startup".
- This is important because if you submit "I'm starting a startup" blog post to Hacker News, it's unlikely to get traction. But "12 startups in 12 months" - it very well might.
- People are attracted to new, never-been-done-before.
- This trope is now over-used so you can't just copy it. But this is a big world and you don't have to be absolutely first and only. Pieter was inspired by 180 websites in 180 days project.
How I went from 100 to 0 things.
- He was robbed.
- It happens to many people but most people don't blog about it and promote that blog on Hacker News

It's more than a catchy headline, though.

The content of those blog posts is good and goes beyond the obvious.

Sharing a lousy story will not do you any good.

Doing it badly

Not spending any time on promotion and marketing.

On the other hand, there are people who simply spam with low-effort, low-quality posts. That is not going to work either.

Pieter is productive

The amount of projects he shipped is impressive.

The depth of some of his projects is impressive.

It's not inhumanely great, though.

When you consider he did it over 10 years, it's more than doable.

The difference between "doable" and "done" is measured in hours spent in front of the monitor, writing code, testing code, learning new things, applying those learnings.

Most people just don't spend enough time and / or don't spend the time productively.

Doing it badly

Writing a book is a time consuming activity.

Writing software is a time consuming activity.

There's no around spending a lot of time on your business (occasional exception notwithstanding).

Pieter learned programming by himself

This is not about self-thought vs. going to college.

This is about doing what's necessary.

Pieter had ideas for software but didn't know how to program.

For most people that's an insurmountable problem. It's learned helplessness.

All you need to learn programming is out there, for free. On YouTube, on blogs.

You just need to start learning and keep going.

You can read what Pieter says about DYI ethos.

Doing it badly

Trying to outsource programming to contractors or someone you find on Upwork, fantasizing about rising VC and hiring programmers, trying to find technical cofounder that will do all the work for 10% of the company.

Some people succeed by hiring programmers or contractors.

Most people don't because most people suck at managing programmers even more than Winklevoss brothers.

If you are a broke college student, you just don't have the money to hire people.

Pieter copies best practices

We are not born being the best programmers, best marketers, best writers.

There is no step-by-step guide to becoming the person that creates great products and markets them well.

But there are many businesses that do things well.

You can deconstruct what they do well and copy it.

It requires an attentive, inquisitive mind.

Let's take https://makebook.io/ as an example. Do you see what Pieter did well there?

Go ahead, look some more. What did he do well on that website?

Answer: social proof.

At the top: "Product Hunt's Book of the year". At the bottom: twitter testimonials of people fawning over his book.

Does your business use social proof to make people more likely to buy?

This is just one of many things that a business can do well.

For another example: go to https://nomadlist.com/ and try to sign up for Nomad List.

It's a master class in how to convert a visitor into a customer.

Figuring it why it makes for a great conversion funnel is left as an exercise for the reader.

Doing it badly

Never asking yourself: how can I do X better? How can I make my business better?

Not noticing the best practices that other businesses use.

Not learning from blogs, books, tweet storms.

Not applying the things you learn to your business.

Pieter is not a technologist

It's counter-intuitive that not being a technologist i.e. someone impressed by and interested in new technologies, can be an advantage when running a software business.

He build his $1.5 million/year empire on PHP, sqlite, and a single VPS server.

Those are not technologies that will impress anyone.

PHP is 27 years old. It might be older than you. It sure isn't Rust or Swift.

SQLite is 21 years old and everyone knows that a serious web service needs to use a serious database, like PostgreSQL

But they get things done. More precisely: Pieter gets things done using them.

Doing it badly

I'm a technologies so trust me, I know the siren song of fancy new languages, Kubernetes clusters, latest web frameworks.

I'm doing it badly. I'm trying to do it less badly.

When it comes to being productive, using a mature tool that you know well beats chasing the latest thing.

For me it's Go on the backend and Svelte on the front-end but as Pieter shows, you can be very prolific with a very old technology.

Pieter surfs the high waves

YouTube analytics in 2013.

Slack channel in 2014.

Virtual Reality in 2016.

Telegram bot in 2015.

Being a nomad in 2015.

QR Menu creator in 2020.

Bitcoin / NFT in 2021.

The guy is at the forefront of trends.

This doesn't necessarily make or break his businesses but it's easier to promote novel things.

This is not blind chasing of fads.

He doesn't have a podcast, a TikTok, or hangs out on Clubhouse.

He did nomadic lifestyle years before it was a trend.

Don't ignore trends but do apply your personal and busines filter.

It's also different than being a technologist.

He didn't rewrite his website on blockchain but he did notice that Bitcoin is a trendy thing and he did a project that is related to Bitcoin and provides value related to Bitcoin.

Doing it badly

Not paying attention to new trends.

Not trying to figure out what product can you build to take advantage of the new trend.

Not being selective about which trends to exploit.

Pieter is humble

The first thing he said in this comment

All of this is possible because I've been a HN reader since 2010 and was inspired by all of you and especially @patio11 on here to bootstrap my own things and do it VERY publicly.

This might not impact his business but it does impact how people perceive him.

Doing it badly

Being an arrogant prick that people dislike and root against.

Doing Things Right bias

Survivorship bias is for losers.

Doing Things Right bias is for winners.

So put that coffee down until you start Doing Things Right.

Learning from Pieter

Do you want to create successful projects? You can learn a lot from Pieter:

watch his How to build a startup without funding talk
read his blog posts
read his book
watch 4 hour AMA
listen to an interview he gave

Don't just read it passively.

You're a business detective. He has a successful business. Your job is to figure out all the things he did right, not just the things he's telling you.

You need to figure out that he's using social proof even if he never talks about using social proof.

Lessons learned from 15 years of SumatraPDF, an open source Windows app

2021-07-25T00:00:00Z

I released first version of SumatraPDF in 2006. That's 15 years ago which seems like a good time for a retrospective.

The app

SumatraPDF is a multi-format (PDF, ePub, Mobi, comic book, DjVu, XPS, CHM) viewer for Windows and currently looks like this:

The code

SumatraPDF is an open-source document reader for Windows. It started as a PDF reader, hence the name. Over time I've added for e-book formats (epub, mobi), comic books (cbz, cbr), DjVu, XPS, image formats etc.

It's about 127k lines of C++ (not counting libraries written by others).

It's written against Win32 API, not using GUI abstraction libraries like Qt. This contributes to making it as small and fast as possible.

Almost all of it was written by 2 people, with occasional contributions from others.

The amount of code written is actually higher. It is the nature of long running code bases that the code gets written and re-written. We delete, add, change.

It's a side project, done after hours, not a full time effort. How does a daily grind of working on an app looks like?

It looks like this:

You can also take a peek at my dev log. I've only started it a year ago so only covers 1 year out of 15.

Why I created SumatraPDF

SumatraPDF is what I call an accidental success.

I never wanted to write a PDF reader for Windows.

In 2006 I was working at Palm and one of my job duties was writing a PDF reader for Foleo, an ARM and Linux powered mini laptop. You never heard of Foleo because it was cancelled weeks before launch for reasons I'm not privy to.

At the time I didn't know that PDF is popular but Palm management did which is why they decided that PDF reader is a must have application. I ended up being the (sole) dev on the project.

Writing a PDF rendering library is a multi-year effort. We didn't have years so I used Poppler open-source library.

My job was to write a basic PDF viewer that used Poppler to render PDF pages into a bitmap in memory and blit those bitmap on screen.

PDF is a complex format and rendering of some PDFs is slow. I wanted to improve the speed because Jeff Bezos told me that speed is something that customers will always care about.

Accidental app

The way to improve speed is to profile the code and look at the result.

Unfortunately, the toolchain for unreleased ARM hardware wasn't very good. Forget about a profiler, kid, be grateful you have a C++ compiler and don't have to enter assembly by typing hex, like Steve Wozniak.

Windows had decent profilers, so I compiled Poppler for Windows.

Once I had the library working on Windows, I wrote simplest GUI app that would show the pages and allow navigating between pages.

What do you know: I had a simple PDF reader for Windows.

I released it on my website. It couldn't do much so I tagged it as version 0.1.

If you're not embarrassed by your app then you've waited too long to release it

I didn't come up with this nugget of wisdom but I agree with it.

Getting early users, learning what features they want the most beats toiling for months or years and implementing lots of features before you know anyone even cares.

Profiling, performance optimization and contributing to open source

Back to profiling: my plan worked.

I profiled the documents that took the longest to render and made a few surprisingly simple and surprisingly effective optimizations.

If memory servers, 2 optimizations had the biggest effect:

optimizing string class to use what is know as "small string optimization" i.e. adding a small buffer inside string class to hold small strings inline (as opposed to always allocating memory for the string). Strings were used frequently and most of them were small
fixing byte-at-a-time i/o by converting it to bulk reads. The way the code was structured in some code-paths it would do a virtual C++ call and a call to C read() function for each byte. Those are extremely cheap but not when you do it 5 million times

As a good boy I did submit my changes to Poppler.

As is my experience with contributing to open source projects, it was more of a miss than a hit.

Yes, I got 13 commits in but the project wasn't very active and the maintainers weren't eager to accept anything beyond small changes. Forget any major refactors.

I'm not one to voluntarily bash my head against the wall so I stopped trying.

(As you can see, I'm a fantastic team player).

Code quality

I want it and you should want it to.

How to maintain high code quality while working mostly solo, with no-one doing code reviews, no dedicated QA team?

Here's how:

test the code yourself. Step through newly added code in the debugger, verify the newly added functionality works as expected and in general use the app a lot
automated crash reporting. Unfortunately it's a pain to build but this is single most important thing you can do to improve quality of your software. Briefly: setup exception handlers to catch crashes in the app, in crash handler download symbols from the server to get readable callstack, create a crash report that includes callstacks of all thread, program and os information, log and submit that to a server. On the server, process those files and generate web pages for easy viewing of the crashes. Like I said: it's a pain to build. Once you have crashes, look at them occasionally and try to figure out what went wrong and fix it
assert(). asserts are well established practice in C++ code: an additional code only executed in debug builds that verifies some conditions are true. If they're not, something went wrong and you should investigate. I wrote wrote my own assert-like function which I enable in non-debug pre-release builds so that I automatically get bug reports from people hitting those conditions. Trust me: there's no amount of testing you can do yourself that would match all the different things that a thousand people will do just by using the app.
logging. When investigating issues it helps to know what sequence of events led to a crash. My tiny logging module logs to a block of memory. That gets sent along with crash report. I also have an option to log to a file and I've recently added logging to a separate logging app via named pipe. This is perfect because most of the time I don't care about the logs but when I do, I don't want to restart the app to enable logging. With separate logging app, SumatraPDF is logging all the time and when it detects that logging app is running, it'll also log to it. Implementation was trivial: logging app creates a named pipe, logger opens the pipe (like a file) and if open succeeds, it means the logger app is running and it reads the logs we write to the pipe
static code analysis: max level of warnings in C++ compiler, make warnings into errors, Visual Studio's `/analyze' option, cppcheck, clang-tidy, GitHub's CodeQL. Run those occasionally and fix the errors and warnings
ASAN (Address Sanitizer), is fantastic. Was added in some point release of Visual Studio 2019. At a very small performance cost it can detect if you over-write memory or try to read uninitialized memory. I have a configuration with ASAN enabled. It's fast enough to be used as a regular build.
stress testing. Sumatra's job is mostly to render complex document format. There often are crashes in specific files due to complexity of the formats. To ensure lack of crashes I wrote a stress test code that reads and renders all files in a directory. I typically run it before a release on a large collection of test files I amassed over the years
unit testing. I don't have a lot of them, they're mostly for testing edge cases for low-level functionality like string formatting. They occasionally find bugs.
memory leaks. It's surprisingly hard to find an easy to use memory leak detection tool. I'm working on a very simple built-in leak detector. In the meantime I'm using Dr. Memory. It works but it's super slow.

Frequent releases

When you don't have many features, improving the app is fast and easy. It doesn't take much effort to implement "Go to" dialog (implemented in v 0.2).

On one hand I don't want to release too often but I also do want the users to get new features as quickly as possible.

My policy of new releases is: release when there's at least one notable, user-visible improvement.

Web apps take it to the extreme (some companies deploy to production multiple times a day).

In desktop software it's a bit more involved and I had to build functionality to make it easy i.e. add a check for new releases, write an installer that can update the program.

BTW: I mean "frequent in proportion to amount of new code written". SumatraPDF releases are not frequent in absolute terms but frequent if you consider that it's a part-time, after hours project.

Treat open source projects like commercial software

Majority of open source projects probably don't fall into this category, but if you want your open source to be as successful as possible, act as if it was a commercial product from a software company.

What does it mean in practice?

From day one I created a website for the app. It had screenshots, it had documentation, it was easy to download and install. Granted, a kind soul on Reddit called it "a website made by a 6-year old". The lesson here is two-fold:

ignore haters and assholes
a website built by a 6-year old is better than no website. It doesn't have to be pretty, it has to be functional

I did basic SEO. Nothing beyond Google's "SEO 101" docs: just pay attention to URLs, put the right meta-data, use the right keywords.

I had a forum for users to ask questions, submit feature requests and occasionally support each other.

I made the installation process as easy as possible.

Everything that is a good idea for promoting commercial software is also a good idea for open source project.

Switching the engine while the car is running

At some point I decided to switch from Poppler to mupdf because mupdf was better and actively maintained.

Changing the app to use completely different library is not something you can do in an afternoon.

It's demoralizing to work long time on code that doesn't even compile.

To keep things compiling while also working towards supporting alternative rendering engine I developed an abstraction for the rendering engine.

The engine would provide the functionality the UI needed: getting number of pages in the document, sizes of each page (to calculate layout), rendering a page as a bitmap etc.

I'm much less enthusiastic about abstractions than most programmers (at least those who like to opine on Hacker News) but in this case it served me well.

I was able to incrementally convert program form using Poppler API to using Poppler via engine abstraction to using mupdf via Engine abstraction.

For a while I supported both engines at the same time but eventually I switched to just mupdf, to keep the app small.

This opened the door for supporting other formats via the same abstraction.

Simplicity vs. customizability

Simplicity sells.

I learned that from the history of Mozilla Firefox.

Before Firefox there was Netscape Navigator. It was a beast of an app, combining web browser with e-mail client.

Netscape couldn't help themselves and was adding features upon features, leading to very complex UI.

A small group of renegades within Mozilla forked the code and focused on simple UI.

Simple Firefox was much more popular than the complex Navigator and eventually ate it completely.

From the beginning my goal was to keep the UI of SumatraPDF as simple as possible. An 80/20 app: 80% of functionality with 20% of the UI.

This requires resolve. I constantly get requests to add more icons to the toolbar and I constantly have to say "no" because adding 2 more icons to the toolbar to satisfy 10% of users makes the app slightly worse for 100% of the users.

Another trap is a siren song of additional settings. Sometimes people suggest that instead of doing X, the program should do Y. Not willing to remove X, they suggest adding a new UI setting "[ ] Do Y instead of X".

Having settings dialog with 100 settings is not a good solution. It makes the app worse for everyone due to overwhelming them with choices and hiding important options in a sea of non-important options.

Not to mention that every conditional behavior requires more code, more potential bugs and more testing.

That being said, I also believe customizability is important. I believe that a big reason for Winamp being such a dominant music player (at the time) was its ability to skin the whole UI.

Some advanced features might only be used by 20% of users but those users are most likely power users that will evangelize the app more than the other 80% of the users.

My solution to UI simplicity vs. customizability: advanced settings file.

I designed a simple, human readable (and human writeable) textual format for advanced settings. Think JSON, but better.

I didn't bother to write UI for changing those advanced settings. I just launch notepad.exe with the file. When user changes the settings and saves the file, I reload it and apply the changes.

Be water, my friend

Change is the only constant. We must adapt to the changes in the world.

I can't believe how many popular projects still use craptastic Sourceforge for source repository or mailing list.

Actually, I can believe: changing things takes effort and the path of least resistance is to do nothing.

I started with Sourceforge, switched to code.google.com and then to github.com.

I switched forum software three times.

I've added a browser plugin and then removed it when browsers stopped supporting such plugins.

I changed the format for storing preferences from binary to human readable text.

Windows XP went from being the OS used by majority of users to no longer being supported (long after Microsoft stopped supporting it).

At first I only had 32-bit build and now I have both but emphasize 64-bit builds.

Think outside of the box

Thinking outside of the box is hard because the box is invisible.

SumatraPDF wasn't the first PDF reader application ever written.

But most PDF readers do not become multi-format readers.

In hindsight it's an obvious idea to support as many document formats as possible but it took me 5 years to realize it.

Most readers are still single format and I do believe being multi-format helped SumatraPDF become popular.

I can't say it's totally unique idea. There were multi-format image viewers long before SumatraPDF and I probably was inspired by them.

Small and fast - pick both

By today's standards SumatraPDF is tiny (installer smaller than 10 MB) and starts up instantly.

I believe being small and seemingly fast was a big reason for adoption.

This comes back to Jeff Bezos' wisdom: there will never be a time when users want bloated and slow apps so being small and fast is a permanent advantage.

How do I keep SumatraPDF small?

I avoid unnecessary abstractions. Window's system of controls is a giant pain in the ass to program against. I could use wrappers like Qt, WxWindows or Gtk. They are easier to use but cause instant, giant bloat.

I'm not afraid to write my own implementation of things. I have my own JSON, HTML / XML parsers that are a fraction of size of the popular libraries for those tasks.

I aggressively take advantage of rich functionality included in Windows.

Let's say I need to do a network request. I could include a monster library like curl or I could write 300 lines of code using win32 APIs. I wrote 300 lines of code.

An absence of bloat is hard to notice because it isn't there.

My pet peeve is over-using XML for storing data.

When I worked at Palm I was at a design meeting for auto-update system for a phone. Part of it was storing information about the current version in the image, downloading information about the latest version and comparing them.

The developer decided to use XML for storing that information. That seemed like a lot of bloat for storing simple information like a version number. An compliant XML parser alone is a lot of code. Surely a simple binary format would be easier to implement, I suggested and was ignored.

If you don't have the power to fire someone, your ideas will be ignored.

(As you can see, I'm a great team player.)

For storing advanced settings I designed and implemented a file format that is smaller than XML, readable and writeable by humans and can be implemented in few hundred lines of code. It's as powerful as JSON and even more readable.

It's so simple that after implementing it I had the time to implement a serialization system for C++ objects and a Go code generator. To add more settings I don't have to write more C++ code. I just add data definition to Go generator, re-run it and get data-driven C++ parsing auto-generated.

It's my project and I act like it

When someone pays you to write code you have to do it the way they like it.

A big attraction of working on code you're not paid for is that there is no one who can tell you what to do or how to do it.

My code would not pass a code review at Google and not because it's bad but because it's often unorthodox. Outside of accepted dogma.

(As you can see, I'm a great team player.)

I always used SumatraPDF as my playground for testing crazy ideas.

Minimize the code size by not using STL? That's crazy but I did it. Granted, in 2006 STL wasn't very good.

I learned about how Plan 9 C code had non-traditional scheme of #include files where they don't put #ifdef wrappers in each .h file to allow multiple inclusion and .h files don't include other .h files. As a result .c files have to include every .h file they need and in the right order. It's a bit of a pain and no other modern C++ codebase I know of maintains such discipline.

But it's my project so I did it and I keep doing it. It prevents circular dependencies between .h files and doesn't inflate C++ build times because of careless including the same files over and over again.

I implemented a CSS inspired UI system. Not great, but mine. And I plan to replace with a different one.

Because I can.

Because no one can tell me not to.

Cross-platform is over-rated

SumatraPDF is unabashedly a Windows only app.

Supporting other platforms (Linux, Mac, Android) is one of the most frequent requests. A request that I have to decline.

First, there is a pragmatic reason: I just don't have the bandwidth to write code for 3 platforms.

Second, I believe an excellent app for one platform can become more popular than a mediocre app for 3 platforms.

Coming back to the first reason: I don't have the bandwidth to write 3 excellent apps. Part of the reason SumatraPDF is small is my use of win32 APIs for the UI.

The only way for one person to even attempt cross-platform app is to use a UI abstraction layer like Qt, WxWidgets or Gtk.

The problem is that Gtk is ugly, Qt is extremely bloated and WxWidgets barely works.

Tests are not necessary, neither are code reviews

I'm not saying tests are bad or that you shouldn't write test or do code reviews.

I'm saying that they are not necessary.

Dogma is powerful. Sometimes in my corporate life I felt like writing tests was just going through motion. Maybe we should spend more time writing code instead, I though?

But try to make a nuanced point about more tests vs. more code to your fellow developers and you'll be burned at stake and your smoldering carcass will be thrown to wild dogs. Village children will use your severed head to play soccer.

(As you can see, I'm a great team player.)

And yet I do know that you can write complex, relatively bug free code without tests, because I did it.

I do know that you can write complex, relatively bug free code without anyone looking over your code, because I did it.

If no one uses your app then who cares if it crashes.

If many people use your app and it crashes, they'll tell you and then you'll fix it.

Overnight success takes a decade

SumatraPDF is relatively popular. Not Facebook popular or DOOM popular, but more popular than most apps. A respectable level of popular.

It all started with v 0.1 and a trickle of downloads. It remained a trickle for many, many months.

I'm not sure there's a lesson here.

Success often takes a long time.

Unfortunately, at that stage it's undistinguishable from (eventual) failure so this wisdom doesn't help you if you're working on a not-yet-successful project and debating if you should continue or abandon

The money

Open source is not a good business model.

If you want to make money do literally anything else: try to sell software, do consulting, build a SAAS and charge monthly for it, rob a bank.

I did experiment with making money and made some.

There was a time AdSense would pay decent CPM so I put AdSense ads on the website and it made some money. I no longer do because the rates did plummet and it isn't worth annoying people. My soul has a price and AdSense can no longer afford it.

Now I'm experimenting with Patreon and Paypal donations. It makes more than $100 a month but not much more than that.

Like I said: don't start open source project with intent to make money.

Rarely you can have both: freedom to do whatever you want and a good pay so pick what is more important to you. Open source gives you freedom but not money.

On to the future

I need to continue being like water.

For years I resisted adding editing features. "It's just a reader" I said. But why not add editing? If people want it, give it to them.

The future of all software is as a web app. Why not bring the spirit of SumatraPDF to the web?

Those are just a few ideas I have today.

Being like water means that in 5 years I'll have other ideas, informed by what's happening at that time.

How I use Roam Research

2021-06-27T00:00:00Z

I take lots of notes and I'm always looking for that perfect note taking app.

Because I take a lot of notes, mobile-first note taking apps (e.g. Google's Keep) are of no interest to me.

To me most important thing is:

lowest possible friction in adding new notes
fast way of finding existing notes

In theory I could use Google Docs as a note taking app but both Notion and Roam Research make it much faster to jot a new note or find a note I wrote in the past.

Over the years I've cycled through many note taking systems: a single big text file, collection of markdown files in a git repository, a hosted wiki, a self-hosted wiki, Evernote, my own note taking web app.

A combination of Notion and Roam Research is, I hope, the end of the road and will be good enough for the rest of my life.

This article describes how I use Roam Research.

Roam use cases

Here are my main use cases of Roam:

taking notes / writing
keeping a log of activity (what I did)
managing todo list
managing projects
bookmarking

Roam can be hard to grasp at first. Things you need to know:

Roam is a collection of named pages
you can trivially link between pages. Writing [[foo]] or #foo anywhere creates a link to a page named foo. A page also lists backlinks (i.e. pages linking to this page) at the end
there's automatically a page for every day and the default view is a page for current day

Knowing that, let's go through my use cases.

Taking notes / writing

Let's say I have an idea to write an article about my use of Roam Research (i.e. this very article).

In most other note taking apps I would first have to think about where to put that note.

In Roam I just start writing in the page for current day and tag it with #draft.

Later on I can go to a draft page and see all the drafts I'm currently working on.

There is a greater point here: Roam is "write now, organize later" system.

When I have a though, I jot it down in today's page and tag it with keywords that will help me find it later.

It's the least amount of friction between having a thought and recording that thought for the future.

Taxonomy is personal. I use #draft for my draft articles, but you could just as well use #article or #articles (using singular vs. plural for tags is surprising hard to decide).

Logging of activity

In personal life logging of activity is just for curiosity.

I write down what I did in a page for current day and tag log entries with #done.

I visit done page to review what I did in the past.

What if I want to track what I did on a specific project, e.g. SumatraPDF?

I further tag the entry with a tag for the project e.g. #sumatrapdf. Roam has filtering capabilities that allows me to see entries tagged with #done AND #sumatrapdf.

In professional life logging your work could help you level up. If no-one knows what you did, did it happen? Will it lead to a promotion?

Roam is perfect for logging what you did so that you can review it later and send a summary to your manager or consulting client.

If I were a Google software engineer, I would tag my entries with #done and #work.

If I was doing a consulting job I would tag them as #done, #consulting and maybe with a tag specific to the project or the client.

Again, taxonomy is personal. You can use other tags if they make more sense to you.

Managing projects

I work on several software projects at the same time.

I have a page per project that serves as a hub of information related to the project.

On that page I have:

most important links (e.g. to github repository, Digital Ocean dashboard for deployed project etc.)
list of things to do next
list of ideas for future improvements
links to competing products
technical overviews. Memory is always fading so it pays to jot down things that might help you remember key information
documentation

Todo list

I don't use dedicated todo list applications. I think that creating hundreds of todo items is overwhelming and counter-prodcutive.

I do believe in one of the tenets of Getting Things Done system: writing down your ideas and todo tasks to get them out of your head but be able to review them in the feature.

I maintain a "maybe do list" instead of "todo list".

For example, if I have an idea for SumatraPDF, I jot it down in Roam and tag it with #sumatrapdf. I can then review those ideas in the future when I have time to work on SumatraPDF.

I do maintain a very short (few items) todo list for near future. Mostly today and tomorrow.

This is an anti-procrastination device.

I find that when I start working on a task, I manage to finish it.

Procrastination is what keeps me from even starting.

Having a list of things to do in the morning helps me start working (as opposed to starting to browse Twitter).

It's worth mentioning that other people do use Roam as a sophisticated todo list. Roam is flexible enough. For example Roam understands dates so you can have pages with tags that record a completion date for a task and you can use Roam's powerful query syntax to find tasks that are supposed to be done in the next week.

Bookmarking

Bookmark managers in browsers are not good at managing large list of bookmarks. Hierarchies don't scale.

Roam's easy tagging makes it a replacement for services like pinboard.in (and once-famous-now-defunct del.icio.us). Found a page about Docker you want to bookmark for later? Drop the link in Roam and tag it with #docker. Find it later in the docker page.

There is a use case for browser's bookmarks: a fast way to get to most frequently used websites. I'm experimenting with using Roam for that as well.

I've created page bookmarks with most frequently used links. Opening a Roam page from scratch is relatively slow so I keep that page open at all times and as a pinned tab (a Chrome thing).

One more thing...

Actually, lots of things.

I'm relatively light user of Roam. I'm still discovering new use cases for Roam and better ways of managing notes.

But even just the basics usage of jotting things down and tagging them for cross-referencing is very useful. Speed and flexibility are important and Roam has both.

Roam can offer much more if you're willing to invest time. Powerful search queries, tags with values, storing files. There's a reason people become obsessed with Roam.

I'm not obsessed but I do appreciate a great tool.

Notion and Roam?

Why use both Notion and Roam? Why not stick with just one of them?

Notion has some use cases that Roam Research doesn't support well. For example sharing a subset of notes with other people.

They also have different philosophies of managing notes.

Notion is strongly hierarchical. Pages are nested within pages. It's great when hierarchy is obvious but sometimes I spend brain cycles figuring out where in the hierarchy should a new page go.

Roam is a flat collection of interlinked pages. It's more difficult to grasp but removes the thinking. Write things down, tag them now and maybe organize later.

Roam is more quirky. While Notion has more polished UI, Roam is often faster to use. Trying to change a link destination in Notion can be a struggle.

Final thoughts

We live in a golden era of note taking apps. I wish I had Notion or Roam Research 10 years ago.

Notion opened floodgates of high quality note taking apps. Before Notion there was just Evernote. After Notion we got lots of other options, including Roam.

I think they are both powerful enough to serve as a note taking system for life.

The things we do to ship desktop software

2019-05-01T00:00:00Z

I wrote a small utility for Windows. It indexes a hard-drive and allows to find a file by name in under a second.

It might surprise you that I spent more time on things that are not related to core functionality. Let's call it a tax of shipping desktop software.

Here are some of the taxes you need to pay to ship a desktop application to users.

Logging

In the long term you want to be able to diagnose problems quickly and logging helps in that.

So I've implemented logging to a file.

I went beyond basics and implemented a way to easily see the logs in a window. Simple to implement as all you need is to use built-in read-only text box control.

You can toggle log window from File menu. As an additional touch, if it logs an error and it's a dev build, it automatically opens a window.

Website

You need to have a website. It's a simple website: few screenshots and a download button, but it does take a few hours to make.

It also has a feedback page, so that people can tell me how to improve the program.

Signing certificate

On Windows anti-virus software and Microsoft's anti-phishing systems are lousy and often flag innocent software as malware.

You can decrease the probability of that by signing your installer and executables with software signing certificate.

Unfortunately, this certificate is both expensive and pain in the ass to buy. Entities selling them want a proof of your (or your company) identity but the rules are often bureaucratic idiocy.

Installer

You need an installer because it makes the life of your users easier.

What installer does is not complicated:

copy the files in the right place
create necessary registry entries for un-installation
create a shortcut on the desktop and in Start Menu
implement un-installation logic to delete that which has been created during installation (files, shortcuts, registry entries)

Auto-update system

I want the users to get the latest version in the easiest possible way, One possible way to implement it:

Build system

To make release process smooth, you need automated build and release system.

In my case it's a Go program that builds the software, signs it, uploads to Digital Ocean Spaces for storage, updates the info needed for auto-update system.

The taxes add up

Those things are not hard to implement. Individually they don't take a lot of time to implement.

But when you add it all up, for this particular project I estimate I spent more time building those system than on the core functionality.

The bright light in the tunnel is:

I can re-use most of this code in other software
it only needs to be written once. The more time I spend on the app, the less expensive those things are as percentage of total development time

Lessons learned porting 50k loc from Java to Go

2019-04-05T00:00:00Z

I was contracted to port a large Java code base to Go.

The code in question is a Java client for RavenDB, a NoSQL JSON document database. Code with tests was around 50 thousand lines.

The result of the port is a Go client.

This article describes what I've learn in the process.

Testing, code coverage

Large projects benefit greatly from automated testing and tracking code coverage.

I used TravisCI and AppVeyor for testing. Codecov.io for code coverage. There are many other services.

I used both AppVeyor and TravisCI because a year ago Travis didn't have Windows support and AppVeyor didn't have Linux support.

Today if I was settings this up from scratch, I would stick with just AppVeyor, as it can now do both Linux and Windows testing and the future of TravisCI is murky, after it was acquired by private equity firm and reportedly fired the original dev team.

Codecov is barely adequate. For Go, they count non-code lines (comments etc.) as not executed. It's impossible to get 100% code coverage as reported by the tool. Coveralls seems to have the same problem.

It's better than nothing but there's an opportunity to do things better, especially for Go programs.

Go's race detector is great

Parts of the code use concurrency and it's really easy to get concurrency wrong.

Go provides race detector that can be enabled with -race flag during compilation.

It slows down the program but additional checks can detect if you're concurrently modifying the same memory location.

I always run tests with -race enabled and it alerted me to numerous races, which allowed me to fix them promptly.

Building custom tools for testing

In a project that big it's impossible to verify correctness by inspection. Too much code to hold in your head at once.

When a test fails, it can be a challenge to figure out why just from the information in the test failure.

Database client driver talks to RavenDB database server over HTTP using JSON to encode commands and results.

When porting Java tests to Go, it was very useful to be able to capture the HTTP traffic between Java client and server and compare it with HTTP traffic generated by Go port.

I built custom tools to help me do that.

For capturing HTTP traffic in Java client, I built a logging HTTP proxy in Go and directed Java client to use that HTTP proxy.

For Go client, I built a hook in the library that allows to intercept HTTP requests. I used it to log the traffic to a file.

I was then able to compare HTTP traffic generated by Java client to traffic generated by my Go port and spot the differences.

Porting process

You can't just start porting 50 thousand lines of code in random order. Without testing and validating after every little step I'm sure I would be defeated by complexity.

I was new to RavenDB and Java code base. My first step was to get a high-level understanding how Java code works.

At the core the client talks to the server via HTTP protocol. I captured the traffic, looked at it and wrote the simplest Go code to talk the server.

When that was working it gave me confidence I'll be able to replicate the functionality.

My first milestone was to port enough code to be able to port the simplest Java test.

I used a combination of bottom-up and top-down approach.

Bottom-up part is where I identified the code at the bottom of call chain responsible for sending commands to the server and parsing responses and ported those.

The top-down part is where I stepped through the test I was porting to identify which parts of the code need to be ported to implement that part.

After successfully porting the first step, the rest of the work was porting one test at a time, also porting all the necessary code needed to make the test work.

After the tests were ported and passing, I did improvements to make the code more Go-ish.

I believe that this step-by-step approach was crucial to completing the work.

Psychologically, when faced with a year-long project, it's important to have smaller, intermediate milestones. Hitting those kept me motivated.

Keeping the code compiling, running and passing tests at all times is also good. Allowing bugs to accumulate can make it very hard to fix them when you finally get to it.

Challenges of porting Java to Go

The objective of the port was to keep it as close as possible to Java code base, as it needs to be kept in sync with Java changes in the future.

I'm somewhat surprised how much code I ported in a line-by-line fashion. The most time consuming part of the port was reversing the order of variable declaration, from Java's type name to Go's name type. I wish there was a tool that would do that part for me.

String vs. string

In Java, String is an object that really is a reference (a pointer). As a result, a string can be null.

In Go string is a value type. It can't be nil, only empty.

It wasn't a big deal and most of the time I could mechanically replace null with "".

Errors vs. exceptions

Java uses exceptions to communicate errors.

Go returns values of error interface.

Porting wasn't difficult but it did require changing lots of function signatures to return error values and propagate them up the call stack.

Generics

Go doesn't have them (yet).

Porting generic APIs was the biggest challenge.

Here's an example of a generic method in Java:

public <T> T load(Class<T> clazz, String id) {

And the caller:

Foo foo = load(Foo.class, "id")

In Go, I used two strategies.

One is to use interface{}, which combines value and its type, similar to object in Java. This is not preferred approach. While it works, operating on interface{} is clumsy for the user of the library.

In some cases I was able to use reflection and the above code was ported as:

func Load(result interface{}, id string) error

I could use reflection to query type of result and create values of that type from JSON document.

And the caller side:

var result *Foo
err := Load(&result, "id")

Function overloading

Go doesn't have it (and most likely will never have it).

I can't say I found a good solution to port those.

In some cases overloading was used to create shorter helpers:

void foo(int a, String b) {}
void foo(int a) { foo(a, null); }

Sometimes I would just drop the shorter helper.

Sometimes I would write 2 functions:

func foo(a int) {}
func fooWithB(a int, b string) {}

When number of potential arguments was large I would sometimes do:

type FooArgs struct {
	A int
	B string
}
func foo(args *FooArgs) { }

Inheritance

Go is not especially object-oriented and doesn't have inheritance.

Simple cases of inheritance can be ported with embedding.

class B : A { }

Can sometimes be ported as:

type A struct { }
type B struct {
	A
}

We've embedded A inside B, so B inherit all the methods and fields of A.

It doesn't work for virtual functions.

There is no good way to directly port code that uses virtual functions.

One option to emulate virtual function is to use embedding of structs and function pointers. This essentially re-implements virtual table that Java gives you for free as part of object implementation.

Another option is to write a stand-alone function that dispatches the right function for a given type by using type switch.

Interfaces

Both Java and Go have interfaces but they are different things, like apples and salami.

A few times I did create a Go interface type that replicated Java interface.

In more cases I dropped interfaces and instead exposed concrete structs in the API.

Circular imports between packages

Java allows circular imports between packages.

Go does not.

As a result I was not able to replicate the package structure of Java code in my port.

For simplicity I went with a single package. Not ideal, because it ended up being very large package. So large, in fact, that Go 1.10 couldn't handle so many source files in a single package on Windows. Luckily it was fixed in Go 1.11.

Private, public, protected

Go's designers are under-appreciated. Their ability to simplify concepts is unmatched and access control is one example of that.

Other languages gravitate to fine-grained access control: public, private, protected specified with the smallest possible granularity (per class field and method).

As a result a library implementing some functionality has the same access to other classes in the same library as external code using that library.

Go simplified that by only having public vs. private and scoping access to package level.

That makes more sense.

When I write a library to, say, parse markdown, I don't want to expose internals of the implementation to users of the library. But hiding those internals from myself is counter-productive.

Java programmers noticed that issue and sometimes use an interface as a hack to fix over-exposed classes. By returning an interface instead of a a concrete class, you can hide some of the public APIs available to direct users of the class.

Concurrency

Go's concurrency is simply the best and a built-in race detector is of great help in repelling concurrency bugs.

That being said, in my first porting pass I went with emulating Java APIs. For example, I implemented a facsimile of Java's CompletableFuture class.

Only after the code was working I would re-structure it to be more idiomatic Go.

Fluent function chaining

RavenDB has very sophisticated querying capabilities. Java client uses method chaining for building queries:

List<ReduceResult> results = session.query(User.class)
                        .groupBy("name")
                        .selectKey()
                        .selectCount()
                        .orderByDescending("count")
                        .ofType(ReduceResult.class)
                        .toList();

This only works in languages that communicate errors via exceptions. When a function additionally returns an error, it's no longer possible to chain it like that.

To replicate chaining in Go I used a "stateful error" approach:

type Query struct {
	err error
}

func (q *Query) WhereEquals(field string, val interface{}) *Query {
	if q.err != nil {
		return q
	}
	// logic that might set q.err
	return q
}

func (q *Query) GroupBy(field string) *Query {
if q.err != nil {
		return q
	}
	// logic that might set q.err
	return q
}

func (q *Query) Execute(result inteface{}) error {
	if q.err != nil {
		return q.err
	}
	// do logic
}

This can be chained:

var result *Foo
err := NewQuery().WhereEquals("Name", "Frank").GroupBy("Age").Execute(&result)

JSON marshaling

Java doesn't have a built-in marshaling and the client uses Jackson JSON library.

Go has JSON support in standard library but it doesn't provide as many hooks for tweaking marshaling process.

I didn't try to match all of Java's functionality as what is provided by Go's built-in JSON support seems to be flexible enough.

Go code is shorter

This is not so much a property of Java but the culture which dictates what is considered an idiomatic code.

In Java setter and getter methods are common. As a result, Java code:

class Foo {
	private int bar;

	public void setBar(int bar) {
		this.bar = bar;
	}

	public int getBar() {
		return this.bar;
	}
}

ends up in Go as:

type Foo struct {
	Bar int
}

3 lines vs. 11 lines. It does add up when you have a lot of classes with lots of members.

Most other code ends up being of equivalent length.

Notion for organizing the work

I'm a heavy user of Notion.so. Simplifying a lot, Notion is a hierarchical note taking application. Think a cross of Evernote and a wiki, exquisitely designed and implemented by top notch software designers.

Here's how I used Notion to organize my work on Go port:

Here's what's there:

not shown above, I have a page that is a calendar view where I take short notes about what I work on on a given day and how much time I spent. This is important information since it was a hourly contract. Thanks to those notes I know that I spent 601 hours over 11 months
clients like to know the progress. I had a page for each month were I summarized the work done like this:
Those pages were shared with the client.
A short-term todo list helps when starting work each day:
I even managed invoices as Notion pages and used "Export to PDF" function to generate PDF version of the invoice

Additional resources

I've provided some additional commentary in response to questions:

in Hacker News discussion
in /r/golang discussion

Other material:

if you need a NoSQL, JSON document database, give RavenDB a try. It's chock full of advanced features
if you're programming in Go, try a free Essential Go programming book
if you're interested in Notion, I'm world's most advanced user of Notion:
- I reverse engineered Notion API
- I wrote an unofficial Go library for Notion API
- all content on this website is written in Notion and published with my custom toolchain

Trade offs in designing versatile log format

2019-03-25T00:00:00Z

This article shows that when designing software even seemingly simple things are complicated and trade offs abound.

I wanted to log events to a file. I had several requirements for my design:

it should be simple and therefore easy to implement
it should be human-readable
it should allow various types of events

It's not a hard problem.

I could log it as stream of JSON objects. It would allow different types of events. It's easy to implement (in the sense that there's a library in every language for the hard part of encoding/decoding JSON).

One thing: it's not particularly human friendly. I wouldn't enjoy looking at tail -f of a JSON log.

How about something like Apache server logs:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"

It's readable but doesn't allow for different types of events.

Implementing this format isn't challenging but the format is ad-hoc. For each kind of data I would have to write a completely new formatter / parser.

How about something more structured:

ip: 127.0.0.1
request: GET /apache_pb.gif HTTP/1.0
status code: 200

We can write a library that serializes key / value pairs and that gives us ability to write different kinds of events to the same file with generic code.

But hold on, how do we know where one event ends and another starts?

We must amend our format add add a record separator, like ---:

ip: 127.0.0.1
request: GET /apache_pb.gif HTTP/1.0
status code: 200
---

We're not there yet. We use newline to separate lines that encode a single key / value pair. What if the value has a newline in it?

body: hello fred
I'm writing to you...

We could escape the value:

body: hello fred\nI'm writing to you...

Escaping is not a good solution.

Without escaping, we can just write out the data as-is.

With escaping we have to scan the whole thing to determine if it needs escaping, and escape if it does.

Same goes for un-escaping.

This is slow and error prone.

Furthermore, what if the value is really large? It won't be readable encoded on a single line.

Let's amend our format to accommodate large values:

body:+33
hello fred
I'm writing to you...

33 is the size of the value. Knowing the size we don't need escaping. It's faster, simpler to implement, more readable and supports binary data (like images).

To formalize, key / value pair can be encoded in 2 ways:

${key}: ${value}\n
${key}:+${sizeOfValue}\n${value}

Let's revisit the idea of using a separator:

ip: 10.0.0.1
request: GET / HTTP/1.0
---
ip: 10.0.0.2
request: GET /index.html HTTP/1.0
---

For delimiting records we can use the same tricks we used for encoding large values:

36
ip: 10.0.0.1
request: GET / HTTP/1.0
43
car: Toyota Corolla
year: 2018
price: 21000

First line is size of data as a string + newline. The data of this size follows.

This is very generic framing, agnostic to what is inside. It could be a picture, a JSON-encoded data or our key / value format.

We've arrived at a layered design:

first layer is encoding arbitrary chunks of data by writing size and then data
second layer is key / value format inside the data

It would be even simpler if the size was 8 byte, 64-bit integer. It wouldn't be human-readable, though, so I picked a slightly more complicated, string-based encoding.

It's not quite right yet.

Above we have logged an HTTP requests info and car info in the same file.

How do we know what type of record did we read?

Let's amend our framing with optional name:

36 httplog
ip: 10.0.0.1
request: GET / HTTP/1.0
43 carinfo
car: Toyota Corolla
year: 2018
price: 21000

Adding optional name (httplog and carinfo) allows us to know what kind of data is encoded in a given chunk of data.

Finally let's add non-optional timestamp in Unix Epoch format in milliseconds:

36 1553564864010 httplog
ip: 10.0.0.1
request: GET / HTTP/1.0
43 1553564864115 carinfo
car: Toyota Corolla
year: 2018
price: 21000

Unlike name, which is optional, I decided timestamp is not optional. You can set it to 0 if you don't need it but for most logging needs you want a timestamp.

Traditional Unix Epoch has precision of a second and that seems not enough. Millisecond-precision seems good enough. Nanosecond was also an option but seems like an over-kill.

What if you need more structure than key / value pairs?

Simplicity means that you can't implement every possible feature. This format doesn't preclude more structure: you can always use JSON as the value in key / value field. It's just not going to be as easy to use.

Implementing the thing

This is not just theoretical exploration. I've implemented this format as a Go package siser.

It took me 2 days to implement. It's just under 500 lines of code:

$ wc -l reader.go record.go util.go writer.go
     167 reader.go
     196 record.go
      79 util.go
      57 writer.go
     499 total

Not counting tests but I do have them because I need this code to be rock solid. Storing data is serious business and has to be reliable.

How fast is it? I benchmarked it against JSON encoding:

BenchmarkSiserMarshal-12      	 1000000	      1136 ns/op
BenchmarkJSONMarshal-12       	 1000000	      1407 ns/op
BenchmarkSiserUnmarshal-12    	 5000000	       374 ns/op
BenchmarkJSONUnmarshal-12     	  500000	      3353 ns/op

The benchmark is mostly to make sure that I didn't make a big performance mistake. It would be embarrassing to be slower than JSON encoding. As a side note: it's impressive how fast JSON marshaling in Go standard library is.

I'm moving all my logging needs to this format.

This format is also good for very simple stores aka databases. The file can be seen as an append-only database log. To update a record I just write a new entry and it'll over-write earlier entry.

The roads not taken

After reading this you might not be impressed. This design is clearly inspired by many others. All I did is put known things in a specific, but very familiar, way.

Simplicity is insidiously non-trivial. The design described here is a v2 of siser library. First design used --- for record separator, didn't have name and timestamp.

Only several months after first version I got enough insight to improve it.

In turn siser was an evolution of less robust ideas I implemented earlier. It took experiencing the limitations of those earlier designs in real use for better ideas to emerge.

This is even more apparent when you look at mistake of others.

MIME is a format used for encoding e-mail messages. While in some ways it's very close to this format, they made a mistake of using a boundary string for separating multiple parts. Compared to framing data with size prefix it's so much hard to implement. A MIME decoder is more than 500 lines of code and doesn't offer more features.

Other example of massive mistakes of the past is choosing XML as a format for describing ant or Visual Studio build files. XML is super slow, unreadable for humans, hard to work with programmatically. A conforming implementation of XML parser requires thousands of lines of code.

Obvious things are often only obvious in hindsight.

How I implemented Oembed Proxy for GitHub

2018-10-13T00:00:00Z

Why Oembed Proxy for GitHub

I'm writing a programming book Essential Go in Notion and I need to include code snippets.

Notion has support for code blocks but it's not good enough for my use case.

I want to make sure the code compiles so I write small programs and store them in GitHub repository. My custom book building script compiles and runs the programs to ensure the code is correct.

Notion supports embedding GitHub's gists but not files from git repositories hosted on GitHub.

I researched things and turns out there's a standard called Oembed that was created to enable embedding arbitrary content from one website in another.

Notion supports Oembed.

I didn't find existing service that can provide Oembed support for GitHub repositories so I built one myself.

This article describes the high-level design of Oembed Proxy for GitHub.

What is Oembed?

Let's say you're implementing a rich-text editor on the web and you want to allow embedding arbitrary content from other services: a tweet, a youtube video, a flickr photo.

You can add code to support each service you know about (and pray that they provide a stable way to get the necessary information) but it's not scalable. There are way too many web services out there and more are created every day.

Some people noticed the problem and created Oembed protocol that provides a standard way to expose content for embedding. Now you need to only write code to support Oembed.

Here's how it works, using Oembed Proxy for GitHub as an example.

In our example Notion is an Oembed client that wants to embed a file from GitHub repository https://github.com/essentialbooks/books/blob/master/README.md in the body of a document.

Notion supports Oembed standard. If GitHub supported Oembed on their servers, you would add embed block and use https://github.com/essentialbooks/books/blob/master/README.md link directly.

GitHub doesn't support Oembed so instead, you can use URL via my Oembed Proxy: https://www.onlinetool.io/gitoembed/widget?url=https%3A%2F%2Fgithub.com%2Fessentialbooks%2Fbooks%2Fblob%2Fmaster%2FREADME.md

This is https://www.onlinetool.io/gitoembed/widget page with GitHub link provided as url argument.

If you view that page it's the file from GitHub with source code highlighting.

Oembed supports auto-discovery. If you peek at HTML of that page, you'll see this is <head> section:

<link rel="alternate" type="application/json+oembed" href="https://www.onlinetool.io/gitoembed/oembed?format=json&url=https://github.com/essentialbooks/books/blob/master/README.md" title="README.md" />
<link rel="alternate" type="text/xml+oembed" href="https://www.onlinetool.io/gitoembed/oembed?format=xml&url=https://github.com/essentialbooks/books/blob/master/README.md" title="README.md" />

Those are instructions telling Oembed client (Notion in this example) how to get embeddable HTML.

Oembed supports 2 formats for providing this information: JSON and XML. In my testing Notion worked with just application/json+oembed but I implemented both just in case other clients only understand XML.

Oembed client parses HTML to extract those links and, if present, gets Oembed information. In our example it's in https://www.onlinetool.io/gitoembed/oembed?format=json&url=https://github.com/essentialbooks/books/blob/master/README.md and looks like this:

{
	"version": 1,
	"type": "rich",
	"provider_name": "gitoembed",
	"provider_url": "https://www.onlinetool.io/gitoembed/",
	"height": 320,
	"width": 720,
	"title": "README.md",
	"html": "\u003ciframe width=\"100%\" height=320 src=\"https://www.onlinetool.io/gitoembed/widget?url=https://github.com/essentialbooks/books/blob/master/README.md\" frameborder=\"0\" onload=\"resizeFrame(this);\"\u003e\u003c/iframe\u003e"
}

I hope the format is mostly self-explanatory.

The interesting bit is html field, which is:

<iframe 
	width="100%"
	height=320 
	src="https://www.onlinetool.io/gitoembed/widget?url=https://github.com/essentialbooks/books/blob/master/README.md" 
	frameborder="0"
	onload="resizeFrame(this);">
</iframe>

We could send the actual HTML content to insert but it's more customary to send an iframe which loads the html.

In my implementation src of the iframe is the same page from which we extracted Oembed JSON link so it serves double-duty as both the content and an Oembed pointer to the content. Those could be different URLs.

Implementation details of Oembed Proxy for GitHub

First I needed a server. Usually, I use Digital Ocean but this time I went for biggest bang for the buck and used C2L server from Scaleway.

For ~$30 I get 8 core server with 32 GB of RAM, 250 GB SSD drive and 600 MBits/s unmetered bandwidth. It's a bare metal server, not a VPS, so it's all mine, eliminating risk of noisy neighbors.

On Digital Ocean the closest server with such specs would be $160. I keep a list of cheap VPS servers for comparison.

The downside is that the servers are in Europe (you can choose between Amsterdam or Paris) so the latency for users in US will be higher than if the server was hosted in US.

For the OS I went with Ubuntu 18.04. I know it best and it's one of the most popular distros.

The server is written in Go. It's my go-to language for writing backend code.

The service isn't very complicated:

it downloads the file via GitHub's https://raw.githubusercontent.com url
to avoid over-loading GitHub servers (and exceeding their throttling limits) I cache downloaded files for a day. It's not infinitely long cache because files on GitHub can change and I don't want to cache outdated version forever
for code highlighting I use chroma library

There's even less of front-end code. Explanation of the service and a way to test it implemented with a form and few lines of JavaScript.

Powering a blog with Notion and Netlify

2018-07-30T00:00:00Z

The last iteration of this blog was a Go program running on Digital Ocean's cheapest VM ($5/month).

Recently I've made 2 big changes:

I converted it to a static site hosted on Netlify
I used Notion for writing the posts instead of writing markdown files in a text editor

Moving to Netlify

My blog was effectively a static website. It didn't need a backend so writing a custom server and running it on a VPS was overkill.

Few months back Netlify reduced the price of their cheapest plan to $0 (from $10/month).

I'm always looking to simplify and cheapify my life so I bit the bullet and converted my custom server to generate static HTML files suitable for hosting on Netlify.

If you want to publish a static website and are starting from scratch, the best approach is to use one of the many static site generators (e.g. Hugo or Jekyll).

I already had a lot of content in .html and markdown files accumulated over the years and code to generate the website for serving from custom web server so I refactored to code to generate static HTML instead.

Refactoring process was time consuming but simple.

For dynamically generated html I changed the code to generate a .html file and setup appropriate url => file mapping using _redirect file.

For local testing I use Caddy and generate Caddyfile with appropriate redirects. There are minor differences when testing locally because there are semantic differences between redirect capabilities of Netlify and Caddy but it's good enough.

Netlify also has an option to do a draft deploy under a unique URL. This is good for previewing the changes before publishing.

At the end of code refactoring I effectively ended up with a custom static site generator.

The verdict

I'm happy with the result.

Netlify has all the features I care about for a basic website.

Notably they provide free SSL with Let's Encrypt, allow custom domains and have capable redirect capabilities. They use CDN so should be faster than hosting on a single host.

The only thing I miss is being able to see analytics for 404. With my own server I was logging all requests for pages that don't exist on my server. Many of them were bots trying to hack me via known vulnerabilities in popular software like WordPress but sometimes it would be caused by my mistake or a request for a valid article incorrectly linked. Seeing those I was able to fix most of them by adding redirects.

I hope Netlify can sustain their generous free plan. Luckily there are plenty of options to host a static website so even if Netlify goes under, it'll be easy to move somewhere else, like Firebase Hosting, surge.sh, GitHub pages, GitLab pages and many others.

Using Notion as Content Management System

The easier it is to write, the more I write.

Using markdown files fails the "as easy as possible" part.

In absolute terms, creating a new markdown file doesn't take much time.

In practice it's enough friction to deter me from writing. In my worst year I only wrote 1 new blog post.

In a perfect world I would open an app and start writing. When I'm done writing I would publish with a click of a button.

Enter Notion

Notion is as close to a perfect writing tool as it gets: open a page and start writing.

The problem is: all that content is trapped in Notion. You can publish (publicly share) a page but I want more flexibility:

I want to host on my own domain; my website is partly a tool for marketing myself
I want integration with Google Analytics
I want a custom design
I want to provide rss feed

Thankfully I'm a programmer: if something can be done with software, I can do it.

I started a one-man Notion Liberation Front. My goal: liberate content trapped inside Notion.

I reverse-engineered their API, wrote a Go library and after another round of code refactoring I had my blog powered by Notion thanks to this Go program.

I imported my old blog posts into Notion. They have a decent markdown importer although I did have to do some cleaning up.

I went even further and published most of my notes to my website as well. I still have many private notes in Notion but most of them can be just as well be publicly visible.

There's no way I could do publish so much content without Notion.

I also automated publishing by using cron functionality in Travis CI. Every day a script downloads latest content from Notion, caches it in git repository and re-generates a website.

Everything now runs on auto-pilot. I can just write new articles in Notion and my website will be automatically updated every day.

How I reverse engineered Notion API

2018-07-23T00:00:00Z

Notion is a great tool for writing but the content is trapped inside the web app.

The company is working on an official API but I'm impatient.

This article describes how I reverse engineered their API and created a Go library notionapi.

It all began with a failure.

My first attempt at extracting notion content was traditional web scraping.

I found a Python script that uses Selenium to recursively spider a Notion page and publish it to Firebase Hosting.

I ported it to Node to use Puppeteer (better technology than Selenium).

While it worked this approach is limited to getting a verbatim HTML of the pages as they are rendered by the Notion application.

I wanted to be able to change the look of the page, add elements like footers and headers and navigation bar.

I briefly considered trying to reconstruct the structure of the page from rendered HTML but at best that would be a lot of ugly guesswork.

The lightbulb moment

Modern Single Page Applications (SPA) work by getting data from the server in structured format (most often JSON) and rendering HTML in the browser with JavaScript.

A trip to Chrome Dev Tools confirmed that Notion works like that.

When loading a Notion page I saw XHR requests like /api/v3/getRecordValues and /api/v3/loadPageChunk.

Lucky for me the API is not obfuscated. It returns responses as JSON data. It isn't hard to figure out the meaning of fields.

Working with the original JSON structure is much easier that trying to reconstruct it from rendered HTML.

Building tools

I could have looked at API requests between client and server in Chrome dev tools but it's not the best workflow.

Instead I wrote node.js script that logs all XHR requests that web browser makes when rendering a given page.

That has several advantages over using dev tools:

I could filter out requests to third-party services like amplitude, fullstory and intercom
I could filter out requests that are not interesting like /api/v3/ping
I could pretty-print JSON
I could write captured traffic to a file for further analysis

Here's the script:

The big picture analysis

After looking at captured data, the structure of Notion content is not complicated.

Everything, including a top-level page, is a block.

Blocks are identified by a unique id which looks like a standard UUID format.

Blocks are arranged into a tree i.e. some blocks have children.

Blocks have metadata, like creation time, last edit time, version etc.

There are different kinds of blocks: a page, text, todo item, list item etc.

Some blocks have properties specific to that block type. For example a page block has title property.

To get the content of a page we start with its UUID which we can find out because it's last part of the URL of the page.

We can issue /api/v3/getRecordValues API to get list of blocks in the page and then /api/v3/loadPageChunk to get content of those blocks.

Majority of work was figuring out what kinds of blocks there are, how are they represented in JSON and writing code to to retrieve the data and present it in a format that is easier to work with than the raw data returned by the server.

Testing different kinds of blocks

Notion page consist of different kinds of blocks and we need to know how each block is represented in JSON response.

To investigate it systematically, I've created a test page for each kind of block and used the request logging script to look at JSON returned by the server for that block.

Writing Go library

Next step was writing a Go library.

I captured sample JSON responses from getRecordValues and loadPageChunk and used Quicktype to generate Go structures.

I had to tweak them a bit to accommodate variations in JSON structure.

The rest of the effort was writing a helper function that abstracts the details of HTTP requests and returns an easy to use struct describing a notion page.

There result of that work is notionapi Go package.

Using the library in practice

This was not just an academic exercise.

This blog was powered by markdown files I stored in GitHub repository.

My goal was to move the content to Notion, so that I can edit it more easily, convert it to HTML and publish as my website/blog.

You can see the code here.

The high-level structure of the code:

I use my Go notionapi library to download the content from Notion
I cache downloaded data and store them in git repository. This is to make sure I have a copy of data even if Notion disappears, to make it faster to tweak publishing code (no need to re-download) and to be nicer to Notion server (no re-downloads unless I absolutely have to)
I convert Notion data to HTML, wrap it in templates for my pages and write HTML files to disk
I deploy to Netlify

Advanced web spidering with Puppeteer

2018-07-18T00:00:00Z

Puppeteer is a node.js library that makes it easy to do advanced web scraping and spidering.

Older generation of web scraping and spidering tools would grab and analyze HTML pages as returned by a web server.

It doesn't work well anymore because less and less website are static HTML pages. Today websites are often applications written in JavaScript that generate HTML on the client, not the server.

To get the final HTML output your scraper needs to run that JavaScript.

That used to be very difficult but Puppeteer makes it easy.

Puppeteer uses Chrome to run web application and uses CDP (Chrome DevTools Protocol) to access the webpage.

This article describes some more advanced techniques but let's start with basic example first.

Save web page to a file

First install the library:

yarn add puppeteer when using yarn
npm --save puppeteer when using npm

This is the simplest possible usage of Puppeteer:

navigate to a page of interest
get content of the webpage as HTML and save it to a file

const puppeteer = require("puppeteer");
const fs = require("fs");

async function run() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
	await page.goto("https://www.google.com/", { waitUntil: "networkidle2" });
	// hacky defensive move but I don't know a better way:
	// wait a bit so that the browser finishes executing JavaScript
	await page.waitFor(1 * 1000);
	const html = await page.content();
	fs.writeFileSync("index.html", html);
	await browser.close();
}

run();

Handling failures

What if a url you tried to load didn't exist?

The web server will return the 'Not Found' page with HTTP status code 404 in the response. The above script would treat such page as a perfectly valid response.

Most times you want to handle this as an error case.

For example, if you're writing a bot that checks for broken links, you want to distinguish 404 NotFound response from 200 Ok response.

In HTTP protocol status codes 4xx and 5xx indicate errors. 2xx indicate success and 3xx indicate successful redirection.

Puppeteer provides Page.setRequestInterception(true) hook for intercepting HTTP requests before they happen as well as inspecting completed HTTP responses.

Here's a program that prints information about all HTTP requests and responses:

const puppeteer = require("puppeteer");

async function run() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
	const mainUrl = "https://blog.kowalczyk.info/pas"
	let mainUrlStatus;
  await page.setRequestInterception(true);
  page.on("request", request => {
    const url = request.url();
    console.log("request url:", url);
    request.continue();
  });
  page.on("requestfailed", request => {
    const url = request.url();
    console.log("request failed url:", url);
  });
  page.on("response", response => {
    const request = response.request();
    const url = request.url();
    const status = response.status();
    console.log("response url:", url, "status:", status);
		if (url === mainUrl) {
			mainUrlStatus = status;
		}
  });
  await page.goto(mainUrl);
	console.log("status for main url:", mainUrlStatus);
  const html = await page.content();
  await browser.close();
}

run();

Here's what it'll print:

$ node test.js
request url: https://blog.kowalczyk.info/pas
response url: https://blog.kowalczyk.info/pas status: 404
request url: https://fonts.googleapis.com/css?family=Roboto:400,700&subset=latin,latin-ext
response url: https://fonts.googleapis.com/css?family=Roboto:400,700&subset=latin,latin-ext status: 200
request url: https://fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmWUlfChc9AMP6lQ.ttf
request url: https://fonts.gstatic.com/s/roboto/v18/KFOmCnqEu92Fr1Mu7GxPKTU1Kg.ttf
response url: https://fonts.gstatic.com/s/roboto/v18/KFOmCnqEu92Fr1Mu7GxPKTU1Kg.ttf status: 200
response url: https://fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmWUlfChc9AMP6lQ.ttf status: 200
status for main url: 404

Notice that fetching a page also fetches all resources used by that page, just like in a web browser. For that reason to find out status code for the url we requested, we have to remember it in a variable in response hook.

requestfailed hook is for errors on network connection level e.g. DNS resolution failed, there's not network at all, network connection got interrupted etc.

See `console.log` from inside the browser

Your JavaScript code is executed in two different contexts:

main script is executed in node.js. In that context console.log("foo") prints to shell
scripts provided to Page.evaluate method are serialized to text, sent to the browser via Chrome DevTools Protocol and executed inside the browser. In that context console.log("foo") prints to browser console, which you can't see.

To see what console.log prints in the browser, you can hook it and re-log to shell:

const browser = await puppeteer.launch();
const page = await browser.newPage();
const url = "https://blog.kowalczyk.info/";

// this hooks `console.log()` in the browser
page.on("console", msg => {
  console.log("The whole message:", msg.text());
  console.log("\nEach argument:");
  for (let arg of msg.args()) {
    // arg is a Promise returning value of type JSHandle
    // https://pptr.dev/#?product=Puppeteer&show=api-class-jshandle
    arg.jsonValue().then(v => {
      console.log(v);
    });
  }
});
await page.goto(url);
await page.evaluate(() => {
  // This is executed inside the browser so not visible in our script
  // unless we hook 'console' events
  console.log("Message from the browser", 5);
});
await browser.close();

Quickly testing `evaluate` scripts

It's slow to test browser script executed via Page.evaluate because you have to start the browser, load the page etc.

To test scripts faster I test them directly in the browser, using excellent Chrome dev tools.

My process is:

prepare the script, in IIFE form, in the editor
copy & paste in console window in Chrome dev tools

What is IIFE form? To avoid conflicts with JavaScripts state from previous runs I wrap the code inside Immediately Invoked Function Expression:

function() {
  // code here is isolated from things outside this function
  console.log("My script");
  // ... my script

	// when debugging I can trigger JavaScript debugger from inside the script
	// with debugger statement:
	debugger;
}() // immediately invoke the function

It's faster to iterate on code this way. You can also use browser's JavaScript debugger.

As shown in the snippet, I can also trigger the debugger for single-stepping through the code with debugger; statement.

Study Puppeteer API

Now that you've seen a few advanced uses of Puppeteer, you should study its API a bit to learn what else is possible. CDP is very powerful:

Page class allows hooking many events, reading and setting cookies, simulating interaction like mouse clicks etc.
Tracing class allows creating a trace file for future inspection in Chrome DevTools
Worker class allows interacting with Web Workers
Coverage class allows measuring JavaScript and CSS coverage
Keyboard class allows simulating keyboard events

Other CDP tools and libraries

Puppeteer is not the only tool that takes advantage of Chrome DevTools protocol. A bunch of them is listed in Awesome Chrome DevTools.

Fuzzing Markdown parser written in Go

2018-01-31T15:32:55-08:00

I finished working on Markdown parser for Go.

To make sure it’s robust (no crashes, no hangs) I decided to fuzz it.

What is fuzzing?

Parsing text or binary formats is tricky. It’s easy to make mistakes that can lead to security exploits.

Fuzzing helps to combat this.

The idea is simple:

magic algorithm generates randomized input data
we feed this data to parsing code
code instrumentation detects if parsing code crashes or overwrites memory

Fuzzing in Go is simple

Writing a fuzzer for C or C++ code is hard.

It’s trivial for Go thanks to go-fuzz.

It took me about an hour to write a fuzzer for my markdown parser.

Fuzzing step-by-step

First we need to get the tools:

$ go get github.com/dvyukov/go-fuzz/go-fuzz
$ go get github.com/dvyukov/go-fuzz/go-fuzz-build

My library has ast := markdown.Parse(markdown []byte, p *parser.Parser) function which takes markdown text as input and parses it into an abstract syntax tree.

I wrote fuzz.go:

// +build gofuzz

package markdown

// Fuzz function to be used by https://github.com/dvyukov/go-fuzz
func Fuzz(data []byte) int {
    Parse(data, nil)
    return 0
}

This implements Fuzz(data []byte) function, which is an API expected by go-fuzz.

Notice // +build gofuzz line. It means that this code will not be compiled by default but only with gofuzz build tag.

Having the fuzzer generate completely random data is better than nothing but it’s much better if fuzzer can be seeded with sample valid content.

As it happens, I already had markdown files used for tests in testdata directory.

I used those to initialize the fuzzer:

# create a working directory for the fuzzer
$ mkdir -p fuzz-workdir

# copy the files that seed fuzzing to corpus sub-directory of working directory
$ mkdir -p fuzz-workdir/corpus
$ cp testdata/*.text fuzz-workdir/corpus

# generate the fuzzing program. This compiles fuzz.go we wrote earlier
# generates fuzzer executable and markdown-fuzz.zip that packages
# data to drive fuzzing process
$ go-fuzz-build github.com/gomarkdown/markdown

The last go-fuzz-build step can take a while.

Finally we start fuzzing process:

$ go-fuzz -bin=./markdown-fuzz.zip -workdir=fuzz-workdir

This runs for as long as you let it. Fuzzing works by generating random input data so it can go forever.

As long as you don’t delete working directory, you can stop and re-start go-fuzz and it’ll pick up where it left off.

Found crashes will be logged in fuzz-workdir/crashers directory.

I’m a fan of automation so I wrote fuzz.sh script.

That way I can re-run it easily after making changes to the parser to verify I didn’t introduce bugs.

The fuzzer has advanced options like fuzzing using multiple machines.

The results

Fuzzing is magic. My library is derived from very popular and widely used blackfriday library (which in turn was derived from a popular C library) and yet the fuzzer found 3 separate issues that either crashed the parser or caused infinite loop.

They are now fixed and covered by unit tests.

57 MicroConf videos for self-funded software businesses

2017-12-24T20:09:15-08:00

MicroConf is a conference for small/indie/self-funded software businesses. Many of their talks are available on Vimeo but not well indexed. They have a better index (and another here) on their website, but also not great.

This is a list of videos and a bit of info about each video. I hope this will help you find a video useful for you.

1. Lizards Thru Doorways: Proven ways to Widen Your Funnel Using Just Your CTAs

Video https://vimeo.com/132932415 by Joanna Wiebe, 39 min.

Joanna is a copywriter for hire i.e. she writes the text of emails, sales pages etc. for other businesses.

Her talk is about writing better copy. Tips and case studies. She also has a lot of free (and paid) tutorials on copywriting on her website https://copyhackers.com/. Recap: https://kaidavis.com/microconf-2015/joanna/

2. An Inside Story of Self-Funded SaaS Growth

Video https://vimeo.com/132932414 by Rob Walling, 51 min.

This is a history of Rob’s SaaS startup https://www.drip.com(email marketing software). He also wrote about it online: https://wpcurve.com/bootstrapped-drip-into-a-7-figure-saas-business/.

Recap: https://kaidavis.com/microconf-2015/rob/

3. Accelerating Growth: Grow Faster Without Working Yourself to Death

Video https://vimeo.com/132139313 by Hiten Shah, 54 min

Recap: https://kaidavis.com/microconf-2015/hiten/

4. Amplification - Content Marketing That Works

Video https://vimeo.com/132139312 by Justin Jackson, 11 min.

Content marketing is about writing blog posts, doing podcasts and videos to attract people to your website so that you can pitch them on your products.

5. How to Aggressively Acquire Customers for your SaaS with an Efficient Outbound Sales Process

Video https://vimeo.com/132139308 by Jordan Gal, 12 min.

Outbound sales process is using email and phone calls (cold-calling) to find customers.

6. How I Designed Our (High-Touch) Sales & Onboarding to Run Without Me (2014)

Video https://vimeo.com/132139307 by Brian Casel, 11 min.

Based on experience from his startup Restaurant Engine, talks about converting leads (from his inbound traffic) into customers and how to automate it.

Recap: https://www.phraseexpander.com/microconf-2014/brian-casel-automated-content-marketing-machine-microconf-2014/

7. Creating an Explosive Email Course

Video https://vimeo.com/132139306 by David Kadavy, 13 min.

Talks about creating email course in order to get more leads (build email list).

8. Do This, Not That: Creating an Exceptional Customer Support Experience from Day 1 (2015)

Video https://vimeo.com/131466750 by Sarah Hattter, 29 min.

Sarah runs CoSupport, which teaches companies how to create good customer support.

Recap: https://kaidavis.com/microconf-2015/sarah/

9. How to Build a Solo SaaS Sales Machine (2015)

Video https://vimeo.com/131441010 by Steli Efti, 56 min.

Recap: https://kaidavis.com/microconf-2015/steli/

10. Lessons Learned Building a WordPress Plugin Business to $10k/month

Video https://vimeo.com/130984501 by Phil Derksen, 15 min.

11. How To Systematically Fight SaaS Churn And Win (2015)

Video https://vimeo.com/130984497 by Robert Graham, 13 min.

Recap: https://bootstrapping.io/microconf-2015/robert/

12. How Bookkeeping Tripled My Revenue in Two Years (and Other Unexpected Cash Flow Advice) (2015)

Video https://vimeo.com/130984492 by Jesse Mecham, 48 min.

Recap: https://kaidavis.com/microconf-2015/jesse/

13. Growing Your Userbase with Better Onboarding (2015)

Video https://vimeo.com/130797721 by Samuel Hulick, 30 min.

Recap: https://kaidavis.com/microconf-2015/samuel/

14. Q&A and Smart Bear Live (2015)

Video https://vimeo.com/130797720 by Jason Cohen, 1hr 7 min.

Recap: https://kaidavis.com/microconf-2015/jason/

15. Micro-ISV to Micro-acquisition: Selling my 11-year one-man software business (2015)

Video https://vimeo.com/130797718 by Jacob Thurman. 11 min.

Recap: https://kaidavis.com/microconf-2015/jacob/

16. How to start a SaaS business in any market with no idea or connections, using only excel, email & phone (2015)

Video https://vimeo.com/130797716 by Pawel Brzeminski, 12 min.

Recap: https://kaidavis.com/microconf-2015/pawel/

17. How I Grew My Productized Consulting Offering To $100K YRR In 12 Months (2015)

Video https://vimeo.com/130797714 by Einer Vollset, 9 min.

Recap: https://kaidavis.com/microconf-2015/einar/. A story of building http://www.appaftercare.com/

18. The 3 Week Startup (2015)

Video https://vimeo.com/130499701 by Keith Perhac, 10 min.

Recap: https://kaidavis.com/microconf-2015/keith/

Tactical tips about how (and why) to build SaaS quickly (in 1 week).

19. Leveling Up (2015)

Video https://vimeo.com/129913527 by Patrick McKenzie, 1 hr.

Recap: https://kaidavis.com/microconf-2015/patio11/

20. How to Validate Your Idea and Launch to $7k in Recurring Revenue (2014)

Video https://vimeo.com/96267945 by Rob Walling, 58 min.

Recap: http://www.christophengelhardt.com/rob-walling-validate-idea-launch-7k-recurring-revenue-microconf-europe-2014/

Recap: https://www.phraseexpander.com/microconf-2014/rob-walling-validate-idea-launch7k-microconf-2014/

21. 6 Tricks That Helped Me Triple My SaaS’ Growth Rate (2014)

Video https://vimeo.com/95680318 by Brennan Dunn, 41 min.

Recap: http://www.christophengelhardt.com/brennan-dunn-6-tricks-helped-triple-saas-growth-rate-microconf-europe-2014/

Recap: https://www.phraseexpander.com/microconf-2014/brennan-dunn-triple-saas-growth-rate-microconf-2014/

22. Lifting the Veil: The Data Behind Successful Product Launches (2014)

Video https://vimeo.com/95680316 by Ryan Delk, 13 min.

Recap: https://www.phraseexpander.com/microconf-2014/ryan-delk-data-behind-successful-product-launches-microconf-2014/

23. 3 Habits for Building (and Growing) a Product Empire (2014)

Video https://vimeo.com/95680313 by Nathan Barry, 38 min.

Recap: https://www.phraseexpander.com/microconf-2014/nathan-barry-3-habits-grow-product-empire-microconf-2014/

24. From Zero to $4M/year Without Quora, Hacker News, or Mixergy (2014)

Video https://vimeo.com/95653848 by Jesse Mecham, 50 min.

Recap: https://www.phraseexpander.com/microconf-2014/jesse-mecham-from-zero-4m-microconf-2014/

25. 10 Business Questions Every Entrepreneur Needs to Ask Their Analytics (And Where to Find the Answers) (2014)

Video https://vimeo.com/95653743 by Annie Cushing, 46 min.

Recap: https://www.phraseexpander.com/microconf-2014/annie-cushing-10-business-questions-analytics-microconf-2014/

26. How To Slay the Customer Support Beast (2014)

Video https://vimeo.com/95052087 by Ian Landsman, 44 min.

Recap: https://www.phraseexpander.com/microconf-2014/ian-landsman-slay-customer-support-beast-microconf-2014/

27. Don’t Burn-up in the Launch: Staying Emotionally and Relationally Healthy While Launching Your Startup (2013)

Video https://vimeo.com/72211933 by Sherry Walling, 19 min.

Recap: http://www.christophengelhardt.com/sherry-walling-dont-burn-up-in-the-launch-microconf-2013/

Recap: http://www.christophengelhardt.com/sherry-walling-dont-burn-up-in-the-launch-staying-emotionally-and-relationally-healthy-while-launching-your-startup-microconf-europe-2013/

28. Playing the Long Game: Making Entrepreneurship a Sustainable Life (2014)

Video https://vimeo.com/95052086 by Sherry Walling, 56 min.

Recap: https://www.phraseexpander.com/microconf-2014/sherry-walling-entrepreneurship-sustainable-life-microconf-2014/

29. From Idea to $5k/mo in 5 Months (2014)

Video https://vimeo.com/94623532 by Josh Pigford, 12 min.

History of building https://baremetrics.com/

Recap: https://www.phraseexpander.com/microconf-2014/josh-pigford-idea-5k-5months-microconf-2014/

30. UX Basics That Convert Users into Customers (2014)

Video https://vimeo.com/94623531 by Samuel Hulick, 7 min.

Recap: https://www.phraseexpander.com/microconf-2014/samuel-hulick-uxbasics-convert-user-into-customers-microconf-2014/

31. Business Hacks & Epic Wins (2014)

Video https://vimeo.com/94623529 by Mike Taber, 1 hr 4 min.

Recap: https://www.phraseexpander.com/microconf-2014/mike-taber-business-hacks-epic-wins-microconf-2014/

32. How to Grow Your Self-Funded Business Faster (2014)

Video https://vimeo.com/94187473 by Hiten Shah, 43 min.

Recap: https://www.phraseexpander.com/microconf-2014/hiten-shah-grow-your-self-funded-business-faster-microconf-2014/

33. Designing the Ideal Bootstrapped Business (2013)

Video https://vimeo.com/74338272 by Jason Cohen, 1hr 5 min.

Recap: http://www.christophengelhardt.com/jason-cohen-microconf-2013/

34. Dude. Marketing is not your thing (2013)

Video https://vimeo.com/72461554 by Jody Burgess, 13 min.

Recap: http://www.christophengelhardt.com/dude-marketing-is-not-your-thing-microconf-2013/

35. How a Non-Technical Founder Built a 6 Figure Saas App Using Only Free Public Data Sources (2013)

Video https://vimeo.com/72461551 by Brecht Palombo, 15 min.

Recap: http://www.christophengelhardt.com/brecht-palombo-how-a-non-technical-founder-built-a-6-figure-saas-app-using-only-free-public-data-sources-microconf-2013/

36. Finding Customers Who Are 100x More Valuable Without 100x the Effort (2013)

Video https://vimeo.com/72456666 by Erica Douglass, 48 min.

Recap: http://www.christophengelhardt.com/erica-douglass-how-to-measurably-move-the-needle-with-your-software-company-microconf-2013/

37. Bootstrapping an App Business (2013)

Video https://vimeo.com/72260021 by Patrick Thompson, 16 min.

Recap: http://www.christophengelhardt.com/patrick-thompson-bootstraping-an-app-business-microconf-2013/

38. Building Things To Help Sell The Things You Build (2013)

Video https://vimeo.com/72140534 by Patrick McKenzie, 1 hr 13 min.

Recap: http://www.christophengelhardt.com/patrick-mckenzie-building-things-to-help-sell-the-things-you-build-microconf-2013/

39. Killer Content Marketing (2013)

Video https://vimeo.com/72081980 by Hiten Shah, 44 min.

Recap: http://www.christophengelhardt.com/hiten-shah-killer-content-marketing-microconf-2013/

40. SEO Demystified: Practical Techniques That Produce Astonishing Results (2013)

Video https://vimeo.com/71547333 by Dave Collins, 57 min.

Recap: http://www.christophengelhardt.com/dave-collins-seo-demystified-practical-techniques-that-produce-astonishing-results-microconf-europe-2013/

41. Shut Up and Take My Money: How to Find Business Ideas Customers Want (2013)

Video https://vimeo.com/71250239 by Josh Kaufman, 42 min.

Recap: http://www.christophengelhardt.com/josh-kaufman-shut-up-and-take-my-money-how-to-find-business-ideas-customers-want-microconf-2013/

42. Copywriting that Converts: How to Sell Without Selling Your Soul (2013)

Video https://vimeo.com/71008640 by Joanna Wiebe, 50 min.

Recap: http://www.christophengelhardt.com/joanna-wiebe-copywriting-that-converts-microconf-2013/

Another: http://www.workhappy.net/2013/05/how-you-can-drastically-improve-the-copy-on-your-site-even-if-you-only-have-5-minutes.html

43. Lean Analytics: How to Focus on What Matters (2013)

Video https://vimeo.com/70981922 by Ben Yoskovitz, 43 min.

Recap: http://www.christophengelhardt.com/ben-yoskovitz-measure-what-matters-microconf-2013/

44. How to Sell Anything to Anyone (2013)

Video https://vimeo.com/70901902 by Mike Taber, 41 min.

Recap: http://www.christophengelhardt.com/mike-taber-microconf-2013-a-k-a-the-liquor-fairy/

45. How to 10x in 15 months (2013)

Video https://vimeo.com/70901901 by Rob Walling, 51 min.

Recap: http://www.christophengelhardt.com/rob-walling-how-to-10x-in-15-months-microconf-2013/

46. Cheap and Easy Customer Support (2012)

Video https://vimeo.com/51306662 by Sarah Hattter, 40 min.

47. Google AdWords: Stop Losing & Start Exploiting (Really) (2012)

Video https://vimeo.com/51187193 by Dave Collins, 58 min.

48. How I Bootstrapped and Sold My Software Company By Maxing Out My Credit Cards (2012)

Video https://vimeo.com/50372726 by Bill Bither, 46 min.

49. From Idea to 7 Figures in 2 years: The Story of Woothemes

Video https://vimeo.com/50209990 by Adii Pienaar, 45 min.

50. Ask Me Anything (2012)

Video https://vimeo.com/49023444 by Peldi Guilizzoni, 48 min.

51. If You Don’t Like Drunk Frat Boys, Don’t Open an Irish Pub… (2012)

Video https://vimeo.com/48962410 by Amy Hoy, 43 min.

Transcript: https://stackingthebricks.com/a-customer-is-your-mvp-a-video-talk-on-making-products-that-sell/

52. Growth Hacking (2012)

Video https://vimeo.com/48592609 by Dan Martell, 39 min.

53. Losers Have Goals, Winners Have Systems (2012)

Video https://vimeo.com/48571431 by Mike Taber, 27 min.

54. Naked Business: How Honesty Makes You More Money (2012)

Video https://vimeo.com/48549019 by Jason Cohen, 55 min.

55. Finding Your Flywheel (2012)

Video https://vimeo.com/47465229 by Rob Walling, 1 hr 04 min.

56. How to Engineer Marketing Success (2012)

https://vimeo.com/47311461 by Patrick McKenzie, 51 min.

57. More Lessons I’ve Learned as a Serial Entrepreneur (2012)

https://vimeo.com/46893380 by Hiten Shah, 48 min.

Related resources:

How to install latest clang (6.0) on Ubuntu 16.04 (xenial) / WSL

2017-11-20T20:37:02-08:00

This article describes installing latest clang (llvm) on Ubuntu 16.04 (Xenial), which is also the default distro for Windows Subsystem for Linux (WSL).

Ubuntu 16.04 ships with relatively old clang. Running sudo apt-get install clang installs version 3.8.

To install latest version (currently 6.0) run:

wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main"
sudo apt-get update
sudo apt-get install -y clang-6.0

clang-6.0 is the name of the executable (and so is lldb-6.0 etc.).

How it works

Let’s deconstruct those commands so that you know what’s happening.

How does apt know what packages are available?

Apt queries package servers to get a list of available deb packages. Default Ubuntu installation knows about official Ubuntu servers but you can run your own server to provide additional packages.

List of servers is in /etc/apt/sources.list. Here’s how it looks by default on Ubuntu 16.04:

$ cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ xenial-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu/ xenial-security main restricted universe multiverse

Ubuntu only provides a relatively old clang 3.8. Luckily, Apple creates deb packages and maintains a server for all llvm/clang releases and most Ubuntu distros.

sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main" adds llvm’s server for Ubuntu 16.04 to /etc/apt/sources.list.

sudo apt-get update downloads the latest list of packages from all servers, including the one we just added

For security, packages are signed with private keys. You need public key to verify package signature.

wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - downloads llvm’s server public key.

sudo apt-get -y install clang-6.0 installs newly available package clang-6.0.

Flag -y disables confirmation prompt.

What if there’s a newer version of clang or a different version of Ubuntu?

There is a new llvm/clang release every 6 months. What to do for newer version?

Visit https://apt.llvm.org/ and locate the equivalent of deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main for desired combo of clang/Ubuntu and correspondingly update apt-add-repository ... line in the above instructions.

Making latest version available as clang

This version is installed side-by-side with the default clang 3.8. To run it, you have to explicitly say clang-6.0. clang will either refer to 3.8 (if you’ve installed it) or nothing at all.

You can reconfigure the system so that clang refers to clang 6:

update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.8 100
update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 1000
update-alternatives --install /usr/bin/clang++ clang /usr/bin/clang-3.8 100
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.8 100
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-6.0 1000
update-alternatives --config clang
update-alternatives --config clang++

This might help if you have scripts that hard-code name of the compiler and don’t allow over-writing it.

Guide to predefined macros in C++ compilers (gcc, clang, msvc etc.)

2017-11-07T17:07:45-08:00

When writing portable C++ code you need to write conditional code that depends on compiler used or the OS for which the code is written.

Here’s a typical case:

#if defined (_MSC_VER)
// code specific to Visual Studio compiler
#endif

To perform those checks you need to check pre-processor macros that various compilers set.

It can either be binary is defined vs. is not defined check (e.g. __APPLE__) or checking a value of the macro (e.g. _MSC_VER defines version of Visual Studio compiler).

This document describes macros set by various compilers.

Checking for OS (platform)

To check for which OS the code is compiled:

Linux and Linux-derived           __linux__
Android                           __ANDROID__ (implies __linux__)
Linux (non-Android)               __linux__ && !__ANDROID__
Darwin (Mac OS X and iOS)         __APPLE__
Akaros (http://akaros.org)        __ros__
Windows                           _WIN32
Windows 64 bit                    _WIN64 (implies _WIN32)
NaCL                              __native_client__
AsmJS                             __asmjs__
Fuschia                           __Fuchsia__

Checking the compiler:

To check which compiler is used:

Visual Studio       _MSC_VER
gcc                 __GNUC__
clang               __clang__
emscripten          __EMSCRIPTEN__ (for asm.js and webassembly)
MinGW 32            __MINGW32__
MinGW-w64 32bit     __MINGW32__
MinGW-w64 64bit     __MINGW64__

Checking compiler version

gcc

__GNUC__ (e.g. 5) and __GNUC_MINOR__ (e.g. 1).

To check that this is gcc compiler version 5.1 or greater:

#if defined(__GNUC__) && (__GNUC___ > 5 || (__GNUC__ == 5 && __GNUC_MINOR__ >= 1))
// this is gcc 5.1 or greater
#endif

Notice the chack has to be: major > 5 || (major == 5 && minor >= 1). If you only do major == 5 && minor >= 1, it won’t work for version 6.0.

clang

__clang_major__, __clang_minor__, __clang_patchlevel__

Visual Studio

_MSC_VER and _MSC_FULL_VER:

VS                        _MSC_VER   _MSC_FULL_VER
1                         800
3                         900
4                         1000
4                         1020
5                         1100
6                         1200
6 SP6                     1200    12008804
7                         1300    13009466
7.1 (2003)                1310    13103077
8 (2005)                  1400    140050727
9 (2008)                  1500    150021022
9 SP1                     1500    150030729
10 (2010)                 1600    160030319
10 (2010) SP1             1600    160040219
11 (2012)                 1700    170050727
12 (2013)                 1800    180021005
14 (2015)                 1900    190023026
14 (2015 Update 1)        1900    190023506
14 (2015 Update 2)        1900    190023918
14 (2015 Update 3)        1900    190024210
15 (2017 Update 1 & 2)    1910    191025017
15 (2017 Update 3 & 4)    1911
15 (2017 Update 5)        1912

More information:

MinGW

MinGW (aka MinGW32) and MinGW-w64 32bit: __MINGW32_MAJOR_VERSION and __MINGW32_MINOR_VERSION

MinGW-w64 64bit: __MINGW64_VERSION_MAJOR and __MINGW64_VERSION_MINOR

Checking processor architecture

gcc

The meaning of those should be self-evident:

__i386__
__x86_64__
__arm__. If defined, you can further check:
- __ARM_ARCH_5T__
- __ARM_ARCH_7A__
__powerpc64__
__aarch64__

Tutorial for github.com/kjk/flex Go package (implementation of CSS flexbox algorithm)

2017-08-04T18:32:51-07:00

Package github.com/kjk/flex implements CSS flexbox layout algorithm in Go.

It’s a pure Go port of Facebook’s Yoga C library.

High-level API overview

Despite implementing CSS flexbox spec, it isn’t tied to CSS/HTML in any way. Yoga, for example, can be integrated with iOS app and used to layout UIView hierarchy.

The library works on abstract tree of nodes. In HTML a node would correspond to a block element like a div. When used in Cocoa app, a node could represent UIView or NSView.

When used on windows, it could represent a HWND-based control.

The high-level use is:

create a tree of nodes that represents a layout you want to represent
set desired flexbox properties on each node using node.StyleSet*() functions
call flex.CalculateLayout(rootNode, parentWidth, parentHeight, direction)
each node is now measured and positioned so you can e.g. size and position widgets associated with each node. You can get the size on position of nodes with node.LayoutGet*() functions
when layout hierachy changes (e.g. the node represents a label and you’ve changed its text, which changes it’s intrinsic size), call node.MarkDirty() and CalculateLayout() to re-calculate new size/position of the nodes

An exmple

Let’s assume that we want to re-create the following HTML layout:

<div id="percentage_multiple_nested_with_padding_margin_and_percentage_values" style="width: 200px; height: 200px; flex-direction: column;">
  <div style="flex-grow: 1; flex-basis: 10%; min-width: 60%; margin: 5px; padding: 3px;">
    <div style="width: 50%; margin: 5px; padding: 3%;">
      <div style="width: 45%; margin: 5%; padding: 3px;"></div>
    </div>
  </div>
  <div style="flex-grow: 4; flex-basis: 15%; min-width: 20%;"></div>
</div>

The equivalent Go code is:

config := flex.NewConfig()

root := flex.NewNodeWithConfig(config)
root.StyleSetWidth(200)
root.StyleSetHeight(200)

rootChild0 := flex.NewNodeWithConfig(config)
rootChild0.StyleSetFlexGrow(1)
rootChild0.StyleSetFlexBasisPercent(10)
rootChild0.StyleSetMargin(EdgeLeft, 5)
rootChild0.StyleSetMargin(EdgeTop, 5)
rootChild0.StyleSetMargin(EdgeRight, 5)
rootChild0.StyleSetMargin(EdgeBottom, 5)
rootChild0.StyleSetPadding(EdgeLeft, 3)
rootChild0.StyleSetPadding(EdgeTop, 3)
rootChild0.StyleSetPadding(EdgeRight, 3)
rootChild0.StyleSetPadding(EdgeBottom, 3)
rootChild0.StyleSetMinWidthPercent(60)
root.InsertChild(rootChild0, 0)

rootChild0Child0 := flex.NewNodeWithConfig(config)
rootChild0Child0.StyleSetMargin(EdgeLeft, 5)
rootChild0Child0.StyleSetMargin(EdgeTop, 5)
rootChild0Child0.StyleSetMargin(EdgeRight, 5)
rootChild0Child0.StyleSetMargin(EdgeBottom, 5)
rootChild0Child0.StyleSetPaddingPercent(EdgeLeft, 3)
rootChild0Child0.StyleSetPaddingPercent(EdgeTop, 3)
rootChild0Child0.StyleSetPaddingPercent(EdgeRight, 3)
rootChild0Child0.StyleSetPaddingPercent(EdgeBottom, 3)
rootChild0Child0.StyleSetWidthPercent(50)
rootChild0.InsertChild(rootChild0Child0, 0)

rootChild0Child0Child0 := flex.NewNodeWithConfig(config)
rootChild0Child0Child0.StyleSetMarginPercent(EdgeLeft, 5)
rootChild0Child0Child0.StyleSetMarginPercent(EdgeTop, 5)
rootChild0Child0Child0.StyleSetMarginPercent(EdgeRight, 5)
rootChild0Child0Child0.StyleSetMarginPercent(EdgeBottom, 5)
rootChild0Child0Child0.StyleSetPadding(EdgeLeft, 3)
rootChild0Child0Child0.StyleSetPadding(EdgeTop, 3)
rootChild0Child0Child0.StyleSetPadding(EdgeRight, 3)
rootChild0Child0Child0.StyleSetPadding(EdgeBottom, 3)
rootChild0Child0Child0.StyleSetWidthPercent(45)
rootChild0Child0.InsertChild(rootChild0Child0Child0, 0)

rootChild1 := flex.NewNodeWithConfig(config)
rootChild1.StyleSetFlexGrow(4)
rootChild1.StyleSetFlexBasisPercent(15)
rootChild1.StyleSetMinWidthPercent(20)
root.InsertChild(rootChild1, 1)
flex.CalculateLayout(root, flex.Undefined, flex.Undefined, DirectionLTR)

After CalculateLayout we can see the position of each node e.g.:

fmt.Printf("root left: %f\n", root.LayoutGetLeft()) // 0
fmt.Printf"root top: %f\n", root.LayoutGetTop()) // 0
fmt.Printf("root width: %f\n", root.LayoutGetWidth()) // 200
fmt.Printf("root height: %f\n", root.LayoutGetHeight()) // 200

To see example for every flexbox property, look into github.com/facebook/yoga/gentest/fixtures. Their names hint at which properties are being used.

Each file there has corresponding *_test.go file in github.com/kjk/flex directory which shows how to express it in Go.

Size of root’s parent

Notice that in this particular example we used flex.Undefined as both height and width of the parent container.

Imagine you’re using flex to implment layout for a dekstop application where each flex.Node represents a control inside the window.

Window is the parent of root node.

In response to user resizing the window, you want to pass width/height of the window to flex.CalculateLayout().

When you create the window initially, you might do the reverse: pass flex.Undefined as width/height of parent container and then use the size of root node as the size of the window, to size it to its content.

Measure function

Imagine that a node represents an OS button. The button has some intrisic size dictated by its text.

To represent that size flex allows setting a measuring function with node.SetMeasureFunc(measureFunc MeasureFunc). It’s definition is:

type MeasureFunc func(node *Node, width float32, widthMode MeasureMode, height float32, heightMode MeasureMode) Size

The functions takes a hint width/height which is the size of parent container and returns intrinsic size of node.

This is usefule e.g. when a node represents a paragraph of text. When you know width of the parent container, you can break it into multi-line text.

If measuring function needs some state, you can use node.Context to store it.

Experience porting 4.5k loc of C to Go (Facebook's CSS flexbox implementation Yoga)

2017-08-02T00:00:00Z

I’m working on desktop GUI apps for Windows in Go.

As it uses native win32 controls I don’t have to implement my own buttons, list boxes etc.

However, win32 doesn’t have anything to help you layout things on the screen. Manual positioning is painful.

I started looking into a writing a layout engine. There are many systems to be inspired by.

I did some work in Winforms so I could copy their ideas.

I could look into copying Android layout logic.

I could look into implementing constraint-based layout like in iOS.

The option I liked the best is CSS flexbox. It’s not the simplest, it’s not the most complicated, but the biggest win is that many people, including myself, are already familiar with it.

I started looking for an implementation in Go. I found one in shiny project, but it’s simplified implementation.

The most complete implementation is Facebook’s Yoga, but it’s written in C (with bindings to many languages).

I could write cgo bindings but I prefer my Go pure so I decided to manually port C code to Go, all 4.5k lines of it.

The result is flex package.

Here’s how porting happened.

Phase zero - picking a name for the package

It was tempting to pick a name that would be some combination of yoga and go e.g. goyoga or yoga-go.

That would be a bad idea. Including go in package or repository name is bad (but unfortunately happens too often).

My rules for a good name for a Go library are:

it’s short
it’s descriptive
package name matches repository name

Therefore github.com/kjk/flex was born.

I could use yoga, to better higlight connection with original project, but it’s not descriptive.

I could use flexbox to better higlight connection with CSS flexbox, but it’s a bit long.

Phase one - getting Go port to compile

You can’t port 4.5k lines of code in one sitting so it was important to have a strategy that allows for making incremental progress.

The important part was keeping track of porting progress. For that I cloned Yoga’s repository. After converting a piece of C code to Go I would delete it from my fork. That way I would keep track of code that still needs to be ported.

It’s important to get an early boost so I started with the simplest parts: structures, enums.

Porting process was boring but relatively uneventful.

Some porting comments:

Go and C have reversed order of declarations. Reversing it manually is boring and error prone. There were many repeated declarations that I could do with a simple search-and-replace, e.g YGNodeRef node => node *YGNode
search-and-replace -> to . as Go uses . for both cases (something that C++-28 should adopt)
Go doesn’t support ternary operator and Yoga’s developers are infatuated with it. That was one part where I had to be extra careful as it was more than mechanical change
Go doesn’t support xor (^) and or (|) operator on bool. As it turns out, they are not neccessary. x ^ y is the same as x != y. a = a | b can be written as: if !a && b { a = true }
switch statement needs attention as in C case falls-through by default
Yoga uses float for coordinates and uses a few functions like fmaxf etc. Go only implements float64 versions of math functions. Missing functions were easy to write.
Yoga code was relatively easy to port partly because of lack of string handling, which is where Go and C differ a lot

It took me 2 days to port C code to Go, get it compiling and passing some tests. But some tests were failing.

Phase two - fixing bugs

Yoga has a lot of tests which gave me confidence that when the port is done, I’ll be able to verify its correctness by porting the tests.

I ported the code, I started porting the tests and they were failing.

I was in a pickle.

2 days was enough to mechanically port the code but not even close enough to understand it.

As far as I can tell you can’t step through tests in the debugger, so I converted a failing test into a test program.

I was pleasently surprised that debugger in Visual Studio Code works decently.

I found one porting mistake by stepping through the code but fixing it didn’t fix test failures.

I spent a bit more time stepping through the code but not knowing what results to expect I decided that it’s not a promising approach.

I decided to re-check every line of code.

Initially I ported the code in random order so I used re-checking time to also re-arrange the code in the same order as C code. That will help re-porting Yoga improvments in the future.

It was predictably boring. It took another day, I found 2 more porting bugs.

The tests were still failing and I was a bit stuck.

I did another pass in the debugger and fortunately an inspiration struck.

Yoga uses NaN value as undefined. I noticed that fmaxf function that I wrote returned NaN when any argument was NaN.

I compared my fmaxf implementation with C implmentation and it turns out that if one of the arguments is NaN but the other one isn’t, it should return non-NaN value.

It was easy to fix and that fixed all the tests.

Phase three - Go-ifying the API

The code was working but it was far from idiomatic Go code.

Next phase was tweaking the API to be more Go-like.

The majority of changes were:

making struct fields public
removing YG prefix as packages obviate the need for that
renaming accessor functions to methods (from func NodeFoo(node *Node) => func (node *Node) Foo()

It’s mechanical, boring process.

Some transformations could be done with regex search & replace. The rest with manual labor.

Having lots of tests really helped in making sure that the code is still correct.

The status

The port is finished. It works and passes all Yoga tests.

The API is still a bit awkward by Go standards. Too many methods, not enough direct access to variables.

Partly it’s because I wanted to stay close to C code so that future changes to Yoga can be ported easily.

Partly it’s because of implementation choices and the nature of the problem.

There are many flexbox properties, only some are given for any given element. The rest is given default values, which might be undefined.

How to represent that in Go? Ideally, the zero value of the type would represent default value so that a style definition can be constructed as &Style{} and then we can set the properties that are non-default.

Unfortunately, that doesn’t work well for properties that are numbers because zero might be a valid value.

Yoga uses float32 for numbers and NaN for undefined value of a CSS property.

Another option is used in sql package for types like sql.NullString that need to indicate null-abilibility. They are represented as a struct that combines a value and bool field Valid. Zero value of bool is false, so by default values start as invalid (undefined).

This isn’t a great API either since it makes setting and reading values more awkward.

At the end of the day it’s better to have solid, working code with awkward API than not having the code at all. Full flexbox implementation is not trivial to write.

Unless someone writes a pure Go implementation designed from scratch with Go idioms in mind, flex is the best option.

Notes on automatic translation

Go is so close to C. Wouldn’t it be great if there was a program that could take C code and turn it into Go code?

There are few attempts to do that:

elliotchance/c2go seems to be the most promising.

I didn’t try any of them as neither seems to be usable yet.

It’s an approach worth keeping an eye on for the future but not yet viable today.

Using MySQL in Docker for local testing In Go

2017-07-23T00:00:00Z

Imagine you’re writing a web application that uses MySQL. You deploy on Linux but code and test on Mac.

What is a good way to setup MySQL database for local developement and testing on Mac?

You can install MySQL on Mac using official MySQL installer or via Homebrew but my favorite way is to use docker.

Docker better isolates MySQL from the rest of the system, which has a couple of advantages:

it’s easier to install the exact version of MySQL that is running in production as there is a docker image for every version
you don’t have to worry that brew upgrade will upgrade MySQL. Auto-upgrade is desired for most software but not a database. You need to remember to use brew pin to disable that
you can run several different versions of MySQL for different projects
since it’s running on Linux, it’s closer to the code running in production

There is a downside to using Docker:

you have to make sure that the database container is running
you need to know the ip address of docker vm running the container

Doing this manually would be annoying and I like to automate so I wrote re-usable bit of Go code to do that.

It’s just a matter of running docker commands and parsing their outputs but it’s difficult enough to worth sharing the complete solution.

Conceptually, what we do is:

run docker ps -a and parse the output
if the container is not running at all, start it with docker run
if the container is stopped, re-start it with docker start
if the container is already running, extract ip address/port from the output

MySQL database is stored in a local directory mounted by the container. That way data persists even if the container is stopped.

The code is re-usable. You can customize it by changing:

the base MySQL container. In my case it’s mysql:5.6
where MySQL data is stored
name of the container, which should by unique to the project
port on which the database is exposed locally. In the container MySQL listens on standard port 3306. It must be exposed locally on a unique port

It can be adapted for other databases, like PostgreSQL.

We should only start docker when running locally. In my software I use cmd-line flag -production to distinguish between running in production and locally.

In production I would use the hard-coded host/ip of MySQL server. Locally I would call startLocalDockerDbMust() go get them.

const (
    dockerStatusExited  = "exited"
    dockerStatusRunning = "running"
)

var (
    // using https://hub.docker.com/_/mysql/
    // to use the latest mysql, use mysql:8
    dockerImageName = "mysql:5.6"
    // name must be unique across containers runing on this computer
    dockerContainerName = "mysql-db-multi"
    // where mysql stores databases. Must be on local disk so that
    // database outlives the container
    dockerDbDir = "~/data/db-multi"
    // 3306 is standard MySQL port, I use a unique port to be able
    // to run multiple mysql instances for different projects
    dockerDbLocalPort = "7200"
)

type containerInfo struct {
    id       string
    name     string
    mappings string
    status   string
}

func quoteIfNeeded(s string) string {
    if strings.Contains(s, " ") || strings.Contains(s, "\"") {
        s = strings.Replace(s, `"`, `\"`, -1)
        return `"` + s + `"`
    }
    return s
}

func cmdString(cmd *exec.Cmd) string {
    n := len(cmd.Args)
    a := make([]string, n, n)
    for i := 0; i < n; i++ {
        a[i] = quoteIfNeeded(cmd.Args[i])
    }
    return strings.Join(a, " ")
}

func runCmdWithLogging(cmd *exec.Cmd) error {
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    return cmd.Run()
}

func decodeContainerStaus(status string) string {
    // convert "Exited (0) 2 days ago" into statusExited
    if strings.HasPrefix(status, "Exited") {
        return dockerStatusExited
    }
    // convert "Up <time>" into statusRunning
    if strings.HasPrefix(status, "Up ") {
        return dockerStatusRunning
    }
    return strings.ToLower(status)
}

// given:
// 0.0.0.0:7200->3306/tcp
// return (0.0.0.0, 7200) or None if doesn't match
func decodeIPPortMust(mappings string) (string, string) {
    parts := strings.Split(mappings, "->")
    panicIf(len(parts) != 2, "invalid mappings string: '%s'", mappings)
    parts = strings.Split(parts[0], ":")
    panicIf(len(parts) != 2, "invalid mappints string: '%s'", mappings)
    return parts[0], parts[1]
}

func dockerContainerInfoMust(containerName string) *containerInfo {
    cmd := exec.Command("docker", "ps", "-a", "--format", "{{.ID}}|{{.Status}}|{{.Ports}}|{{.Names}}")
    outBytes, err := cmd.CombinedOutput()
    panicIfErr(err, "cmd.CombinedOutput() for '%s' failed with %s", cmdString(cmd), err)
    s := string(outBytes)
    // this returns a line like:
    // 6c5a934e00fb|Exited (0) 3 months ago|0.0.0.0:7200->3306/tcp|mysql-db-multi
    s = strings.TrimSpace(s)
    lines := strings.Split(s, "\n")
    for _, line := range lines {
        parts := strings.Split(line, "|")
        panicIf(len(parts) != 4, "Unexpected output from docker ps:\n%s\n. Expected 4 parts, got %d (%v)\n", line, len(parts), parts)
        id, status, mappings, name := parts[0], parts[1], parts[2], parts[3]
        if containerName == name {
            return &containerInfo{
                id:       id,
                status:   decodeContainerStaus(status),
                mappings: mappings,
                name:     name,
            }
        }
    }
    return nil
}

// returns host and port on which database accepts connection
func startLocalDockerDbMust() (string, string) {
    // docker must be running
    cmd := exec.Command("docker", "ps")
    err := cmd.Run()
    panicIfErr(err, "docker must be running! Error: %s", err)
    // ensure directory for database files exists
    dbDir := expandTildeInPath(dockerDbDir)
    err = os.MkdirAll(dbDir, 0755)
    panicIfErr(err, "failed to create dir '%s'. Error: %s", err)
    info := dockerContainerInfoMust(dockerContainerName)
    if info != nil && info.status == dockerStatusRunning {
        return decodeIPPortMust(info.mappings)
    }
    // start or resume container
    if info == nil {
        // start new container
        volumeMapping := dockerDbDir + "s:/var/lib/mysql"
        dockerPortMapping := dockerDbLocalPort + ":3306"
        cmd = exec.Command("docker", "run", "-d", "--name"+dockerContainerName, "-p", dockerPortMapping, "-v", volumeMapping, "-e", "MYSQL_ALLOW_EMPTY_PASSWORD=yes", "-e", "MYSQL_INITDB_SKIP_TZINFO=yes", dockerImageName)
    } else {
        // start stopped container
        cmd = exec.Command("docker", "start", info.id)
    }
    runCmdWithLogging(cmd)

    // wait max 8 seconds for the container to start
    for i := 0; i < 8; i++ {
        info := dockerContainerInfoMust(dockerContainerName)
        if info != nil && info.status == dockerStatusRunning {
            return decodeIPPortMust(info.mappings)
        }
        time.Sleep(time.Second)
    }

    panicIf(true, "docker container '%s' didn't start in time", dockerContainerName)
    return "", ""
}

Code for this chapter: https://github.com/kjk/go-cookbook/blob/master/start-mysql-in-docker-go

Rotate log files daily in Go

2017-07-20T00:00:00Z

If you log to a file, it’s a good idea to rotate logs. That way they won’t become too large.

After rotation you can backup them up to online storage or delete them.

Rotating daily has a good balance of how often to rotate vs. how large the log file can become.

You can write logs to stdout and use external program, like logrotate, to do the rotation.

I prefer the simplicity of handling that in my own code and Go’s io.Writer interface makes it easy to implement a re-usable file ration code.

I did just that in package dailyrotate (documentation).

Here’s basic use:

import (
    "github.com/kjk/dailyrotate"
)

var (
    rotatedFile *dailyrotate.File
)

// called when file is closed. If didRotate is true, it was closed due to rotation
// at UTC midnight. Otherwise it was closed due to regular Close()
func onCloseHappened(path string, didRotate bool) {
    fmt.Printf("we just closed a file '%s', didRotate: %v\n", path, didRotate)
    if !didRotate {
        return
    }
    // we block writes until this returns so expensive processing should be done in
    // background goroutine
    go func() {
        // here you can implement things like:
        // - compressing rotated file
        // - deleting old files to free up disk space
        // - upload rotated file to backblaze/google storage/s3 for backup
        // - analyze the content of the file
    }()
}

func initRotatedFileMust() {
    pathFormat := filepath.Join("dir", "2006-01-02.log")
    w, err := dailyrotate.NewFile(pathFormat, onCloseHappened)
    panicIfErr(err)
}

func logString(s string) error {
    _, err = io.WriteString(rotatedFile, s)
    return err
}

func shutdownLogging() {
    rotatedFile.Close()
}

Here’s a real-life example of processing rotated file:

dailyrotate.File is io.Writer and safe to use from multiple goroutines.

In addition to Write(d []byte) it also implements a Write2(d []byte, flush bool) (string, int64, int, error). It has 2 improvements over Write:

allows to flush in a single call. Flushing after each write is slower but less likely to loose data or corrupt the file when program crashes
it returns file path and offset in the file at which the data was written. This is important for building a random-access index to records written to dailyrotate.File

Other real-world uses

Rotation is not limited to log files. I use it as part of poor-man’s web server analytics system.

I log info about web requests to dailyrotate.File using my siser simple serialization format.

When a file is rotated, I compress it, upload to backblaze for backup and delete local files older than 7 days to free up space.

I also calculate basic daily statistics and e-mail a summary to myself.

I know, I could just use Google Analytics, and I do. My little system has advantages: * it tells me about missing pages (404). That alerts me if break something or if others link incorrectly to my website. Knowing bad links, I can add re-directs for them * by getting daily summaries via e-mail, I keep an eye on things with minimal effort

Simple serialization format for logging and analytics in Go

2017-07-17T00:00:00Z

I was looking to save multiple records with somewhat flexible schema to a file. Go has plenty of options but I couldn’t find anything that was just right.

My desired features were:

human-readable so that I can use tools like grep or tail to look at the file
records are not fixed i.e. can have variable number of fields
no need to support nested records
allows for simple and efficient implementation

Here are some most popular available options and why they don’t exactly fit the bill:

csv - efficient but uses fixed records; not very readable if there are many fields on a single line
json - not very readable
protocol buffers - binary so unreadable; needs up-front scheme

I designed and implemented my own format in package siser (which stands for Simple Serialization).

You’ll not be surprised by how it looks:

url: http://blog.kowalczyk.info/index.html
code: 200
large field:+13789
this is large data, 13789 bytes in size...
---

It’s a typical key/value serialization with one neat feature: support for large data (e.g. an image or long text).

The format is line oriented. Each line is ${key}: ${value}\n.

If the value is larger than 120 bytes or is not ascii text (with bytes outside of 32-127 range), I serialize it as large value:

${key}:+${value_length}\n
${value}\n

To separate records I use ---\n.

I use this format for 2 main purposes:

structured logging to help in debugging
logging events for analytics

Using the library

When used for analytics, each record represents an event. I save the events to a file and later on process the whole file record-by-record and calculate desired statistics.

Here’s how we would log info about HTTP requests:

func logHTTPRequest(w io.Writer, url string, ipAddr string, statusCode int) error {
    var r siser.Record
    // you can append multiple key/value pairs at once
    r.Append("url", url, "ipaddr", ipAddr)
    // or assemble with multiple calls
    r.Append("code", strconv.Itoa(statusCode))
    /// ... more fields
    d := r.Marshal()
    _, err := w.Write(d)
    return err
}

Here’s a full example of logging HTTP requests.

Let’s say we wrote the data to http_access.log file. Here’s how we would process the records:

f, err := os.Open("http_access.log")
panicIfErr(err)
defer f.Close()
r := siser.NewReader(f)
for r.ReadNext() {
    _, record := r.Record()
    code, ok := r.Get("code")
    // get rest of values and do something with them
}
panicIfErr(r.Err())

Here’s a full example of calculating basic daily statistics on HTTP requests (most frequently visited pages, most frequent 404s, most frequent referrers).

Annoyances

The library doesn’t offer marshaling directly to/from structs. Could be added with a bit of reflection.

It doesn’t directly support non-string types like int or time.Time. You have to convert them to/from string yourself.

Implementation notes

Some implementation decisions were made with performance in mind.

Given key/value nature of the record, an easy choice would be to use map[string]string as argument to encode/decode functions.

However []string is more efficient than a map. A slice can be reused across multiple records. We can clear by re-slicing as empty slice and reusing underlying array. A map would require allocating a new instance for each record, which would create a lot of work for garbage collector.

When serializing, you need to use Reset method to get the benefit of efficient re-use of the record.

When reading and deserializing records, siser.Reader uses this optimization internally.

The format avoids the need for escaping keys and values, which helps in making encoding/decoding fast.

How does that play out in real life? I wrote a benchmark comparing siser vs. json.Marshal. It’s about 30% faster:

$ go test -bench=.
BenchmarkSiserMarshal-8          1000000          2108 ns/op
BenchmarkJSONMarshal-8            500000          3181 ns/op

BenchmarkSiserUnmarshal-8         500000          2908 ns/op
BenchmarkJSONUnmarshal-8          200000          7482 ns/op

The format is binary-safe and works for serializing large values e.g. you can serialize png image

It’s also very easy to implement in any language.

Embedding build number in Go executable

2017-07-13T00:00:00Z

So you’ve deployed your web application to production and it’s running on a server far, far away.

When debugging problems it’s good to know what version of the code is running.

If you’re using git, that would be sha1 of the revision used to build the program.

We can embed that version in the executable during build thanks Go linker’s -X option which allows to change any variable in the program.

In our Go program we would have:

package main

var (
    sha1ver   string // sha1 revision used to build the program
    buildTime string // when the executable was built
)

We can set that variable to sha1 of the git revision in our build script build.sh:

#!/bin/bash

# notice how we avoid spaces in $now to avoid quotation hell in go build command
now=$(date +'%Y-%m-%d_%T')
go build -ldflags "-X main.sha1ver=`git rev-parse HEAD` -X main.buildTime=$now"

Full example: embed-build-number/build.sh

On Windows we would write powershell script build.ps1:

# notice how we avoid spaces in $now to avoid quotation hell in go build command
$now = Get-Date -UFormat "%Y-%m-%d_%T"
$sha1 = (git rev-parse HEAD).Trim()

go build -ldflags "-X main.sha1ver=$sha1 -X main.buildTime=$now"

Full example: embed-build-number/build.ps1

Let’s deconstruct:

git rev-parse HEAD returns sha1 of the current revision, e.g. e5ce06c1f604efb1de91d515d5de865e7e164d59
-X main.sha1ver=${foo} tells Go linker to set variable sha1ver in package main to ${foo}
-ldflags "${flags}" tells Go build tool to pass ${flags} to go linker

We also need an easy way to see that version. We can add -version cmd-line flag to print it out:

var (
    flgVersion bool
)

func parseCmdLineFlags() {
    flag.BoolVar(&flgVersion, "version", false, "if true, print version and exit")
    flag.Parse()
    if flgVersion {
        fmt.Printf("Build on %s from sha1 %s\n", buildTime, sha1ver)
        os.Exit(0)
    }
}

If this is a web application, we can additionally add a debug page that would show the version. I often do it like that:

func servePlainText(w http.ResponseWriter, s string) {
    w.Header().Set("Content-Type", "text/plain")
    w.Header().Set("Content-Length", strconv.Itoa(len(s)))
    w.WriteHeader(http.StatusOK)
    w.Write([]byte(s))
}

// /app/debug
func handleDebug(w http.ResponseWriter, r *http.Request) {
    s := fmt.Sprintf("url: %s %s", r.Method, r.RequestURI)
    a := []string{s}

    a = append(a, "Headers:")
    for k, v := range r.Header {
        if len(v) == 0 {
            a = append(a, k)
        } else if len(v) == 1 {
            s = fmt.Sprintf("  %s: %v", k, v[0])
            a = append(a, s)
        } else {
            a = append(a, "  "+k+":")
            for _, v2 := range v {
                a = append(a, "    "+v2)
            }
        }
    }

    a = append(a, "")
    a = append(a, fmt.Sprintf("ver: https://github.com/kjk/go-cookbook/commit/%s", sha1ver))
    a = append(a, fmt.Sprintf("built on: %s", buildTime))

    s = strings.Join(a, "\n")
    servePlainText(w, s)
}

func makeHTTPServer() *http.Server {
    mux := &http.ServeMux{}

    mux.HandleFunc("/app/debug", handleDebug)

    return &http.Server{
        ReadTimeout:  5 * time.Second,
        WriteTimeout: 5 * time.Second,
        IdleTimeout:  120 * time.Second,
        Handler:      mux,
    }
}

func startHTTPServer() {
    httpAddr := "127.0.0.1:4040"
    httpSrv := makeHTTPServer()
    httpSrv.Addr = httpAddr
    fmt.Printf("Visit http://%s/app/debug\n", httpAddr)
    err := httpSrv.ListenAndServe()
    if err != nil {
        log.Fatalf("httpSrv.ListendAndServe() failed with %s\n", err)
    }
}

In addition to printing version of the code I also show HTTP headers. During debugging, the more information, the better.

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/embed-build-number

3 ways to iterate in Go

2017-07-09T00:00:00Z

Iteration is a frequent need, be it iterating over lines of a file, results or of SELECT SQL query or files in a directory.

There are 3 common iteration patterns in Go programs: * callbacks * an iterator object with Next() method * channels

Iteration mixed with processing

Before discussing different ways of designing iteration API, let’s see how we would iterate without encapsulating iteration logic.

Our example is iterating over even numbers, starting with 2 up to a given max number (inclusive).

To show handling of errors we’ll consider max less than 0 to be invalid.

This is intentionally the simplest possible iterator so that we can focus on the implementation of the iterator API and not generating the values to iterate over.

Our processing is simple as well: we print the number.

Here’s an example of iteration intertwined with processing.

func printEvenNumbers(max int) {
    if max < 0 {
        log.Fatalf("'max' is %d, should be >= 0", max)
    }
    for i := 2; i <= max; i += 2 {
        fmt.Printf("%d\n", i)
    }
}

Full example: 3-ways-to-iterate/inlined.go.

This is fine if iteration logic is simple. If our iteration was complex, like iterating over lines in a file, every time we needed to do different processing of the lines, we would end up copy & pasteing a lot of code.

For easy reuse we want to encapsulate complex iteration logic and provide simple API to callers.

Iterating via callback

The caller provides callback function to be called with each value.

Client side of iteration:

func printEvenNumbers(max int) {
    err := iterateEvenNumbers(max, func(n int) error {
        fmt.Printf("%d\n", n)
        return nil
    })
    if err != nil {
        log.Fatalf("error: %s\n", err)
    }
}

We need a way to stop iteration from within the callback which is why the callback returns an error. Returning non-nil error from callback stops iteration.

Implementation of iterator:

func iterateEvenNumbers(max int, cb func(n int) error) error {
    if max < 0 {
        return fmt.Errorf("'max' is %d, must be >= 0", max)
    }
    for i := 2; i <= max; i += 2 {
        err := cb(i)
        if err != nil {
            return err
        }
    }
    return nil
}

Full example: 3-ways-to-iterate/callback.go.

This pattern is used in filepath.Walk API in standard library.

Iterating with `Next()`

Another pattern is to implement iterator struct with 3 functions: * Next() advances iterator to next value. It returns false to indicate end of iteration (which can be due to error) * Value() to get the current value of the iterator. The name depends on the kind of value we retrieve * optional Err() function which returns iteration error

Client code:

func printEvenNumbers(max int) {
    iter := NewEvenNumberIterator(max)
    for iter.Next() {
        fmt.Printf("n: %d\n", iter.Value())
    }
    if iter.Err() != nil {
        log.Fatalf("error: %s\n", iter.Err())
    }
}

Notice how Next() fits nicely with for loop thanks to returning bool and indicating end of iteration with false.

Unfortunately, the nice API on the caller side requires complicated implementation of the iterator.

We need to carry state across Next() calls and remember iteration errors:

// EvenNumberIterator generates even numbers
type EvenNumberIterator struct {
    max       int
    currValue int
    err       error
}

// NewEvenNumberIterator creates new number iterator
func NewEvenNumberIterator(max int) *EvenNumberIterator {
    var err error
    if max < 0 {
        err = fmt.Errorf("'max' is %d, should be >= 0", max)
    }
    return &EvenNumberIterator{
        max:       max,
        currValue: 0,
        err:       err,
    }
}

// Next advances to next even number. Returns false on end of iteration.
func (i *EvenNumberIterator) Next() bool {
    if i.err != nil {
        return false
    }
    i.currValue += 2
    return i.currValue <= i.max
}

// Value returns current even number
func (i *EvenNumberIterator) Value() int {
    if i.err != nil || i.currValue > i.max {
        panic("Value is not valid after iterator finished")
    }
    return i.currValue
}

// Err returns iteration error.
func (i *EvenNumberIterator) Err() error {
    return i.err
}

Full example: 3-ways-to-iterate/next.go.

Notes:

this method requires the largest amount of boilerplate
Next() should return false if there was an error
Value() panics if accessed after iteration has finished

This pattern is used in standard library:

Rows.Next to iterate over results of SQL SELECT statement
Scanner.Scan to iterate over text
Decoder.Token for XML parsing
Reader.Read in CSV reader

Some of those iterators combine Next() and Value() into a single function returning multiple values.

Iterating with a channel

Channels and goroutines are Go’s banner features and can be used as iterators.

Caller side:

func printEvenNumbers(max int) {
    for val := range generateEvenNumbers(max) {
        if val.Err != nil {
            log.Fatalf("Error: %s\n", val.Err)
        }
        fmt.Printf("%d\n", val.Int)
    }
}

generateEvenNumbers() returns a channel which will be closed to indicate end of iteration. Closing the channel ends for loop.

If there is no possibility of failing we can send just values over the channel.

In our case a failure is possiblity, so we have to send a struct that packages the value and possible error:

type IntWithError struct {
    Int int
    Err error
}

Generator side:

func generateEvenNumbers(max int) chan IntWithError {
    ch := make(chan IntWithError)
    go func() {
        defer close(ch)
        if max < 0 {
            ch <- IntWithError{
                Err: fmt.Errorf("'max' is %d and should be >= 0", max),
            }
            return
        }

        for i := 2; i <= max; i += 2 {
            ch <- IntWithError{
                Int: i,
            }
        }
    }()
    return ch
}

Full example: 3-ways-to-iterate/channel.go.

We could use buffered channel, e.g.: ch := make(chan IntWithError, 128). That would speed up things if both generation and processing are time consuming by parallelizing those 2 processes.

Adding cancellation to channel-based iterator

In the above example the client doesn’t have a way to stop the channel-based iterator.

If it just stops processing values from the channel before it’s closed, the generator goroutine will be forever blocked trying to send on a channel no-one is reading from. The goroutine will leak.

We can add ability to stop a channel iterator by using context created with context.WithCancel.

Here’s slightly modified generator function:

func generateEvenNumbers(ctx context.Context, max int) chan IntWithError {
    ch := make(chan IntWithError)
    go func() {
        defer close(ch)
        if max < 0 {
            ch <- IntWithError{
                Err: fmt.Errorf("'max' is %d and should be >= 0", max),
            }
            return
        }

        for i := 2; i <= max; i += 2 {
            if ctx != nil {
                // if context was cancelled, we stop early
                select {
                case <-ctx.Done():
                    return
                default:
                }
            }
            ch <- IntWithError{
                Int: i,
            }
        }
    }()
    return ch
}

Full example: 3-ways-to-iterate/channel-cancellable.go.

Here’s a user of the above code that stops the generator a bit early:

func printEvenNumbersCancellable(max int, stopAt int) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    ch := generateEvenNumbers(ctx, max)
    for val := range ch {
        if val.Err != nil {
            log.Fatalf("Error: %s\n", val.Err)
        }
        if val.Int > stopAt {
            cancel()
            // notice we keep going in order to drain the channel
            continue
        }
        // process the value
        fmt.Printf("%d\n", val.Int)
    }
}

Important to note:

we must call cancel() on the context we get from context.WithCancel() or else it’ll leak
it’s safe to call cancel() multiple times
cancel() asks for the generator to stop but cannot guarantee it. After cancelling we’ll get more values queued on the channel

It’s important to fully drain the channel or else we’ll leak the goroutine that is trying to write to it. In this example we skip processing after reaching stop value.

Alternatively, we could break the processing loop and then have a loop just to drain the channel:

for _ = range ch {
}

Which way is the best?

The one that best fits your scenario.

The callback pattern makes for a simple implementation of the iterator but callbacks in Go have akward syntax.

Using Next() is the hardest to implement but presents nice interface to the caller. It’s most commonly used in Go standard library for complex iterators.

Channel-based iterator is easy to implent and use by the caller but most expensive. Only in exceptional circumstances the cost should be of concern.

It’s also the only one that is concurrent by nature.

At the same time, channels are heavy machinery. Don’t over-use them, don’t use channels for things that can be done without them.

Channels are meant to serve as a coordination mechanism between goroutines. If you don’t need to launch a goroutine, then you probably don’t need to use channels.

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/3-ways-to-iterate

HTTPS for free in Go, with little help of Let's Encrypt

2017-07-02T00:00:00Z

Why HTTPS?

Having HTTPS for your website is important:

HTTPS encrypts the traffic between browser and server. Passwords of your users are protected from traffic sniffing on naughty intermediary servers or miscreants sniffing wifi packets in a cafe.
new HTTP/2 protocol is faster than HTTP/1.1 but only works over HTTPS.
if you care about SEO, Google ranks HTTPS websites higher than HTTP ones.
browser vendors will increasingly scare users of HTTP websites

Using someone else to provide HTTPS

Before we learn how to support HTTPS directly in your Go web server, let’s talk about simpler options.

You can use a third-party service like Cloudflare.

Their free plan offers acting as HTTPS proxy to your HTTP-only website.

To use Cloudflare: * configure your domain to use CloudFlare’s DNS servers * in CloudFlare’s DNS settings point the domain to your server and set Status to “DNS and HTTP proxy (CDN)” * in CloudFlare’s Crypto settings, set SSL to “Flexible” (browser talks to CloudFlare via HTTPS, CloudFlare talks to * configure CloudFlare’s HTTPS proxy in their web interface by providing IP address of your server. * for good measure, also enable “Always use HTTPS”

Browser talks to CloudFlare, which takes care of provisioning SSL certificate and proxies the traffic to your server. This might be slower due to additional traffic or faster due to CloudFlare servers being faster than yours (being faster is their business).

AWS, Google Cloud and some other hosting providers also provide free HTTPS for servers hosted on their infrastructures.

Another option is to run your server behind reverse proxy supporting HTTPS, like Caddy.

Directly supporting HTTPS

Not long ago, if you wanted a SSL certificate, you had to pay many dollars a year for each domain.

Let’s Encrypt changed that. It’s a non-profit organization that provides certificates for free and offers HTTP API for obtaining certificates. API allows automating the process.

Before Let’s Encrypt you would buy a certificate, which is just a bunch of bytes. You would save certificate to a file and configure your web server to use it.

With Let’s Encrypt you can use their API to obtain the certificate for free, automatically, when your server starts.

Thankfully all the hard work of talking to the API has already bee done by others. We just need to plug it in.

There are a couple of Go libraries that implement Let’s Encrypt support.

I’ve been using golang.org/x/crypto/acme/autocert, which is developed by Go core developers. It’s been several months now and it works flawlessly.

Here’s how to start HTTPS web server that uses free SSL certificates from Let’s Encrypt.

const (
    htmlIndex    = `<html><body>Welcome!</body></html>`
    inProduction = true
)

func handleIndex(w http.ResponseWriter, r *http.Request) {
    io.WriteString(w, htmlIndex)
}

func makeHTTPServer() *http.Server {
    mux := &http.ServeMux{}
    mux.HandleFunc("/", handleIndex)

    // set timeouts so that a slow or malicious client doesn't
    // hold resources forever
    return &http.Server{
        ReadTimeout:  5 * time.Second,
        WriteTimeout: 5 * time.Second,
        IdleTimeout:  120 * time.Second,
        Handler:      handler,
    }
}

func main() {
    var httpsSrv *http.Server
    var m *autocert.Handler

    // when testing locally it doesn't make sense to start
    // HTTPS server, so only do it in production.
    // In real code, I control this with -production cmd-line flag
    if inProduction {
        // Note: use a sensible value for data directory
        // this is where cached certificates are stored
        dataDir := "."
        hostPolicy := func(ctx context.Context, host string) error {
            // Note: change to your real domain
            allowedHost := "www.mydomain.com"
            if host == allowedHost {
                return nil
            }
            return fmt.Errorf("acme/autocert: only %s host is allowed", allowedHost)
        }

        httpsSrv = makeHTTPServer()
        m := &autocert.Manager{
            Prompt:     autocert.AcceptTOS,
            HostPolicy: hostPolicy,
            Cache:      autocert.DirCache(dataDir),
        }
        httpsSrv.Addr = ":443"
        httpsSrv.TLSConfig = &tls.Config{GetCertificate: m.GetCertificate}

        go func() {
            err := httpsSrv.ListenAndServeTLS("", "")
            if err != nil {
                log.Fatalf("httpsSrv.ListendAndServeTLS() failed with %s", err)
            }
        }()
    }

    httpSrv := makeHTTPServer()
    if m != nil {
        // allow autocert handle Let's Encrypt auth callbacks over HTTP.
        // it'll pass all other urls to our hanlder
        httpSrv.Handler = m.HTTPHandler(httpServ.Handler)
    }
    httpSrv.Addr = ":80"
    err := httpSrv.ListenAndServe()
    if err != nil {
        log.Fatalf("httpSrv.ListenAndServe() failed with %s", err)
    }
}

Full example: free-ssl-certificates/main.go.

There are some important things to note.

1. The standard port for HTTPS is 443

2. You can run only HTTP, only HTTPS or both.

3. If the server doesn’t have a certificate, it’ll use HTTP API to ask Let’s Encrypt servers for it.

Those requests are throttled to 20 per week to avoid over-loading Let’s Encrypt servers.

It’s therefore important to cache the certificate somewhere. In our example we cache them on disk, using autocert.DirCache cache.

Cache is an interface so you could implement your own storage e.g. in a SQL database or Redis.

4. You must set up DNS correctly.

To verify that you’re the owner of domain for which you want a certificate, Let’s Encrypt server calls back your server.

For that to work, DNS name must resolve to the IP address of your server.

This means that local testing of HTTPS code-path is hard. I usually don’t bother.

If you really want to, you can use ngrok to expose your local port to the internet, set DNS to resolve your domain to public DNS name that ngrok creates, wait a bit to make sure that DNS information propagates to Let’s Encrypt computers.

5. You might be wondering: what is this HostPolicy business?

As I mentioned, Let’s Certificate throttles certificate provisioning so you need to ensure the server won’t ask for certificates for domains you don’t care about. Autocert docs explain this well

Our example assumes most common case: a server that only responds to a single domain. You can easily change the logic.

6. We’re not running HTTPS when testing locally.

When testing locally on your laptop, there’s no point in running HTTPS version. Your computer most likely doesn’t have publicly visible IP address so Let’s Encrypt servers can’t reach you, so you won’t get the certificate.

We also won’t be able to bind to HTTPS port 443 (only root processes can bind to ports lower than 1024).

In the example I use inProduction flag to decide if I should start HTTPS server.

In real code I add code that checks for -production cmd-line flag and use that.

Redirecting from HTTP to HTTPS

If you can do HTTPS there’s no point in providing plain HTTP.

We can re-direct all HTTP request to HTTPS equivalent, for better security and SEO (Google doesn’t like duplicate content so your SEO rank will be better with a single version of the website).

func makeServerFromMux(mux *http.ServeMux) *http.Server {
    // set timeouts so that a slow or malicious client doesn't
    // hold resources forever
    return &http.Server{
        ReadTimeout:  5 * time.Second,
        WriteTimeout: 5 * time.Second,
        IdleTimeout:  120 * time.Second,
        Handler:      mux,
    }
}

func makeHTTPToHTTPSRedirectServer() *http.Server {
    handleRedirect := func(w http.ResponseWriter, r *http.Request) {
        newURI := "https://" + r.Host + r.URL.String()
        http.Redirect(w, r, newURI, http.StatusFound)
    }
    mux := &http.ServeMux{}
    mux.HandleFunc("/", handleRedirect)
    return makeServerFromMux(mux)
}

func main() {
    httpSrv := makeHTTPToHTTPSRedirectServer()
    httpSrv.Handler = m.HTTPHandler(httpServ.Handler)
    httpSrv.Addr = ":80"
    fmt.Printf("Starting HTTP server on %s\n", httpSrv.Addr)
    err := httpSrv.ListenAndServe()
    if err != nil {
        log.Fatalf("httpSrv.ListenAndServe() failed with %s", err)
    }
}

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/free-ssl-certificates

How certificates came to be free

Arguably due to a design mistake, SSL protocol not only encrypts but also proves site’s identity to the browser.

It provides accountability so that we can trace the ownership of google.com and see that it is indeed owned by Google, Inc in US, and not Ivan The Hacker in Moscow.

We implement that accountability by trusting a very small number of companies (Certificate Authorities) to issue certificates that prove the identity of the website owner.

When you apply for a certificate, Certificate Authority has to verify your identity. They do it by checking your papers.

Verifying the papers requires labor. Keeping certificates safe requires labor. It’s reasonable that Certificate Authorities charge for the service of issuing certifcates.

The trust doesn’t scale. Browser and OS vendors can trust 10 companies to not issue invalid certificates, but they can’t trust a thousand.

We don’t want any random company to become a rogue certificate authority and start issuing certificates for google.com domains to Ivan The Hacker.

It would be too much effort to continuosly audit thousands of Certificate Authority companies so as a result we ended up with just a few.

A market controlled by small number of companies tends to become a cartel that keeps prices high due to lack of competition.

That’s exactly what happened in SSL certificates market. You can have a low-end server for $60/year and a certificate alone would cost more than that.

That was a problem because the cost of SSL certificates was a significant barrier to adopting encryption by all websites.

A few companies decided to pool their resources and solve that problem for the greater good of the web.

They funded Let’s Encrypt which became a Certificate Authority, wrote necessary software and is running the servers that do the work of issuing certificates. It’s been a raging success.

And that’s how free certificates came to be.

Solo founders with profitable businesses, collected stories

2017-06-23T23:44:19Z

People sometimes wonder: can I have a successful business as a single founder?

The answer is: yes.

This is a collection of solo-preneur success stories (with occasional 2 people bands).

I only include businesses generating significant revenues. In this context it’s around $5k/mo or more, enough to replace full-time salary.

Maybe it’ll inspire you to start your own, solo business.

Before you get too excited, keep the following in mind.

This list is the pinnacle of survivorship bias. Solo-preneur software business is not different that any other business, and most businesses fail. You have 10-20% chance of success so pick your idea wisely, work hard and if you fail, do it again.

At the same time, this is only tip of the iceberg. Those are the stories that people shared, cribbed from a few online sources.

There are 100x more successful solo founders that don’t share their numbers publicly. A silent majority of successful solo businesses.

1. Anonymous making $750k/year with a desktop app, sold via his website

Source: https://news.ycombinator.com/item?id=13168965

Adivice:

make a desktop app that you can sell for $50-$300.
attack a large market that has stagnated or has entranched players with lousy apps i.e. make a better mousetrap.
electron is a good technology to write such app.

Important factors of success:

SEO
good reputation
big market
staying alive long enough for word of mouth to kick in

2. JollyTurns

Source: https://news.ycombinator.com/item?id=13170798

Web/iOS/Android app for ski resorts.

Income: unknown.

How it makes money: in-app purchase on iOS/Android ($1 per ski resort or $15 for all of them, see https://itunes.apple.com/us/app/jollyturns/id719208522)

First version (iOS only) released in Dec 2013 after 2 1/2 years of work (https://jollyturns.com/blog/first-public-release).

Insight: code is the easy part, marketing is hardest.

Tried to find a partner in SV but couldn’t. Wrote code himself, hired people to collect data about ski resorts.

3. CRM plugin that finds location of customer’s offices based on address of the hotel you’re traveling to

Source: https://news.ycombinator.com/item?id=13167930

Sold for $50. “made pretty good money”.

4. Niche app, $200k/year after 6 years

Source: https://news.ycombinator.com/item?id=13170346

Most likely USB driver that allows using USB devices remotely over a network (https://www.virtualhere.com, https://forum.openwrt.org/search.php?action=show_user_posts&user_id=129043 is same user name as on HN post).

No marketing, gets sales via word of mouth and internet searches.

5. Website templates, $100k/year profit

Source: https://news.ycombinator.com/item?id=13935663

Sold on https://themeforest.net/. Working about 4hrs a week.

6. Shopify plugins

Source: https://news.ycombinator.com/item?id=13167990

Makes enough money to hire full-time developer for onboarding of new clients, support and documentation. Does product development himself.

Insight: the trick for coming up with product ideas is to first do custom development. When enough clients are willing to spend a few thousand for a personal implementation of something, that’s when you know you have an opportunity to charge 40$ per month for a SaaS version.

7. $20k/mo from 3 niche SaaS products

Source:

Also in the past ran real estate SaaS, making $70k/mo at its height (it then crashed when real estate market crashed).

Advice:

tip for a newbie would be to look for something niche that solves a pain point or allows your customer to make or save money. For example: if your customer can spend $10/mo on your software and make or save $30/mo from that, you will have no problem getting & keeping customers.
don’t focus on becoming a unicorn. You can make some serious money and build a very comfortable life running a $300k to $1m dollar business, and your chances of succeeding at that are much greater.
Look for things outside of tech. There are so many problems to solve in small businesses. Many will say there is no money to make with small businesses.
Learn everything you can about advertising. Get really good at it and be willing to spend money on advertising.
Be willing to kill something off quickly if it doesn’t make money. Test your market early to make sure people will pay for it. I have made the mistake of not doing this and I have lost a lot of money because of it. Now, I need to be able to see a positive ROI on my spend within 3 to 4 months. So if I spent $100 to acquire a customer, I want to be able to get that back + more within 3 to 4 months. I know this timeline is probably really short for a VC funded company, but I have always been bootstrapped so don’t have the luxury of risking a longer time-frame for return.
it is not easy! Prepare to put in long hours especially in the beginning. Prepare for it to take a mental toll at times. You will second guess yourself, feel insecure, be consumed oftentimes with your business.

8. Ngrok

Source: https://news.ycombinator.com/item?id=13168185

Full time for past 4 years.

Insight: minimize support by improving UX, documentation, error messages.

9. Football betting analysis/predictions website, £75k / year

Source: https://news.ycombinator.com/item?id=11216868

http://betalyst.com/ makes £75,000 per year in advertising/sponsorship/affiliate revenue.

Gets !25k visitors a month.

Works 2-3 hrs a week.

Website traffic generation:

Android app with 75k users
10% of traffic is from organic search
majority of traffic from email (20k subscribers)

10. Watermarking desktop app for Mac/Windows

Source:

Makes $3k-$5k per month after 5 years.

Sells for $30/$60/$140.

11. Real estate startup, $130k/year of profit

Source: https://news.ycombinator.com/item?id=10887978

Profit:

130k in 2013
$100k in 2014
$140k in 2015

Source of revenues:

30% AdSense
20% users
50% affiliate marketing

Sources of traffic:

50% organic
37% referral
13% direct

No marketing, no blog, no social presence.

12. Dan Grossman, improvely and w3counter, $45k/mo

Source:

https://news.ycombinator.com/item?id=10881301
https://news.ycombinator.com/item?id=9726951 : how he started
https://news.ycombinator.com/item?id=10201718, https://news.ycombinator.com/item?id=8396497 : business tips
https://news.ycombinator.com/item?id=8406049

https://www.w3counter.com/ uses freemium model, people pay subscription for advanced features

https://www.improvely.com/ : SaaS priced $29/$79/$149/$299 / mo.

First customers for improvely came from $100-$200/month AdWords advertising for the first few months and $79/month banner ad on a web stats site bought via BuySellAds.com.

Used SnapEgage chat widget on the website to talk visitors to sign up.

Word of mouth and referrals started quickly after that.

Now referrals are biggest signup drivers.

13. VNC application for Mac and iOS

Sells for $30 on Mac and $20 on iOS.

Source:

14. NomadList, $400k/year revenue

Source:

Revenue source: membership fees for community of digital nomads and remote workers.

15. B2B Windows desktop app, $1 million/year sales

Source: https://news.ycombinator.com/item?id=12066104

Wrote scrach-my-itch app, was side project for 10 years until it started making $120k/mo. Went full time after that. Is in a very crowded niche.

Insight: coding is easy, marketing is hard. Must be persistent.

16. Pinboard, bookmarking web service, $200k/year

Source:

Revenue history: https://blog.pinboard.in/2016/07/pinboard_turns_seven/

17. s3stat, “equivalent of a nice Senior Developer salary”

Source:

18. StoreSlider, $700k in 2016

Source: https://news.ycombinator.com/item?id=14438303

Makes money with affiliate revenue from eBay.

Built with Lumen on PHP 7.1, Nginx, running on Linode.

Source of traffic: word of mouth, social sharing, Google search.

Did a lot of A/B testing to maximize conversion.s.

19. BuiltiWith.com, estimated $12 million/year

Source:

20. tarsnap, backup service, “better than Google salary”

Source: https://news.ycombinator.com/item?id=14442425

21. Sidekiq, $1 million/year

Source: https://www.indiehackers.com/businesses/sidekiq

Open source library for Ruby, a background job framework.

Sells pro version for $950/year and enterprise version. Only needs 800 customer

22. Balsamiq, $2 millions in revenue after 18 months

Source:

It’s no longer a single person but it was created in 2008 by a single person and within 18 monts reached $2 million in revenue.

It’s a desktop app for creating mockups.

23. John Gruber, $32k/mo

Source: https://daringfireball.net/feeds/sponsors/

Makes $8k per week for sponsorship (ads) on his very popular, Apple-oriented website.

24. Sales tracking & CRM app for small business.

Source: https://news.ycombinator.com/item?id=14439284

https://www.bottomlinehq.com/, 6-digit revenue.

Freemium model, $30/year. Web and iOS.

25. https://officesnapshots.com/, full-time salary.

Source: https://news.ycombinator.com/item?id=12065574

Started 9 years ago, full-time for last 4 years.

Revenue: AdSense and later selling his own advertising (ads are sold as $/month and sold in blocks of 1 to 12 months.

26. pubexchange.com

Source:

Stated in 2013, operated solo since then. Profitable since 2014.

In early days pitched his service on linkedin, now people come from referrals.

27. park.io, $125k/mo

Source:

Started in 2014. In his spare time he wrote a script to auto-buy domain when it expires and turned that into paid service by adding registration, payments etc.

Most users find it from either parked domains or word of mouth.

Insight: automate all the things.

28. cronitor, $6k/mo revenue after 3 years

Source:

Started in 2014, written part-time by 2 people. They wrote it because it solved a real problem they had at a startup he worked at.

Marketing tactics:

did ShowHN
answered questions on StackOverflow
added a link from a popular, open-source PHP library they had
created Stackshare page
submitted to startupli.st (site defunct)
submitted to “One Thing Well” website
wrote high-quality docs for SEO (topic-based articles on ‘how to use cronitor to do X’)

Raised prices after 6 months from $7/$20/$50 => $10/$25/$50 and then $24/$70/$150.

29. bugmuncher, $4k/mo revenue

Source:

Web-based bug tracking software.

Started as a side project in 2010, went full time in Nov 2015, reached living wage in Nov 2016.

30. https://info-beamer.com, close to living wage

Source: https://news.ycombinator.com/item?id=13514865

Digital signage for Raspberry PI. Started as a side project, turned into profitable business.

31. Anonymous app, $5k/mo profit on $7k/mo revenue

Source: https://news.ycombinator.com/threads?id=gaeappthrowaway

App hosted on App Engine, ~50 users paying between $30/mo and $500/mo. Analytics API.

32. Wordpress theme, $5k/mo

Source: https://news.ycombinator.com/item?id=8107836

Sold via ThemeForest.

33. Radio Silence, main income

Source:

Mac app (https://radiosilenceapp.com/) that evolved from side project to providing main income for the author. Sells for $9.

Author was able to quit his job.

Business tip: build related free app and host on the same domain.

34. Ryan Clark, 10 Games, $3+ million over 10 years

Source: http://www.gamasutra.com/blogs/RyanClark/20150917/253842/What_Makes_an_Indie_Hit_How_to_Choose_the_Right_Design.php

Working full time since 2004 on his games. Wrote 10 games in 11 years. 8 been profitable, 3 grossed more than $1M.

35. https://PhantomJsCloud.com, ramen profitable for Seattle

Source: https://news.ycombinator.com/item?id=13327835

36. Desktop app, seating planning, $120k+/year

Source:

2 desktop apps for Windows, written in C++.

Over 10 years, sold 40 thousand licenses of first desktop app, the cheapest is $30, which is at least $120k/year.

37. Desktop app in construction industry, making a living

Source: https://news.ycombinator.com/item?id=11659140

21-year old app for Windows, written in Delphi 5.

38. Video games, making a living

Source:

Multi-platform games written in C#, based on Unity game engine, released on Steam. Makes a game every X months.

39. Pinegrow Web Editor, comfortable living

Source:

Desktop web editor built with NWJS/Electron. Started by a single person, grew to 3 full-time people.

Launched in January 2014 after 2.5 years in development, sold $100k the first year. Sells for $49/$79.

Marketing: website and asking for e-mail address when starting the trial to build e-mail database to send promotions to.

Tried Carbon, Google and Reddit ads but was losing money on them.

40. https://ipinfo.io, full-time job

Source:

Wrote and launched in a couple of hours as a response to StackOverflow question about. Posted as a response, forgot about it, it became popular so he implemented paid plans and started charging for it.

41. http://duetapp.com, $3-4k/mo

Self-hosted, web-based invoicing and project management app.

Source: https://news.ycombinator.com/item?id=8630931

42. https://betterexplained.com, $6k/mo after 10 years

A website with math tutorials. Started in 2006. Content is free. Makes money selling ebooks (on Amazon kindle and directly from the website), amazon affiliate links and newsletter sponsorships.

Source:

43. Website with special-interest news, $!0-15k/mo

Money from AdSense.

Source: https://news.ycombinator.com/item?id=8630369

44. $5.5k/mo from Udemy course

Source: https://news.ycombinator.com/item?id=8631035

45. Storemapper, $21k/mo

Source:

46. Brendan Dunn, $451k revenue in 2014 from several products

Source:

His income:

https://doubleyourfreelancing.com/rate/, online course, $207k
http://doubleyourfreelancing.com/leads/, online course, $40k
https://planscope.io/, SaaS app, $71k
http://buildaconsultancy.com/, live classes, $44k from 3 live classes
consulting and coaching, $89k from 6 weeks of consulting

47. Cooking blog, $5-6k/mo

He does the design/programming/marketing/monetization work behind http://www.theyummylife.com, his mother does the writing. Revenue from Amazon affiliates, ads and ebook.

Source:

48. Few dozens entertainment-related websites, $122k/mo

Revenue from AdSense, 8 million monthly uniques, after 6 years.

Source: https://news.ycombinator.com/item?id=4468067

49. https://www.tiki-toki.com/, $5k/mo

Web and desktop app for creating pretty timelines. Makes money from premium accounts and selling desktop app.

Source: https://news.ycombinator.com/item?id=4469672

50. Webapp in education space, $90k/m

Revenue: AdSsense.

Source: https://news.ycombinator.com/item?id=4468535

51. Large web community, $90-110k/mo

Revenu from subscription, adsense, other ad revenue, license and royalty revenue

Source: https://news.ycombinator.com/item?id=2567487

52. Zencaster, $12k/mo

Zencaster is a web-based tool that helps podcasters record their guests in studio quality.

Source:

53. Workflowy, $800k/year

Jesse Patel learn how to program building https://workflowy.com/. After 9 months of working on it alone, he asked a friend he knew from college to join him. They got into YC to work on a different idea but pivoted back to Workflowy.

They started charging 2 years after they launched and got enough revenue to pay for living expenses.

They have 100k paying users and $800k/year revenue.

Source:

https://www.indiehackers.com/podcast/037-jesse-patel-of-workflowy

Advanced command execution in Go with os/exec

2017-06-23T09:43:49Z

Go has excellent support for executing external programs. Let’s start at the beginning.

In our examples we'll be running ls -lah command as it produces an output. There is no ls on Windows so you can change that to e.g. tasklist.

Running a command

	cmd := exec.Command("ls", "-lah")
	if runtime.GOOS == "windows" {
		cmd = exec.Command("tasklist")
	}
	err := cmd.Run()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}

Full example at https://codeeval.dev/gist/8e5b7808e6a21f3d9d70e844646b55de

If you run it, nothing seems to happen. Fear not, the command has actually been executed.

If we were running ls -lah in the shell, the shell would copy programs' stdout and stderr to console, so that we can see it.

We're executing the program via Go standard library function and by default stdout and stderr are discarded.

Running a command and showing output

To let the human see the output, we can connect the output (cmd.Stdout and cmd.Stderr) of the program we're executing to os.Stdout and os.Stderr, which is the output of our program:

	cmd := exec.Command("ls", "-lah")
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	err := cmd.Run()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}

Full example at https://codeeval.dev/gist/08e519d3d47a6d82ca289ccff53e28b7

cmd.Stdout and cmd.Stderr are declared as io.Writer interface so we can set them to any type that implements Write() method, like os.File or an in-memory buffer bytes.Buffer.

io.Reader and io.Writer are very simple, yet very powerful, abstractions.

Running a command and capturing the output

The above examples allows human to see the output but sometimes we want to capture the output and analyze it:

func main() {
    cmd := exec.Command("ls", "-lah")
    out, err := cmd.CombinedOutput()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    fmt.Printf("combined out:\n%s\n", string(out))
}

Full example at https://codeeval.dev/gist/4dfceac2f8229a077595c6cae84b3a56

CombinedOutput runs a command and returns combined stdout and stderr.

Behind the scenes of `CombinedOutput`

The good thing about Go is that it's open source so we can peek at how a given functionality is implemented.

Another good thing is that most of the code in the standard library is simple. Here's how CombinedOutput is implemented:

func (c *Cmd) CombinedOutput() ([]byte, error) {
	if c.Stdout != nil {
		return nil, errors.New("exec: Stdout already set")
	}
	if c.Stderr != nil {
		return nil, errors.New("exec: Stderr already set")
	}
	var b bytes.Buffer
	c.Stdout = &b
	c.Stderr = &b
	err := c.Run()
	return b.Bytes(), err
}

Notice that it's almost as simple as our second example. Instead of setting cmd.Stdout and cmd.Stderr to standard output, we set them to a single in-memory buffer.

When program finishes, we returned everything written to that buffer.

Don't be afraid to peruse the code of standard library.

Capture stdout and stderr separately

What if you want to do the same but capture stdout and stderr separately?

func main() {
    cmd := exec.Command("ls", "-lah")
    var stdout, stderr bytes.Buffer
    cmd.Stdout = &stdout
    cmd.Stderr = &stderr
    err := cmd.Run()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    outStr, errStr := string(stdout.Bytes()), string(stderr.Bytes())
    fmt.Printf("out:\n%s\nerr:\n%s\n", outStr, errStr)
}

Full example at https://codeeval.dev/gist/8d5cf19aa518c45f6ea8e35ae5250cda

Capture output but also show progress

What if the command takes a long time to finish?

It would be nice to see its progress on the console as it happens in addition to capturing stdout/stderr.

It’s a little bit more involved, but not terribly so.

First, a helper function that copies from reader to a writer and also captures copied data:

func copyAndCapture(w io.Writer, r io.Reader) ([]byte, error) {
    var out []byte
    buf := make([]byte, 1024, 1024)
    for {
        n, err := r.Read(buf[:])
        if n > 0 {
            d := buf[:n]
            out = append(out, d...)
            _, err := w.Write(d)
            if err != nil {
                return out, err
            }
        }
        if err != nil {
            // Read returns io.EOF at the end of file, which is not an error for us
            if err == io.EOF {
                err = nil
            }
            return out, err
        }
    }
}

Handling errors from Read is subtle. An error io.EOF means that we've read everything. It's not an actual error so we turn io.EOF into nil.

The meat of the code:

func main() {
  cmd := exec.Command("ls", "-lah")

	var stdout, stderr []byte
	var errStdout, errStderr error
	stdoutIn, _ := cmd.StdoutPipe()
	stderrIn, _ := cmd.StderrPipe()
	err := cmd.Start()
	if err != nil {
		log.Fatalf("cmd.Start() failed with '%s'\n", err)
	}

	// cmd.Wait() should be called only after we finish reading
	// from stdoutIn and stderrIn.
	// wg ensures that we finish
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		stdout, errStdout = copyAndCapture(os.Stdout, stdoutIn)
		wg.Done()
	}()

	stderr, errStderr = copyAndCapture(os.Stderr, stderrIn)

	wg.Wait()

	err = cmd.Wait()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}
	if errStdout != nil || errStderr != nil {
		log.Fatal("failed to capture stdout or stderr\n")
	}
	outStr, errStr := string(stdout), string(stderr)
	fmt.Printf("\nout:\n%s\nerr:\n%s\n", outStr, errStr)
}

Full example at https://codeeval.dev/gist/7cca5f1526b2785e03febc16542b96ea

We have two outputs to copy. To avoid serializing them, we'll read one in a goroutine.

As documentation of StdoutPipe warns, Wait will close the pipes when the process finishes. This might lead to losing some output if we haven't finished reading it.

To prevent that we use sync.WaitGroup to ensure that the gorutine handling os.Stdout finishes reading before we call cmd.Wait.

I encourage you to read the implementation of cmd.StdoutPipe. You'll be surprised by how short it is.

Capture output but also show progress #2

Previous solution works but copyAndCapture looks like we’re re-implementing io.Copy. Thanks to Go’s use of interfaces we can re-use io.Copy.

We’ll write CapturingPassThroughWriter struct implementing io.Writer interface. It’ll capture everything that’s written to it and also write it to underlying io.Writer.

// CapturingPassThroughWriter is a writer that remembers
// data written to it and passes it to w
type CapturingPassThroughWriter struct {
    buf bytes.Buffer
    w io.Writer
}

// NewCapturingPassThroughWriter creates new CapturingPassThroughWriter
func NewCapturingPassThroughWriter(w io.Writer) *CapturingPassThroughWriter {
    return &CapturingPassThroughWriter{
        w: w,
    }
}

func (w *CapturingPassThroughWriter) Write(d []byte) (int, error) {
    w.buf.Write(d)
    return w.w.Write(d)
}

// Bytes returns bytes written to the writer
func (w *CapturingPassThroughWriter) Bytes() []byte {
    return w.buf.Bytes()
}

func main() {
    cmd := exec.Command("ls", "-lah")

		var errStdout, errStderr error
		stdoutIn, _ := cmd.StdoutPipe()
		stderrIn, _ := cmd.StderrPipe()
		stdout := NewCapturingPassThroughWriter(os.Stdout)
		stderr := NewCapturingPassThroughWriter(os.Stderr)
		err := cmd.Start()
		if err != nil {
			log.Fatalf("cmd.Start() failed with '%s'\n", err)
		}
	
		var wg sync.WaitGroup
		wg.Add(1)
	
		go func() {
			_, errStdout = io.Copy(stdout, stdoutIn)
			wg.Done()
		}()
	
		_, errStderr = io.Copy(stderr, stderrIn)
		wg.Wait()
	
    err = cmd.Wait()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    if errStdout != nil || errStderr != nil {
        log.Fatalf("failed to capture stdout or stderr\n")
    }
    outStr, errStr := string(stdout.Bytes()), string(stderr.Bytes())
    fmt.Printf("\nout:\n%s\nerr:\n%s\n", outStr, errStr)
}

Full example at https://codeeval.dev/gist/2e4acc63f4f67eafc9c9a30776c01a22

Capture output but also show progress #3

Turns out Go’s standard library implements io.MultiWriter, which is a more generic version of CapturingPassThroughWriter. Let’s use that instead:

func main() {
	cmd := exec.Command("ls", "-lah")
	if runtime.GOOS == "windows" {
		cmd = exec.Command("tasklist")
	}

	var stdoutBuf, stderrBuf bytes.Buffer
	cmd.Stdout = io.MultiWriter(os.Stdout, &stdoutBuf)
	cmd.Stderr = io.MultiWriter(os.Stderr, &stderrBuf)

	err := cmd.Run()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}
	outStr, errStr := string(stdoutBuf.Bytes()), string(stderrBuf.Bytes())
	fmt.Printf("\nout:\n%s\nerr:\n%s\n", outStr, errStr)
}

Full example at https://codeeval.dev/gist/0a1c2b1df8f675c86429768ae496c6f0

It’s good to be able to write the code ourselves, but it’s even better to know standard library well!

Writing to program’s stdin

We know how to read program’s stdout but we can also write to its stdin.

There is no Go library to do bzip2 compression (only decompression is available in standard library).

We can use bzip2 to do the compression by: * writing the data to a temporary file * call bzip2 -c ${file_in} and capture its stdout

It would be even better if we didn’t have to create a temporary file.

Most compression programs accept data to compress/decompress on stdin.

To do that on command-line we would use the following command: bzip2 -c <${file_in} >${file_out}.

Here’s the same thing in Go:

// compress data using bzip2 without creating temporary files
func bzipCompress(d []byte) ([]byte, error) {
    var out bytes.Buffer
    // -c : compress
    // -9 : select the highest level of compresion
    cmd := exec.Command("bzip2", "-c", "-9")
    cmd.Stdin = bytes.NewBuffer(d)
    cmd.Stdout = &out
    err := cmd.Run()
    if err != nil {
        return nil, err
    }
    return out.Bytes(), nil
}

Full example at https://codeeval.dev/gist/1d572914e41d8d88f9ec92230dae25b7

We can also call cmd.StdinPipe(), which returns io.WriteCloser. It’s more complicated but gives more control over writing.

Changing environment of executed program

Things to know about environment variables in Go:

os.Environ() returns []string where each string is in form of FOO=bar, where FOO is the name of environment variable and bar is the value
os.Getenv("FOO") returns the value of environment variable FOO.

Sometimes you need to modify the environment of the executed program.

Go supports that by setting Env member of exec.Cmd. cmd.Env has the same format as os.Environ().

If Env is not set, the process inherits environment of the calling process.

Usually, you don’t want to construct a completely new environment from scratch but pass a modified version of an environment of the current process. Here's how to add a new variable:

		cmd := exec.Command("programToExecute")

    additionalEnv := "FOO=bar"
    newEnv := append(os.Environ(), additionalEnv))
    cmd.Env = newEnv

    out, err := cmd.CombinedOutput()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    fmt.Printf("%s", out)

Full example

Things get more complicated if you want to delete an environment or to ensure you're not setting the same variable twice. Package shurcooL/go/osutil offers an easier way of manipulating environment variables.

Check early that a program is installed

Imagine you wrote a program that takes a long time to run. At the end, you call executable foo to perform an essential task.

If foo executable is not present, the call will fail.

It’s a good idea to detect lack of executable foo at the beginning and fail early with descriptive error message.

You can do it using exec.LookPath.

func checkLsExists() {
    path, err := exec.LookPath("ls")
    if err != nil {
        fmt.Printf("didn't find 'ls' executable\n")
    } else {
        fmt.Printf("'ls' executable is in '%s'\n", path)
    }
}

Full example at https://codeeval.dev/gist/2a80bc3e48d7f36f3299367660a0aedd

Another way to check if a program exists is to try to execute it in a no-op mode (e.g. many programs support --help option).

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/advanced-exec

Generating good unique ids in Go

2017-06-21T18:47:07Z

Imagine you’re writing a note taking application.

Each note needs a unique id.

Generating unique ids is easy if you can coordinate.

The simplest way is to get database to do it: use AUTOINCREMENT column and the database will generate unique id when you insert a new note row into a table.

What if you can’t coordinate?

For example, you want the app to also generate unique note id when offline, when it cannot talk to the database.

The requirement of non-coordinated generation of unique ids comes up often in distributed systems.

A simple solution is to generate a random id. If you give it 16 bytes of randomness, the chances of generating the same random number are non-existent.

It’s such a common problem that over 30 years ago we created a standard for this called UUID/GUID.

We can do better than GUID. A good random unique id:

is unique; we can’t skip the basics
can be sorted by its string representation
is time-clustered i.e. ids generated at the same time are close to each other when sorted
string representation can be used as part of URL without escaping
the shorter, the better

There are few Go implementation of such id, following the same basic idea:

use time as part of the id to achieve time-clustering
fill rest of the id with random data
encode as a string in a way that allows lexicographic sorting and is url-safe

Here are Go packages for generating unique id and how their ids look like in string format:

GUID libraries

Package	Id	Format
github.com/segmentio/ksuid	`0pPKHjWprnVxGH7dEsAoXX2YQvU`	4 bytes of time (seconds) + 16 random bytes
github.com/rs/xid	`b50vl5e54p1000fo3gh0`	4 bytes of time (seconds) + 3 byte machine id + 2 byte process id + 3 bytes random
github.com/kjk/betterguid	`-Kmdih_fs4ZZccpx2Hl1`	8 bytes of time (milliseconds) + 9 random bytes
github.com/sony/sonyflake	`20f8707d6000108`	~6 bytes of time (10 ms) + 1 byte sequence + 2 bytes machine id
github.com/oklog/ulid	`01BJMVNPBBZC3E36FJTGVF0C4S`	6 bytes of time (milliseconds) + 8 bytes random
github.com/chilts/sid	`1JADkqpWxPx-4qaWY47~FqI`	8 bytes of time (ns) + 8 random bytes
https://github.com/lithammer/shortuuid	`dwRQAc68PhHQh4BUnrNsoS`	UUIDv4 or v5, encoded in a more compact way
github.com/satori/go.uuid	`5b52d72c-82b3-4f8e-beb5-437a974842c`	UUIDv4 from RFC 4112 for comparison
https://github.com/google/uuid	`c01d7cf6-ec3f-47f0-9556-a5d6e9009a43`	UUIDv4

You can see how the values change over time by refreshing this test page a couple of times.

How to generate unique ids using different libraries:

import (
	"fmt"
	"log"
	"math/rand"
	"time"

	"github.com/chilts/sid"
	guuid "github.com/google/uuid"
	"github.com/kjk/betterguid"
	"github.com/lithammer/shortuuid"
	"github.com/oklog/ulid"
	"github.com/rs/xid"
	"github.com/satori/go.uuid"
	"github.com/segmentio/ksuid"
	"github.com/sony/sonyflake"
)

func genShortUUID() {
	id := shortuuid.New()
	fmt.Printf("github.com/lithammer/shortuuid: %s\n", id)
}

func genUUID() {
	id := guuid.New()
	fmt.Printf("github.com/google/uuid:         %s\n", id.String())
}

func genXid() {
	id := xid.New()
	fmt.Printf("github.com/rs/xid:              %s\n", id.String())
}

func genKsuid() {
	id := ksuid.New()
	fmt.Printf("github.com/segmentio/ksuid:     %s\n", id.String())
}

func genBetterGUID() {
	id := betterguid.New()
	fmt.Printf("github.com/kjk/betterguid:      %s\n", id)
}

func genUlid() {
	t := time.Now().UTC()
	entropy := rand.New(rand.NewSource(t.UnixNano()))
	id := ulid.MustNew(ulid.Timestamp(t), entropy)
	fmt.Printf("github.com/oklog/ulid:          %s\n", id.String())
}

func genSonyflake() {
	flake := sonyflake.NewSonyflake(sonyflake.Settings{})
	id, err := flake.NextID()
	if err != nil {
		log.Fatalf("flake.NextID() failed with %s\n", err)
	}
	// Note: this is base16, could shorten by encoding as base62 string
	fmt.Printf("github.com/sony/sonyflake:      %x\n", id)
}

func genSid() {
	id := sid.Id()
	fmt.Printf("github.com/chilts/sid:          %s\n", id)
}

func genUUIDv4() {
	id, err := uuid.NewV4()
	if err != nil {
		log.Fatalf("uuid.NewV4() failed with %s\n", err)
	}
	fmt.Printf("github.com/satori/go.uuid:      %s\n", id)
}

Full example: generate-unique-id/main.go

Which one to use?

All of them are good.

I would pick either rs/xid or segmentio/ksuid.

oklog/ulid allows custom entropy (randomness) source but pays for that with complex API.

sony/sonyflake is the smallest but also the least random. It’s based on Twitter’s design for generating IDs for tweets.

For simplicity the example code serializes sony/snoflake in base16. It would be even shorter in base62 encoding used by other libraries, but other libraries provide that out-of-the-box and for sony/snoflake I would have to implement it myself.

The last one is UUID v4 from RFC 4112, for comparison.

To learn more:

https://segment.com/blog/a-brief-history-of-the-uuid/ : the history of ksuid
https://firebase.googleblog.com/2015/02/the-2120-ways-to-ensure-unique_68.html : inspiration for betterguid design
http://antoniomo.com/blog/2017/05/21/unique-ids-in-golang-part-1/
http://antoniomo.com/blog/2017/05/28/unique-ids-in-golang-part-2/
http://antoniomo.com/blog/2017/06/03/unique-ids-in-golang-part-3/
https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake.html : details of Twitter’s snowflake on which sonyflake is based

Code for this chapter: https://github.com/kjk/go-cookbook/tree/master/generate-unique-id