Krzysztof Kowalczyk blog

15minutes for mac now available

2009-06-23T02:37:12Z

My simple productivity application 15minutes used to only run on Windows. I just finished writing Mac OS X version so a blog post announcing it seems appropriate.

Cocoa and Objective-C reference

2009-06-15T03:00:14Z

I’ve started programming in Cocoa again. I’m only an advanced beginner in that framework so I often have to look up how to achieve even simple things. To make programming a little bit faster, I started working on a Cocoa quick reference.

It lists the most frequently used methods and classes in Cocoa, with links to official Apple docs.

On a side note, I think that existing quick reference guides on the web aren’t very good. I have a few ideas on how they could be improved and hope to use those ideas in my Cocoa reference.

Shared http caching

2009-06-13T21:27:24Z

In March I wrote about an idea for tagging links with their sha1 checksum for better caching in web browser. Now I’m not alone.

Network drives, .net, security and virtualbox

2009-06-05T06:28:28Z

I have Windows 7 installed on my MacBook under VirtualBox. To keep things simple, I keep my code on mac side, sharing the directory with Windows side.

Tip #1: VirtualBox 2.2.4 has weak implementation of shared folders. For example, Visual Studio 2008 sees the files on shared drive as write-protected (even though other applications don’t). Visual Studio 2010 doesn’t have this problem, but it can’t finish compilation, complaining it cannot write some intermediary build files to a shared folder. Also, trying to mark shared folder as trustworthy for .NET doesn’t seem to work (see below).

Solution: use Mac’s built-in file sharing instead and shared things out using built-in samba (smb) sharing in System Preferences/Sharing.

Another problem is that .NET by default doesn’t trust code on network drives and Visual Studio will tell you that “The Project Location is Not Trusted” if you open a project from a network drive. The magic incantation to make a given shared folder (in my case it’s drive z:) trusted is:

caspol -q -machine -addgroup 1.2 -url file://z:/* FullTrust

This command has to be executed as Administrator.

One way to have caspol.exe available in the PATH is to start Visual Studio cmd-line (shortcut is installed in Start menu).

Setting unicode rtf text in rich edit control

2009-04-15T06:36:41Z

Despite being around for thousands of years, win32 API is poorly documented in many areas.

Today I spent an hour battling a baffling problem with rich edit control.

I was setting rtf-formatted text using WM_SETTEXT message.

In non-unicode build, where text was ansi, it worked and the text showed up nicely formatted.

In unicode build, however, the text wasn’t parsed as rtf and showed verbatim.

An hour of trying various ways to make it work ended with me finding a working work-around: using EM_SETTEXTEX and utf-8 encoded text with CP_UTF8 code-page. I’m using WTL so the code ended up being:

#ifdef UNICODE
		// Don't know why I have to do this, but SetWindowText() with unicode
		// doesn't work (rtf codes are not being recognized)
		const char *sUtf = WstrToUtf8(s);
		m_statusMsgEdit.SetTextEx((LPCTSTR)sUtf, ST_DEFAULT, CP_UTF8);
#else
		m_statusMsgEdit.SetWindowText(s);
#endif

MSDN claims I could have used code-page 1200 for unicode text, but that didn’t work either.

It’s quite possible I’ve botched something somewhere in my code and this work-around isn’t needed but I can hardly think of anything. If you ever run into problem like this, the above is one thing to try.

Accessing Mac file shares from Windows 7

2009-04-11T22:40:54Z

Windows 7 beta 7000 can’t access files shared from Mac through built-in samba file sharing out of the box. It’s rather infuriating, because XP has no such problem and the error message given by windows when you try to navigate to a folder shared from mac is cryptic and not helpful at all. Turns out the culprit is purely how Windows 7 security is configured by default. Good news is it can be fixed via configuration tweaks as described here.

In case this post goes away, here’s what it says:

Windows 7 will not work with Mac OS X Windows file sharing support by default. If you attempt to access a folder shared from Mac OS X, Vista will display a logon error repeatedly. The problem is that Vista, by default, will only use NTLMv2 for authentication, which is not supported by Mac OS X’s Windows Sharing service. The other problem is the Minimum Session Security for NTVLM SSP based Clients.
To get around this:

1. In Vista, open the Control Panel
2. Switch to “Classic” view
3. Double-click Administration Tools
4. Double-click Local Security Policy
5. Or Secpol.msc
6. Expand “Local Policies” and select “Security Options”
7. Alternate : Type secpol.msc to get editor up then
8. Locate “Network Security: LAN Manager Authentication Level” in the list and double-click it.
9. Change the setting from “Send NTMLv2 response only” to “Send LM & NTLM – use NTLMv2 session if negotiated”
10. Network Security: Minimum session security for NTLM SSP Based (including secure RPC) Clients
11. Change the setting from “require 128 bit” to unchecked (No Minimum)
12. Click OK

the real difference between vista and windows 7 procedure is 10 and 11

It’s rather evil for Microsoft to choose defaults that make file sharing not interoperable with latest Mac OS X. It’s not like they didn’t test it so it must have been a conscious decision to screw with Mac users.

15minutes - a simple productivity tool

2009-04-09T05:48:04Z

15minutes is a simple productivity tool I wrote.

One of productivity techniques is time boxing. Pomodoro technique is a similar idea.

Procrastination is avoiding work. To avoid procrastination you just have to start working.

One reason we avoid work is because the tasks seem big and overwhelming. Time boxing is a mental trick – instead of thinking about doing the whole task, we focus about working on a task for a fixed, short period of time.

To implement time boxing you need a timer which will alert you about the end of time boxing period.

Any timer will do – a physical timer, iPhone’s built-in clock etc.

I wrote a small application to do it, since most of my work is done on a computer. 15minutes is that application.

Recent mupdf work

2009-04-06T05:21:28Z

I’ve been working a little bit on Sumatra and mupdf recently. The result of which are 9 patches submitted to mupdf upstream.

Valgrind on mac

2009-03-28T04:04:36Z

As of Mar 27 2009 Valgrind doesn’t officially support mac but a port that is good enough is in progress. You can learn more and monitor the progress by reading Nicholas Nethercote’s blog (main person working on Valgrind mac port).

This post is a summary of how I built Valgrind on Mac (OS X 10.5.6).

Since this involves compiling, you probably need developer tools. I have both XCode and a healthy does of MacPorts dev packages installed. I’m not sure how much you need, but I’m guessing at least gcc, make, autotools.

svn co svn://svn.valgrind.org/valgrind/trunk valgrind
cd valgrind
./autogen.sh
./configure
make
sudo make install

So far I’ve only tried Valgrind on mupdf and it worked, so this is very promising. Valgrind is invaluable tool for C/C++ memory leak detection, detecting memory mismanagement issues and profiling the code.

An unfortunate deficiency is lack of a GUI for inspecting callgrind results (i.e. kcachegrind equivalent on mac). Maybe the cheapest way to get that would be to convert callgrind results to shark format so that shark tools can be used. Should be possible – I believe WebKit’s JavaScript implementation saves profiling results in shark format, so there is code that could be used as a template for that.

Unladden-swallow - making Python faster

2009-03-28T03:02:47Z

When I’m right, I’m right. About a year ago I said that Google should sponsor faster Python VM. And that’s exactly what they did by having 2 full time engineers working on unladden-swallow, a project to make Python much faster, with full intention to get all their improvements into mainline Python.

The project looks very promising.

It also shows that small teams can do great things. 2 people team is as small as it gets and if they manage to achieve their goal, they’ll make a big difference in many programmer’s lives.

Interesting mac source code

2009-03-16T03:06:44Z

Interesting mac sourcecode:

Understudy – GPL, front row plugin that adds support for Hulu, Netflix, YouTube.

HTTP Client – BSD, UI for sending HTTP requests and inspecting HTTP requests/responses

MacShark, network sniffer, nmap, dig, whois etc., New BSD

Eavesdrop, GPLv2, network sniffer

hexfiend, BSD, hex editor

iLife 08-like source list, New BSD, ui widget

AppHider – moves apps from doc to menubar

RoaringDiff – graphical diff

ReaderNotifier – shows Google Reader status in menu bar

Quick Search Box – from Google

MenuMeters – GPL, shows stuff in menu bar

Http upload sample

Silverlight info

2009-03-14T23:16:43Z

http://blogs.msdn.com/jstegman/archive/2008/04/21/dynamic-image-generation-in-silverlight.aspx has code to dynamically generate bitmaps in Silverlight.

FJCore – Jpeg decoder and encoder + resizing, in just C#

Interesting win32 source code

2009-03-14T23:14:12Z

Interesting win32 source code:

puttycyg – cygwin terminal, putty modified to also be a cygwin terminal, MIT
putty, MIT, putty Tray, enhancements to Putty
qemacs, LGPL
WTL
WDL – Justin Frankel’s library for win32 & win32 emulation for mac, BSD-like
TaskSwitchXP – BSD-like (?)
stb-img, GPL with parts in public domain (zlib, png, jpeg decompressors)
qsniffer, GPLv2, network sniffer in qt, cross-platforms
msnwcrec – msn webcam recorder, GPLv2
orionsniffer, tcp/ip packet sniffer, GPLv2, uses winpcap
ansi codes in cmd.exe, article with source code about patching cmd.exe
very sleepy, GPL, simple profiler for windows
FileZill, GPL, C++

Debugging/rec-related code:

opendbg, GPLv3
leaktrap, GPL, gdi/user handle leak tracking and analysis, hooks win32 apis
cdbg, GPLv3, console user-mode debugger
ngdbg, GPLv3, live kernel-mode debugger
openrce-snippets, MIT,
pefile, MIT, Python library to work with pe format
paimei, GPLv2, reverse engineering framework in pyton
vmachine, PC emulator

Debugging,rec tools:

peid, unpacker, decryptor of pe files for win, also has process viewer, pe details, disassembler etc.

PDF-related source code:

openjpeg, New BSD, JPEG 2000 in C

Forcing basic http authentication for HttpWebRequest (in .NET/C#)

2009-03-14T23:12:55Z

HttpWebRequest is a handy .NET class for doing HTTP requests. It has built-in support for HTTP basic authentication via credentials. However, it doesn’t work the way I expected: supplying credentials doesn’t send Authorization HTTP header with the request but only in response to server’s challenge. It often breaks in real world, where servers might not issue a challenge and simply not authenticate a request.

Fortunately fixing it by manually adding Authorization HTTP header to the request is simple and this code snippet shows how to do it:

public void SetBasicAuthHeader(WebRequest req, String userName, String userPassword)
{
    string authInfo = userName + ":" + userPassword;
    authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo));
    req.Headers["Authorization"] = "Basic " + authInfo;
}

WTL resources

2009-03-14T22:56:22Z

WTL programming resources:

WTL articles on codeproject.com
Threads for WTL
INET for WTL
WTL for MFC: ATL GUI Classes
WTL for MFC: WTL GUI Base Classes
WTL for MFC: toolbars and status bars
WTL for MFC: dialogs and controls
WTL for MFC: advanced dialog ui classes
WTL for MFC: Hosting ActiveX controls
WTL for MFC: Splitter Windows
WTL for MFC: Property Sheets and Wizards
WTL for MFC: GDI Classes, common dialogs and utility classes
WTL for MFC: drag & drop source
Using WTL‘s Built-in Dialog Resizing Class
10 WTL tips (window creation size, centering a window, setting min/max size of a window, dialog text and background color, dynamically swap buttons, default font)
Creating dialogs at runtime
Saving window state
Sizing framework based on WinMgr sizing article

scdiff update (Windows git/subversion/cvs gui diff previewer)

2009-03-12T04:40:44Z

I’ve updated my little scdiff program and added git support.

If you work with git, Subversion or CVS from command line, scdiff is a handy program to preview your changes before committing them.

It works by creating before/after files in a temporary directory and launching a diff viewer (WinDiff by default, but can be configured) to show the diffs.

It auto-detects which scm is being used in a given directory, so you launch it by simply typing scdiff.

It requires git/svn/cvs executable be in the path. There still is no decent windows build of git, so I use cygwin to get git.

Parsing s3 log files in python

2009-03-08T23:54:37Z

After we took care of compacting s3 logs, it’s time to parse them. s3 log format is well documented and can be parsed with a simple regular expression.

It took me some time to craft it, so here it is. You can also view the source here

#!/usr/bin/env python
import re

s3_line_logpats  = r'(\S+) (\S+) \[(.*?)\] (\S+) (\S+) ' \
           r'(\S+) (\S+) (\S+) "([^"]+)" ' \
           r'(\S+) (\S+) (\S+) (\S+) (\S+) (\S+) ' \
           r'"([^"]+)" "([^"]+)"'

s3_line_logpat = re.compile(s3_line_logpats)

(S3_LOG_BUCKET_OWNER, S3_LOG_BUCKET, S3_LOG_DATETIME, S3_LOG_IP,
S3_LOG_REQUESTOR_ID, S3_LOG_REQUEST_ID, S3_LOG_OPERATION, S3_LOG_KEY,
S3_LOG_HTTP_METHOD_URI_PROTO, S3_LOG_HTTP_STATUS, S3_LOG_S3_ERROR,
S3_LOG_BYTES_SENT, S3_LOG_OBJECT_SIZE, S3_LOG_TOTAL_TIME,
S3_LOG_TURN_AROUND_TIME, S3_LOG_REFERER, S3_LOG_USER_AGENT) = range(17)

s3_names = ("bucket_owner", "bucket", "datetime", "ip", "requestor_id", 
"request_id", "operation", "key", "http_method_uri_proto", "http_status", 
"s3_error", "bytes_sent", "object_size", "total_time", "turn_around_time",
"referer", "user_agent")

def parse_s3_log_line(line):
    match = s3_line_logpat.match(line)
    result = [match.group(1+n) for n in range(17)]
    return result

def dump_parsed_s3_line(parsed):
    for (name, val) in zip(s3_names, parsed):
        print("%s: %s" % (name, val))

def test():
    l = r'607c4573f2972c26aff39f7e56ff0490881a35c19b9bf94072cbab8c3219f948 kjkpub [06/Mar/2009:23:13:28 +0000] 41.221.20.231 65a011a29cdf8ec533ec3d1ccaae921c C46E93FF2E865AC1 REST.GET.OBJECT sumatrapdf/rel/SumatraPDF-0.9.1.zip "GET /sumatrapdf/rel/SumatraPDF-0.9.1.zip HTTP/1.1" 206 - 43457 1003293 697 611 "http://kjkpub.s3.amazonaws.com/sumatrapdf/rel/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"'
    parsed = parse_s3_log_line(l)
    dump_parsed_s3_line(parsed)

if __name__ == "__main__":
    test()

This snippet only shows how to break one s3 log line into its components (parse_s3_log_line). Some work is needed to build upon this for parsing log files and extracting useful information out of them. For doing that, I recommend techniques described in Generator Tricks for Systems Programmers

Compacting s3 aws logs

2009-03-08T02:39:07Z

Amazon’s s3 can serve the files via HTTP protocol, so it can be used as an HTTP server for static files. With a little effort you can configure a given s3 bucket to save access log files to a different s3 bucket. This is useful if you’re interested e.g. in figuring out how often a given file is being downloaded. Logging doesn’t happen automatically. To set it up, carefully follow this official document

Unfortunately, standard s3 logs have a problem: they’re split among lots of files. Around 200 log files are generated for one day which leads to enormous number of files stored on s3 (e.g. merely 6 months of logs generated ~35k of log files).

To fix that I wrote a python script that compacts the logs so that we only get one file per day, which is a more reasonable number (35k files shrank to only 171 for me). As a bonus, it also compresses the log. Given very repetitive nature of log files, it’s a big win (the file shrinks by 94% when compressed with bzip2 algorithm).

The logic of the script is simple:

download all the files for a given day locally
create one file that is compressed concatenation of those files
upload compressed file back to s3
delete the original files
repeat

It took me a few hours to get right, so in the spirit of sharing, here it is. You’ll have to change a few things in the source:

s3 bucket name that stores the logs
where the logs are downloaded locally. I preserve them although they could be deleted if you’re short on space

Also, the script expects a awscreds.py file with access/secret key i.e.:

# This is awscreds.py file
access = "access key"
secret = "secret key"

One unexpected thing I noticed is that apparently Amazon sometimes saves the log file with screwy permissions so that it can’t be read or even fixed by setting new acl (both those operations return 403 forbidden). The only operation that works is deleting the file from s3, so that’s what I do (it’s only a log file, after all).

Another thing I noticed is that boto sometimes hangs when talking to s3 (exception on timeout would be much better). If that happens, you can kill the script and restart it – it’s smart enough to not re-download files and only deletes the original log files on s3 after compressed log has been saved in s3.

Expected usage of the script is as a daily script followed by a script that analyzes the logs. Due to boto hanging it might not work well in such scenario, though.

How content-based addressing can help web performance

2009-03-06T00:54:31Z

The problem with HTTP caching

Currently an HTTP resource (e.g. an image, a JavaScript file, a CSS script) is uniquely identified by a uri. Unfortunately a web browser doesn’t know if a given resource has changed, so in the simplest case, it has to re-download it over and over again. This is very inefficient, so we came up with ways to tell the browser to avoid re-downloading content.

One method is by setting Expires HTTP header by which a web server can tell the browser for how long can it cache a given resource. The problem with this approach is that if the resource is changed before the expiration date, the browser will see an outdated version.

Another method is ETag. When returning a resource web server also returns an ETag header which uniquely identifies the resource. If subsequent response contains the same etag, the browser might skip downloading the content. This still requires making a small request.

None of this methods works globally i.e. if two websites use the same version of jQuery library, even if they both instruct the browser to cache them, the browser still ends up downloading two versions. Google’s AJAX Libraries API is an attempt to solve this problem for the most popular web libraries.

What if there was a backwards-compatible way, efficient way of caching popular resources that worked globally (across multiple websites)? In theory it’s possible and not that hard.

Enter content-addressable resources

The idea of content-addressable data has been recently used to great benefit in e.g. git or BitTorrent protocol.

The idea is simple: we can uniquely identify any piece of data by calculating sha1 (or some other cryptographically secure hash) of its content.

All we need is a way to tell the browser the sha1 hash of a resource. If a resource with this hash is already cached locally, the browser doesn’t need to re-download it.

Compared to existing solutions, it has following advantages:

it works globally. If 10 websites host the same version of jQuery library, the browser only need to download it once
it doesn’t have the ‘oops, we set the expiration date too far into future’ problem that setting Expires header
it’s more efficient than ETag because it doesn’t require making any requests

How to tell the browser about the hash

The simplest way would be to define another attribute e.g. sha1hash that could be added tags that refer to resources, like script, a, link etc.

I’m not a web technologist and I don’t know this particular solution would be acceptable, but I’m sure there is a way.

HTTP info

2009-02-26T08:13:33Z

Response codes

200 - OK
206 - Partial Content (if successfully returned part of the file)

301 - Moved permanently 
302 - Found (temporary redirect)
303 - See other
304 - Not Modified

400 - Bad Request
401 - Not Authorized
403 - Forbidden
404 - Not Found
405 - Method Not Allowed
406 - Not Acceptable

Basic and Digest authentication

Covered by rfc 2617

Basic adds Authorization: header, e.g.: Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ== where the value after Basic is base64 encoding of string ${userid} ”:” ${password}

Digest is more complicated. Server responds with e.g.:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Digest
   realm="testrealm@host.com",
   qop="auth,auth-int",
   nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",
   opaque="5ccc069c403ebaf9f0171e9517f40e41"

And client has to reply with:

Authorization: Digest username="Mufasa",
   realm="testrealm@host.com",
   nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",
   uri="/dir/index.html",
   qop=auth,
   nc=00000001,
   cnonce="0a4f113b",
   response="6629fae49393a05397450978507c4ef1",
   opaque="5ccc069c403ebaf9f0171e9517f40e41"

Essential software

2009-02-26T08:12:43Z

Windows:

Source Insight – still the best editor for C/C++ code
cygwin with mintty (run it as C:\cygwin\bin\mintty.exe bash—login -i)
firefox and chrome
7-zip
WinMerge
Notepad2 and/or Notepad++
WireShark
Consolas font (if on Windows XP)
Python (ActivePython build)
Subversion (SilkSVN builds)
SysInternals live, regular
TaskSwitchXP (XP only, not working and largely not needed under Vista and 7)
digsby for IM client
NSIS
Virtual CloneDrive for mounting ISO images
Media Player Classic – HomeCinema (an update to Media Player Classic)
IcoFX – free icon editor
Greenshot – free screenshot grabber
WinDBG

Visual Studio add-ins:

RockScroll
SonicFileFinder – a better way to switch between files, assigned to Ctrl-O to match Source Insight

Firefox extensions:

Mac:

MacPorts (MacPorts essentials)

Misc Windows:

another list of useful software
nirsoft.net – CurrPorts, SmartSniff, OpenedFilesView, WhatInStartup etc.
FileZilla

File/Directory operations in C#

2009-02-26T08:12:08Z

directory operations

using System.IO;
bool Directory.Exists(string dirName);
Directory.CreateDirectory(string dirName);
string[] Directory.GetFiles(string dirName);
string[] Directory.GetFiles(string dirName, string fileNamePattern);

Programming in C interesting articles

2009-02-24T22:34:17Z

Volatile: C keyword myth dispelled – explains the exact meaning of volatile in C

Password-less authentication with ssh

2009-02-24T19:05:39Z

Short version:

generate public and private key (e.g. with ssh-keygen -q -f ~/.ssh/id_rsa -t rsa) (or -t dsa)
on the machine from which you connect: put private key in ~/.ssh/id_rsa (for rss key) or ~/.ssh/id_dsa (for dsa key), with 0600 permissions
on the machine you connect to: append public key to ~/.ssh/authorized_keys (this file should also have 0600 permission)

You can read long version

Where do bugs come from?

2009-02-24T17:59:29Z

Where do bugs come from?

Let’s assume you know, more or less, what you want to write and are ready to start cranking the code. You’ll write bugs. What’s the source of those bugs and how can you avoid them?

Ignorance and carelessness

There are only 2 sources of bugs in software:

ignorance
carelessness

Many faces of ignorance

Ignorance is not knowing what you’re doing. It’s a broad category, so let’s list some of the many ways to be ignorant.

Not knowing the language

For example, you don’t know that in C you need to free() each malloc(). Or that you can’t return a pointer to a local variable. Or you don’t understand the difference between sizeof(TCHAR) and sizeof(TCHAR*). Or any other of the infinite number of things to know. It takes years to master a programming language.

Not knowing the language very well

Even if you’re not writing outright bugs, there are still millions of ways to misuse a language. A trivial example in C might be malloc()ing a 10-byte temporary buffer in a function instead of using stack (which is faster and generates smaller code). Again, it takes years to really master a language.

Not knowing operating system and its APIs.

Windows has thousands of APIs. Mac OS has thousands of APIs. Way too much to keep details of every API call in your head. It’s easy to misuse the APIs and misuse leads to bugs.

Not knowing file format/protocol/specification.

Even if you know your language and APIs, there’s still plenty of sources of potential ignorance. Software doesn’t live in a vacuum but in a very rich and complicated ecosystem of existing solutions that are usually poorly designed or over-complicated due to historical reasons that no longer apply. Unfortunately our software has no choice but to talk to those systems.

If your software talks HTTP protocol, did you really carefully read the full HTTP 1.1 spec and made sure to handle all possible cases?

If you use XML, did you take time to read and fully comprehend XML spec?

If you’re writing a text editor, did you research character encodings, Unicode, different line endings on various platforms?

It’s easy to gain a superficial understanding of a complex system. Isn’t XML just stuff inside brackets? Bugs come from missed details. In XML‘s case it’s handling BOM, all possible encodings of XML files, entities, CDATA sections, comments, knowing which bytes cannot be put inside XML file (and more).

Inevitable mistakes

Even if we assume that we know everything, there’s the issue of mistakes. Even an experienced writer cannot write few pages of text without making a grammatical mistake. Programming is no different – no matter how good you are, you’ll make a number of mistakes per N lines of code.

The more careful you are, the less mistakes you will make, but no matter what the effort, the mistakes will be there

Avoiding bugs – the art of education and compensation

Now that we know where bugs come from, what can we do to decrease the number of bugs?

The methods (and apparatuses) we can use to decrease number of bugs in software fall into two categories:

education
compensation

We educate ourselves to minimize ignorance. We compensate for inherent fallibility of human mind to reduce number of mistakes we make or find them faster. Here’s a list of some specific things you can do.

Read

Read books, papers and web articles about your language, operating system, tools etc. That’s pretty obvious.

Study the APIs you use. Way too often we use APIs having only partial understanding of what they do, how can they fail, what consequences do they have etc. For Windows, MSDN documents API calls in great detail. Read the whole description, including remarks, not just the API signature. A similar resource to MSDN exists for every system out there.

Study the source code of other software. This is an open-source era. We have source code for thousands of systems, small and large, easily available for us to learn from. There’s no excuse not to do it.

You can read the source code in an exploratory way i.e. randomly browse the code fishing for nice tricks you can adopt in your own software.

You can do it in a focused way e.g. if you know you need to implement hash table, you can look at source of different implementations of hash tables to learn the techniques used.

Look for low-level implementation tricks and techniques as well as big picture (e.g. how the code is organized into modules).

Reading code is hard and boring. Even high-level languages operate on a very low level and it’s hard to see the forest because all those trees obstruct the view. It is, however, a skill that can be improved with practice.

Be reflective. When you find it hard to understand the code, try to figure out why is it hard (lack of comments? confusing comments? poor structure? poor naming of variables and functions?) and avoid doing things that make the code hard to read in your own code.

Conversely, when you see exceptionally well written code, try to figure out what makes it good and imitate.

Ask for code reviews

The amount of knowledge about programming is enormous. It takes years to master just one programming language or API set. Ignorance is a permanent state of being for a programmer and the worst part is that we don’t know what we don’t know.

Wouldn’t it be great if there was a magic fairy that would just tell us about the mistakes we made? No such fairy exists but the next best thing is your coworker. He might know that you should free() your malloc()s. He might know a better (simpler, faster) way to do something. He might spot a mistake. All you need to do is ask for a code review.

Talk about programming

Discuss programming topics with your peers. Ask them to teach you tricks they’ve learned. Tell them about the tricks you’ve learned. No matter how much you know, there’s always more to learn. For example, after umpteen years of writing code I’ve learned a cool linked list trick from interview with Anders Hejlsberg (it’s at the very end of the interview).

Use automatic tools for finding bugs.

Static checker analyzes the source code looking for mistakes. There are static code checkers for Java (FindBugs), C# (FxCop), Python (PyLint), C/C++ (Coverity, Visual Studio 2005 Team (aka. “expensive”) Edition has integrated static checker) and probably other languages.

There are run time tools like BoundsChecker, AppVerifier, Valgrind, AQTime and a countless number of memory debugging tools.

Learn about those tools and start using them regularly.

Use assert()

In C assert() macro checks if an expression is true. If it’s not, it’ll do something to alert the programmer (details depend on implementation, in gcc it calls abort(), in Visual Studio it shows a dialog box that gives the programmer to break into an app and inspect the state of the program).

Assert-like functionality exists in other languages as well.

The idea is to add checks for situations that should never, ever happen.

There’s a subtle distinction between something that almost never happens but is possible and something that should never, ever happen and I’ve seen asserts misused because people fail to understand that distinction (again, ignorance!).

The most common example of misusing asserts is validating that malloc() didn’t return NULL. While it’s unlikely to happen on a desktop machine, it’s a perfectly valid for malloc() to fail and return NULL.

An example of a valid assert is checking that after you insert a node in a list, the length of the list will increase by one if the insert was successful.

Assert impossible, not unlikely.

I’ve learned to assert() a lot while working at Microsoft since the code I worked on was littered with asserts (an example of learning good habits by reading other people’s code). Many times asserts noticed me about bugs that I would have missed and that’s why I think it’s worthwhile to use them.

Write unit tests.

Lots have been said on this topic since eXtreme programming became popular. I’m not going to explain why it’s a good thing – others have done it better that I could.

The reality of unit-testing is not as good as XP hype would have you believed. My random sampling of open-source projects and projects I’ve seen inside companies shows that almost no-one writes unit tests

Writing unit tests is difficult because it’s boring. When we understand the problem, our first instinct is to dive in and code the damn thing because it’s the shortest path to having things work. At the time of writing the code, it has obvious cost (time spent writing additional code for unit tests) but no obvious benefit.

Unit tests do payoff in longer term when they save you from long, hair-pulling debugging sessions. Tthe longer you expect the code to live, the bigger payoff from writing unit tests.

Not that everyone agrees with that

Write automated tests

Manual testing i.e. running your software and using it to see if it works as expected, is expensive. Automated testing is also expensive to setup and maintain, but it pays off in the long run.

Unit tests is only one example of possible automated tests. Unit tests are usually confined to thoroughly testing a small thing: a function or a class. There are other ways to automatically test the code that are beneficial. The golden rule of testing is that the more of it you have, the more bugs it’ll catch, the better the quality of your software.

For example, in Mozilla project, Java Script engine has a large number of relatively small Java Script code snippets that ensure that the implementation works the way it was designed. Also in Mozilla, Gecko rendering engine has has a number of layout test cases ensuring that changes to layout engine code do not change the way HTML pages are rendered on the screen. Sqlite database engine has a battery of tests to exercise the engine.

Automated tests are a rich topic. Think about ways you can automatically test your code (different strategies work for different kinds of code).

Do manual testing.

Oh yeah, just because you have automated tests doesn’t mean you can skip manual testing. Let me remind you a golden rule of testing: the more of it, the better.