Home
Software
Writings
Advanced command execution in Go with os/exec
part of Go Cookbook
go
This article describes how to use Go’s excellent os/exec standard package to execute programs.
Code for this article: https://github.com/kjk/the-code/tree/master/go/advanced-exec
In many examples we’ll be running ls -lah (tasklist on Windows).

Running a command

cmd := exec.Command("ls", "-lah")
	if runtime.GOOS == "windows" {
		cmd = exec.Command("tasklist")
	}
	err := cmd.Run()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}
See full example.
If you run it, nothing seems to happen. Fear not, the command has actually been executed.
If we were running ls -lah in the shell, the shell would copy programs’ stdout and stderr to console, so that we can see it.
We’re executing the program via Go standard library function and by default stdout and stderr are discarded.

Running a command and showing output

To let the human see the output, we can connect the output (cmd.Stdout and cmd.Stderr) of the program we’re executing to os.Stdout and os.Stderr, which is the output of our program:
cmd := exec.Command("ls", "-lah")
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	err := cmd.Run()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}
See full example.
cmd.Stdout and cmd.Stderr are declared as io.Writer interface so we can set them to any type that implements Write() method, like os.File or an in-memory buffer bytes.Buffer.
io.Reader and io.Writer are very simple, yet very powerful, abstractions.

Running a command and capturing the output

The above examples allows human to see the output but sometimes we want to capture the output and analyze it:
func main() {
    cmd := exec.Command("ls", "-lah")
    out, err := cmd.CombinedOutput()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    fmt.Printf("combined out:\n%s\n", string(out))
}
See full example.
CombinedOutput runs a command and returns combined stdout and stderr.

Behind the scenes of CombinedOutput

The good thing about Go is that it’s open source so we can peek at how a given functionality is implemented.
Another good thing is that most of the code in the standard library is simple. Here’s how CombinedOutput is implemented:
func (c *Cmd) CombinedOutput() ([]byte, error) {
	if c.Stdout != nil {
		return nil, errors.New("exec: Stdout already set")
	}
	if c.Stderr != nil {
		return nil, errors.New("exec: Stderr already set")
	}
	var b bytes.Buffer
	c.Stdout = &b
	c.Stderr = &b
	err := c.Run()
	return b.Bytes(), err
}
Notice that it’s almost as simple as our second example. Instead of setting cmd.Stdout and cmd.Stderr to standard output, we set them to a single in-memory buffer.
When program finishes, we returned everything written to that buffer.
Don’t be afraid to peruse the code of standard library.

Capture stdout and stderr separately

What if you want to do the same but capture stdout and stderr separately?
func main() {
    cmd := exec.Command("ls", "-lah")
    var stdout, stderr bytes.Buffer
    cmd.Stdout = &stdout
    cmd.Stderr = &stderr
    err := cmd.Run()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    outStr, errStr := string(stdout.Bytes()), string(stderr.Bytes())
    fmt.Printf("out:\n%s\nerr:\n%s\n", outStr, errStr)
}
See full example.

Capture output but also show progress on stdout

If we set cmd.Stdout or cmd.Stderr, we can’t capture the output via cmd.CombinedOutput().
We can use io.MultiWriter to create a proxy io.Writer which will write to one or more other io.Writers. In our case we’ll write to os.Stdout so that it’ll show on the console and to bytes.Buffer to also capture the output.
func main() {
	cmd := exec.Command("ls", "-lah")
	if runtime.GOOS == "windows" {
		cmd = exec.Command("tasklist")
	}

	var stdoutBuf, stderrBuf bytes.Buffer
	cmd.Stdout = io.MultiWriter(os.Stdout, &stdoutBuf)
	cmd.Stderr = io.MultiWriter(os.Stderr, &stderrBuf)

	err := cmd.Run()
	if err != nil {
		log.Fatalf("cmd.Run() failed with %s\n", err)
	}
	outStr, errStr := string(stdoutBuf.Bytes()), string(stderrBuf.Bytes())
	fmt.Printf("\nout:\n%s\nerr:\n%s\n", outStr, errStr)
}
See full example.

Write to program’s stdin

We know how to read program’s stdout but we can also write to its stdin.
There is no Go library to do bzip2 compression (only decompression is available in standard library).
We can use bzip2 to do the compression by:
It would be even better if we didn’t have to create a temporary file.
Most compression programs accept data to compress/decompress on stdin.
To do that on command-line we would use the following command: bzip2 -c <${file_in} >${file_out}.
Here’s the same thing in Go:
// compress data using bzip2 without creating temporary files
func bzipCompress(d []byte) ([]byte, error) {
    var out bytes.Buffer
    // -c : compress
    // -9 : select the highest level of compresion
    cmd := exec.Command("bzip2", "-c", "-9")
    cmd.Stdin = bytes.NewBuffer(d)
    cmd.Stdout = &out
    err := cmd.Run()
    if err != nil {
        return nil, err
    }
    return out.Bytes(), nil
}
See full example.
We can also call cmd.StdinPipe(), which returns io.WriteCloser. It’s more complicated but gives more control over writing.

Changing environment of executed program

Things to know about environment variables in Go:
Sometimes you need to modify the environment of the executed program.
Go supports that by setting Env member of exec.Cmd. cmd.Env has the same format as os.Environ().
If Env is not set, the process inherits environment of the calling process.
Usually, you don’t want to construct a completely new environment from scratch but pass a modified version of an environment of the current process. Here’s how to add a new variable:
cmd := exec.Command("programToExecute")

    additionalEnv := "FOO=bar"
    newEnv := append(os.Environ(), additionalEnv))
    cmd.Env = newEnv

    out, err := cmd.CombinedOutput()
    if err != nil {
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    fmt.Printf("%s", out)
Things get more complicated if you want to delete an environment or to ensure you’re not setting the same variable twice. Package shurcooL/go/osutil offers an easier way of manipulating environment variables.

Check early that a program is installed

Imagine you wrote a program that takes a long time to run. At the end, you call executable foo to perform an essential task.
If foo executable is not present, the call will fail.
It’s a good idea to detect lack of executable foo at the beginning and fail early with descriptive error message.
You can do it using exec.LookPath.
func checkLsExists() {
    path, err := exec.LookPath("ls")
    if err != nil {
        fmt.Printf("didn't find 'ls' executable\n")
    } else {
        fmt.Printf("'ls' executable is in '%s'\n", path)
    }
}
See full example.
Another way to check if a program exists is to try to execute it in a no-op mode (e.g. many programs support --help option).
Code for this article: https://github.com/kjk/the-code/tree/master/go/advanced-exec
Part of Go Cookbook
Topics: go
Found a mistake, have a comment? Let me know.

Feedback about page:

Feedback:
Optional: your email if you want me to get back to you: