This article describes how to use Go’s excellent os/exec standard package to execute programs.
In many examples we’ll be running ls -lah (tasklist on Windows).
Running a command
cmd := exec.Command("ls", "-lah")
if runtime.GOOS == "windows" {
cmd = exec.Command("tasklist")
}
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
If you run it, nothing seems to happen. Fear not, the command has actually been executed.
If we were running ls -lah in the shell, the shell would copy programs’ stdout and stderr to console, so that we can see it.
We’re executing the program via Go standard library function and by default stdout and stderr are discarded.
Running a command and showing output
To let the human see the output, we can connect the output (cmd.Stdout and cmd.Stderr) of the program we’re executing to os.Stdout and os.Stderr, which is the output of our program:
cmd := exec.Command("ls", "-lah")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
cmd.Stdout and cmd.Stderr are declared as io.Writer interface so we can set them to any type that implements Write() method, like os.File or an in-memory buffer bytes.Buffer.
io.Reader and io.Writer are very simple, yet very powerful, abstractions.
Running a command and capturing the output
The above examples allows human to see the output but sometimes we want to capture the output and analyze it:
func main() {
cmd := exec.Command("ls", "-lah")
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("combined out:\n%s\n", string(out))
}
CombinedOutput runs a command and returns combined stdout and stderr.
Behind the scenes of CombinedOutput
The good thing about Go is that it’s open source so we can peek at how a given functionality is implemented.
Another good thing is that most of the code in the standard library is simple. Here’s how CombinedOutput is implemented:
func (c *Cmd) CombinedOutput() ([]byte, error) {
if c.Stdout != nil {
return nil, errors.New("exec: Stdout already set")
}
if c.Stderr != nil {
return nil, errors.New("exec: Stderr already set")
}
var b bytes.Buffer
c.Stdout = &b
c.Stderr = &b
err := c.Run()
return b.Bytes(), err
}
Notice that it’s almost as simple as our second example. Instead of setting cmd.Stdout and cmd.Stderr to standard output, we set them to a single in-memory buffer.
When program finishes, we returned everything written to that buffer.
Don’t be afraid to peruse the code of standard library.
Capture stdout and stderr separately
What if you want to do the same but capture stdout and stderr separately?
func main() {
cmd := exec.Command("ls", "-lah")
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
outStr, errStr := string(stdout.Bytes()), string(stderr.Bytes())
fmt.Printf("out:\n%s\nerr:\n%s\n", outStr, errStr)
}
Capture output but also show progress on stdout
If we set cmd.Stdout or cmd.Stderr, we can’t capture the output via cmd.CombinedOutput().
We can use
io.MultiWriter to create a proxy
io.Writer which will write to one or more other
io.Writers. In our case we’ll write to
os.Stdout so that it’ll show on the console and to
bytes.Buffer to also capture the output.
func main() {
cmd := exec.Command("ls", "-lah")
if runtime.GOOS == "windows" {
cmd = exec.Command("tasklist")
}
var stdoutBuf, stderrBuf bytes.Buffer
cmd.Stdout = io.MultiWriter(os.Stdout, &stdoutBuf)
cmd.Stderr = io.MultiWriter(os.Stderr, &stderrBuf)
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
outStr, errStr := string(stdoutBuf.Bytes()), string(stderrBuf.Bytes())
fmt.Printf("\nout:\n%s\nerr:\n%s\n", outStr, errStr)
}
Write to program’s stdin
We know how to read program’s stdout but we can also write to its stdin.
There is no Go library to do bzip2 compression (only decompression is available in standard library).
We can use bzip2 to do the compression by:
- writing the data to a temporary file
- call
bzip2 -c ${file_in} and capture its stdout
It would be even better if we didn’t have to create a temporary file.
Most compression programs accept data to compress/decompress on stdin.
To do that on command-line we would use the following command: bzip2 -c <${file_in} >${file_out}.
Here’s the same thing in Go:
// compress data using bzip2 without creating temporary files
func bzipCompress(d []byte) ([]byte, error) {
var out bytes.Buffer
// -c : compress
// -9 : select the highest level of compresion
cmd := exec.Command("bzip2", "-c", "-9")
cmd.Stdin = bytes.NewBuffer(d)
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
return nil, err
}
return out.Bytes(), nil
}
We can also call cmd.StdinPipe(), which returns io.WriteCloser. It’s more complicated but gives more control over writing.
Changing environment of executed program
Things to know about environment variables in Go:
os.Environ() returns []string where each string is in form of FOO=bar, where FOO is the name of environment variable and bar is the valueos.Getenv("FOO") returns the value of environment variable FOO.
Sometimes you need to modify the environment of the executed program.
Go supports that by setting Env member of exec.Cmd. cmd.Env has the same format as os.Environ().
If Env is not set, the process inherits environment of the calling process.
Usually, you don’t want to construct a completely new environment from scratch but pass a modified version of an environment of the current process. Here’s how to add a new variable:
cmd := exec.Command("programToExecute")
additionalEnv := "FOO=bar"
newEnv := append(os.Environ(), additionalEnv))
cmd.Env = newEnv
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("%s", out)
Things get more complicated if you want to delete an environment or to ensure you’re not setting the same variable twice. Package
shurcooL/go/osutil offers an easier way of manipulating environment variables.
Check early that a program is installed
Imagine you wrote a program that takes a long time to run. At the end, you call executable foo to perform an essential task.
If foo executable is not present, the call will fail.
It’s a good idea to detect lack of executable foo at the beginning and fail early with descriptive error message.
You can do it using exec.LookPath.
func checkLsExists() {
path, err := exec.LookPath("ls")
if err != nil {
fmt.Printf("didn't find 'ls' executable\n")
} else {
fmt.Printf("'ls' executable is in '%s'\n", path)
}
}
Another way to check if a program exists is to try to execute it in a no-op mode (e.g. many programs support --help option).
