This article describes how to use Go’s excellent os/exec standard package to execute programs.
The code blocks are separate examples; import the packages used by each one. exec.Command passes arguments directly and does not invoke a shell, expand wildcards, or interpret pipes and redirects. Pass each argument separately.
In many examples we’ll be running ls -lah (tasklist on Windows).
Running a command
cmd := exec.Command("ls", "-lah")
if runtime.GOOS == "windows" {
cmd = exec.Command("tasklist")
}
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
If you run it, nothing seems to happen. Fear not, the command has actually been executed.
If we were running ls -lah in the shell, the shell would copy programs’ stdout and stderr to console, so that we can see it.
We’re executing the program via Go standard library function and by default stdout and stderr are discarded.
Running a command and showing output
To let the human see the output, we can connect the output (cmd.Stdout and cmd.Stderr) of the program we’re executing to os.Stdout and os.Stderr, which is the output of our program:
cmd := exec.Command("ls", "-lah")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
cmd.Stdout and cmd.Stderr are declared as io.Writer interface so we can set them to any type that implements Write() method, like *os.File or an in-memory buffer *bytes.Buffer.
io.Reader and io.Writer are very simple, yet very powerful, abstractions.
Running a command and capturing the output
The examples above let a human see the output but sometimes we want to capture the output and analyze it:
func main() {
cmd := exec.Command("ls", "-lah")
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("combined out:\n%s\n", string(out))
}
CombinedOutput runs a command and returns combined stdout and stderr.
Captured output is also available when the command fails, so include it in diagnostics when appropriate. CombinedOutput buffers the entire result in memory; stream large outputs to a writer instead.
Behind the scenes of CombinedOutput
The good thing about Go is that it’s open source so we can peek at how a given functionality is implemented.
Another good thing is that most of the code in the standard library is simple. Here’s how CombinedOutput is implemented:
func (c *Cmd) CombinedOutput() ([]byte, error) {
if c.Stdout != nil {
return nil, errors.New("exec: Stdout already set")
}
if c.Stderr != nil {
return nil, errors.New("exec: Stderr already set")
}
var b bytes.Buffer
c.Stdout = &b
c.Stderr = &b
err := c.Run()
return b.Bytes(), err
}
Notice that it’s almost as simple as our second example. Instead of setting cmd.Stdout and cmd.Stderr to standard output, we set them to a single in-memory buffer.
When the program finishes, we return everything written to that buffer.
Don’t be afraid to peruse the code of standard library.
Capture stdout and stderr separately
What if you want to do the same but capture stdout and stderr separately?
func main() {
cmd := exec.Command("ls", "-lah")
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
outStr, errStr := string(stdout.Bytes()), string(stderr.Bytes())
fmt.Printf("out:\n%s\nerr:\n%s\n", outStr, errStr)
}
Capture output but also show progress on stdout
If we set cmd.Stdout or cmd.Stderr, we can’t capture the output via cmd.CombinedOutput().
We can use
io.MultiWriter to create a proxy
io.Writer which will write to one or more other
io.Writers. In our case we’ll write to
os.Stdout so that it’ll show on the console and to
bytes.Buffer to also capture the output.
func main() {
cmd := exec.Command("ls", "-lah")
if runtime.GOOS == "windows" {
cmd = exec.Command("tasklist")
}
var stdoutBuf, stderrBuf bytes.Buffer
cmd.Stdout = io.MultiWriter(os.Stdout, &stdoutBuf)
cmd.Stderr = io.MultiWriter(os.Stderr, &stderrBuf)
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
outStr, errStr := string(stdoutBuf.Bytes()), string(stderrBuf.Bytes())
fmt.Printf("\nout:\n%s\nerr:\n%s\n", outStr, errStr)
}
Write to program’s stdin
We know how to read program’s stdout but we can also write to its stdin.
The standard library’s compress/bzip2 package supports decompression only. To demonstrate stdin handling, we’ll use the external bzip2 program for compression.
We can use bzip2 to do the compression by:
- writing the data to a temporary file
- calling
bzip2 -c ${file_in} and capturing its stdout
It would be even better if we didn’t have to create a temporary file.
Most compression programs accept data to compress/decompress on stdin.
To do that on command-line we would use the following command: bzip2 -c <${file_in} >${file_out}.
Here’s the same thing in Go:
// compress data using bzip2 without creating temporary files
func bzipCompress(d []byte) ([]byte, error) {
var out bytes.Buffer
// -c : write the result to stdout
// -9 : select the highest level of compression
cmd := exec.Command("bzip2", "-c", "-9")
cmd.Stdin = bytes.NewReader(d)
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
return nil, err
}
return out.Bytes(), nil
}
We can also call cmd.StdinPipe(), which returns io.WriteCloser. It’s more complicated but gives more control over writing.
Changing environment of executed program
Things to know about environment variables in Go:
os.Environ() returns []string where each string is in form of FOO=bar, where FOO is the name of environment variable and bar is the valueos.Getenv("FOO") returns the value of environment variable FOO.
Sometimes you need to modify the environment of the executed program.
Go supports that by setting Env member of exec.Cmd. cmd.Env has the same format as os.Environ().
If Env is not set, the process inherits environment of the calling process.
Usually, you don’t want to construct a completely new environment from scratch but pass a modified version of an environment of the current process. Here’s how to add a new variable:
cmd := exec.Command("programToExecute")
additionalEnv := "FOO=bar"
newEnv := append(cmd.Environ(), additionalEnv)
cmd.Env = newEnv
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("%s", out)
Use cmd.Environ() (Go 1.19 or later) to obtain the command’s environment, then append overrides. When a variable appears more than once, the last value wins. Set cmd.Dir first so the returned environment reflects the working directory on platforms that use PWD. To remove a variable, filter it from the slice; setting an empty value is different from removing it.
Check early that a program is installed
Imagine you wrote a program that takes a long time to run. At the end, you call executable foo to perform an essential task.
If foo executable is not present, the call will fail.
It’s a good idea to detect lack of executable foo at the beginning and fail early with descriptive error message.
You can do it using exec.LookPath.
func checkLsExists() {
path, err := exec.LookPath("ls")
if err != nil {
fmt.Printf("didn't find 'ls' executable\n")
} else {
fmt.Printf("'ls' executable is in '%s'\n", path)
}
}
Another way to check if a program exists is to try to execute it in a no-op mode (e.g. many programs support --help option).
Timeouts, cancellation, and exit status
Use a context to limit how long a command runs. WaitDelay, added in Go 1.20, also bounds waiting for a process to exit after cancellation and for I/O pipes left open by descendants:
func runGitStatus() error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "git", "status", "--short")
cmd.WaitDelay = 2 * time.Second
output, err := cmd.CombinedOutput()
fmt.Printf("%s", output)
if ctx.Err() != nil {
return ctx.Err()
}
if err != nil {
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
return fmt.Errorf("git exited with status %d: %w", exitErr.ExitCode(), err)
}
return fmt.Errorf("run git: %w", err)
}
return nil
}
This helper needs context, errors, fmt, os/exec, and time. The default cancellation kills the direct child process; it is not a portable process-tree shutdown mechanism. Check ctx.Err() because cancellation can also surface as an exit error.
When using StdoutPipe or StderrPipe, start the command, read the pipes, and then call Wait. Read both pipes concurrently if both can fill. Do not call Run and expect to read a pipe afterward. Always call Wait after a successful Start to release process resources.
Since Go 1.19, executable lookup rejects an implicit match in the current directory with
exec.ErrDot. If you intend to run a local executable, specify its path, such as
./tool or an absolute path, rather than disabling this protection. See the
os/exec documentation for the lookup and cancellation contracts.
