The Go standard library provides convenient methods to run external commands easily. Generally we use the methods under the os/exec package to execute external commands and interact with external commands. os/exec
wraps the os.StartProcess method for easier access to input and output, providing I/O pipe and other features. I will dedicate two articles to Go’s methods for starting new processes and executing external commands, this is the first one, dedicated to the os/exec
library.
The os/exec
library provides methods similar to those defined in the POSIX
standard for the C language, but provides further encapsulation to make it easier to use. I will introduce you to each of them next.
Run a command
The simplest way to run an external command is to call the Command method, you need to pass in the program to be executed and parameters, it will return a *Cmd
data structure, representing an external command to be executed, the main call to its Run
, Output
, CombinedOutput
method after this object can not be reused, and generally we will not reuse this object, but generate a new one when needed.
The following code is to execute the ls
command and pass it the parameter -lah
.
If you execute this command, there is no output in the console, in fact the command has been executed, only our Go program does not capture and process the output, so there is no output in the console.
Run method will execute the external command and wait for the command to finish, if the command is executed normally without errors and the return code is 0, then Run returns err == nil, then it returns an *ExitError
, sometimes you need to read cmd.
If you don’t want to wait, then you can call the Start method. If Start succeeds, the Process and ProcessState
fields will be set. You can check ProcessState.Exited()
to determine if the program has exited. If you want to block again and wait for the program to finish, you can call the Wait method.
In fact, the Run
method is implemented using the Start
and Wait
methods
Display the output of external commands
The Cmd command contains input and output fields that you can set to enable customization of input and output:
If Stdin
is null, then the process will read from null device
(os.DevNull).
If Stdin
is an *os.File
object, then it will read from this file.
If Stdin
is os.Stdin
, then it will read from standard input like command line.
Stdout
and Stderr
represent the standard output and error output of the external program process, and if they are null, then they are output to the null device.
If Stdout
and Stderr
are *os.File objects, then the data is output to a file.
If Stdout
and Stderr are set to os.Stdout and os.Stderr respectively, it will output to the command line.
Let’s adapt the previous example to show the command output.
Work Dir
By default the working path of the process is the folder where the process is called, but you can also specify it manually, for example we specify the working path as the root path.
External program path
Cmd.Path
is the path of the program to be executed, if it is a relative path, then it calculates the relative path based on Cmd.Dir
. If the program is already in the system $PATH path, then you can write the program name directly.
Set environment variables
Cmd also has a field called Env
, which is used to set the environment variable of the process in the format key=value
.
If Env
is empty, then the new process will use the environment variables of the caller process.
For example, in the following example, we set the myvar
variable, you can comment out the line cmd.Enc = ...
that line to compare the results.
The underlying Process and ProcessState
As I mentioned above, os/exec
is a wrapped convenience library, and at the bottom it is implemented using os.StartProcess
, so you can get the underlying Process object and ProcessState object, which represent the process and the process state respectively
|
|
Determine if an external command exists
Sometimes you need to check if an external command exists when you execute it, you can use the LookPath
method.
If the incoming argument contains a path separator, then it will look for the program based on the relative or absolute path of Cmd.Dir
. If it does not contain a path separator, then it will look for the file from the PATH
environment variable
Get command results
Cmd provides the Output()
method to get the bytes of the command execution result if the command is executed correctly:
If there is an error with the command, the error message can be obtained with Stderr
:
Combining Stdout and Stderr
If you want to have one way to get the output regardless of errors, you can call the CombinedOutput()
method, which will return either normal or error output, and a second return err to indicate whether an error was executed
The implementation of the CombinedOutput
method is also very simple, in fact it is implemented by sharing the same bytes.Buffer
|
|
Read Stdout and Stderr separately
Once we understand the implementation of CombinedOutput
, we can set bytes.Buffer
for Stdout
and Stderr
respectively, to achieve independent reads.
Show command execution progress
Since we have been able to use our own io.Writer
to set Stdout/Stderr
, we can do something richer.
For example, if we use the curl
command to download a large file, we can display the size of the downloaded data in real time:
|
|
Set Stdin
While the previous examples demonstrated handling Output
, this next example shows how to set up Stdin
.
The wc
command reads the main.go
file and counts how many lines it has in total.
Pipe
You can use the output of one command as input to the next command, and so on, to string multiple commands into a pipe.
os/exec
provides StderrPipe
, StdinPipe
, and StdoutPipe
methods to get the pipe object.
For example, the following command takes the output of cat main.go
as input to the wc -l
command:
|
|
First, the pipeline is created, and then the Start method and Wait method of each command are called in turn.
Generic Pipe method
Here is a more generic way to create a Cmd pipeline.
|
|
bash pipe
If you execute it via the bash command, you can use the bash pipe feature, which is simpler to write.
Orphan Process
When the parent process ends before the child process does, then the child process is called an orphan process and the ppid of the child process is set to 1 at this time.
When the main program exits, this curl sub-process will keep downloading and its ppid is set to 1. You can use ps to view the process information, for example in a Mac.
|
|
Kill child processes when the program exits
If we want to kill the child process started by the program when it exits, then a stupid way is to get the Process object of the child process and call its Kill method to kill it. But unfortunately it can’t kill the Sun process.
For Linux systems, you can kill the grandchild process as well with the following settings.
Or more generally, use the following settings:
see https://groups.google.com/g/golang-nuts/c/XoQ3RhFBJl8
Pass the file opened by the parent process to the child process
In addition to the standard input and output of 0,1,2 files, you can also pass files from the parent process to the child process via the Cmd.ExtraFiles
field.
One of the more common scenarios is graceful restart, where the new process inherits the net.Listener
that the old process is listening to, so that the network connection does not need to be closed and reopened.
For example, one of the earliest articles introducing the go graceful restart technique, Graceful Restart in Golang, has the following code
|
|
Reference https://colobu.com/2020/12/27/go-with-os-exec/