SSH Command Action

Starting with Flux 8.0.10, one can perform remote process management using Flux SSH command actions. Since SSH does not provide a mechanism to manage processes remotely, one can now simply implement a Flux workflow to handle this. See SSH Actions for initial setup.

To perform remote process management, one can do the following:

• start a remote process in background mode and then,
• store the process id returned in the stdout of the RESULT from the Flux SSH Command Action in a workflow variable “pid”.

Here is an example remote command that the business would like to run, and ensure it completes within a specified duration.

Remote Command=nohup staging/generateReports.sh > out 2> err < /dev/null & echo $!

This command would return the process id of the running program which is then stored in the workflow variable “pid”. Assume it is expected that this program is to complete in 2 hours. A workflow is desired to check if the process is still running after that time. This workflow queries the process id of the generatereports script using another remote command, which simply returns the name of the process if it is still running or none if the script has finished executing:

Remote Command=ps -p ${pid} -o comm=

In normal circumstances, one would see the process finish executing and then initiating downstream processing further in the workflow, but if it is still running the workflow will shutdown the process and notify administrative personnel. Issuing a kill command as shown below within an SSH Command Action will shutdown the running process.

Remote Command=kill -9 ${pid}

An example Flux workflow illustrating this workflow is shown below.