Runtime Configuration

There are two configuration components for the Flux engine:

  1. Static. Static configuration options are described in greater detail on the Engine Configuration page.
  2. Runtime. Runtime configuration options typically determine how workflows run (for example, limiting the number of workflows that can run at once) or contain data that the workflows will use while running. Unlike the static configuration, the runtime configuration is dynamically reloaded, so runtime configuration settings can change while the engine is running. Runtime configuration options are applied to nodes, or branches, on the workflow tree, allowing you to define runtime configuration properties on a per-namespace level.

Every engine is controlled by both its static (non-runtime) configuration and its runtime configuration.

The runtime configuration is optional. To enable the runtime configuration, you must first specify the configuration file location, then set the properties you would like to use.

Setting the Runtime Configuration File

To use a runtime configuration file, you can simply add the following property to your engine configuration:

RUNTIME_CONFIGURATION_FILE=/path/to/myfile.config

Refreshing the Runtime Configuration File

If you make changes to the runtime configuration file, Flux can automatically and dynamically pick up your changes while the engine is running (without requiring an engine restart). This refresh is performed at a specific interval, which you can adjust by setting the RUNTIME_CONFIGURATION_FILE_REFRESH_FREQUENCY like so:

RUNTIME_CONFIGURATION_FILE_REFRESH_FREQUENCY=+30s

This property is a time expression. The default value is +m (every one minute).

Setting Properties in the Runtime Configuration

The Flux runtime configuration is a standard Java properties file. This means that each line of the file is expected to contain a single key/value pair.

In the runtime configuration file, the key is expected to be a namespace corresponding with your workflow namespace (see Tree of Configuration Properties below for more information) plus the name of the property you want to set. The value of each pair is the intended value of the property you are setting.

There are two types of properties available:

  • Runtime Configuration Property. These properties configure how workflows are executed on the engine. Runtime configuration properties may include concurrency throttle settings, priority values, FIFO settings, and more. See the List of Runtime Configuration Properties for a complete list of available properties of this type.
  • Runtime Variables. These are custom variables that you can use to provide data to your workflows. Typically, this would be used when a single value (for example, a username or password) is used in several workflows. Using runtime variables allows you to specify variable values in a single location, rather than re-entering them across several workflows. You can use variable substitution or runtime data mapping to access the runtime variable values inside a workflow.

Runtime configuration properties are set like so:

/namespace/<PROPERTY TYPE>=<value>

Runtime variables, similarly, are set like:

/namespace/<VARIABLE NAME>=<value>

Runtime configuration and variable properties can be set on any namespace, including the root (“/”). These settings are applied to all workflows on the namespace, as well as all workflows on child namespaces.

A sample runtime configuration might look like:

/MAIL_SERVER_VARIABLE=mymailserver.com
/CONCURRENCY_THROTTLE=8
/namespace/CONCURRENCY_THROTTLE=5

Runtime configuration namespaces

Runtime configuration properties can only be applied to a namespace, not directly to a particular workflow or workflow name. This means, for example, that a workflow with the full namespace (including workflow name) of:

/MyNamespace/MySubNamespace/MyWorkflowName

Will pick up any properties set in the runtime configuration for namespaces:

/MyNamespace

and

/MyNamespace/MySubNamespace

But a property set on the full name of:

/MyNamespace/MySubNamespace/MyWorkflowName

will be ignored. Keep this in mind when naming your workflows and designing the runtime configuration.

List of Runtime Configuration Properties

The following properties can be set in the runtime configuration. All of the properties listed can be specified across the entire engine (by setting them on the root namespace), or assigned to an individual namespace.

CONCURRENCY_THROTTLE

The total number of workflows that can run concurrently within this namespace. For example, a concurrency throttle of 5 would mean that 5 workflows in this namespace should be allowed to run at once. Note that there is one exception to this: if a workflow contains a split (or, in general, any concurrently executing actions), each action will count toward the total concurrency. For example, if a workflow contains 5 actions running at once, and the namespace has a concurrency throttle of 5, that workflow will use all of the available concurrency slots and no other workflows will be allowed to run in the same namespace until the actions complete.

The concurrency throttle setting allows you to conserve limited resources (like processing power and I/O bandwidth) by preventing an unbounded number of workflows from running at once.

You can also set a concurrency throttle of 0 to prevent a certain namespace from running at all. For example, if you have workflows that require specific resources available on the machine, you could use concurrency throttles to limit which machines those workflows are allowed to run on.

CONCURRENCY_THROTTLE_CLUSTER_WIDE

Identical to the CONCURRENCY_THROTTLE setting, except that this property specified the total number of workflows in this namespace that can be allowed to run across the entire cluster. For example, if engines A and B are running in a cluster, the CONCURRENCY_THROTTLE_CLUSTER_WIDE for each engine is set to 8, engine A is running 5 workflows, and engine B is running 3 workflows, then neither engine will accept any new workflows until the concurrency slots are available again.

This property overrides the individual concurrency throttle for each engine – an engine will not accept a workflow if the cluster-wide throttle for that workflow’s namespace is full, even if there is room in the engine’s CONCURRENCY_THROTTLE settings for that namespace.

DEFAULT_FLOW_CHART_ERROR_HANDLER

The error handler that workflows in this namespace should use. This property should be set as a path to the error handler file. The path can be absolute, like “/path/to/myfile.ffc”, or it can be relative to the Java home directory (that is, the directory where the JVM was started, usually the directory containing the script to launch the Flux engine). A relative path may look like “folder/myfile.ffc”.

FIFO_SCHEDULING_ENABLED

Normally, workflows that are eligible for execution are scheduled for execution based on their priority and their “next execution date”. Sometimes, however, it is desirable to schedule workflows to run in a first-in-first-out (FIFO) order. For example, suppose 10,000 trucking manifests need to be generated, and the manifest workflows that were scheduled first should be, generally speaking, processed first. Flux normally processes such workflows in this manner by default. However, the existence of a trigger or an error handler in a workflow can cause a workflow to “go to the back of the queue”. In these cases, it may be that the trucking manifest workflow that was scheduled first may end up running behind the ones that were originally submitted afterward.

To ensure that workflows are executed in the order in which they were originally submitted to an engine, you can use Flux’s FIFO Scheduling feature. When enabled, this feature ensures that workflows are scheduled for execution based on the timestamp at which they were originally submitted to an engine (except for workflow priorities, which take precedence over FIFO Scheduling). If a workflow is paused, it is not considered in the FIFO Scheduling algorithm.

Continuing the trucking manifest example, if an engine’s concurrency throttle is 10 and there are 100 trucking manifest workflows submitted to an engine, up to 10 manifest workflows may execute simultaneously. FIFO Scheduling ensures that those 10 running workflows were submitted to the engine earlier than the remaining 90 workflows.

To achieve a true queue-like behavior, simply set the concurrency throttle to one. That configuration ensures that one workflow must either run to completion or reach a trigger before the net flow can execute.

FIFO scheduling is disabled by default.

INTERNAL_LOGGER_DEBUG_ENABLED

If this property is set to true, action and trigger properties and variables for workflows in this namespace will be logged in greater detail to the log file and audit trail. As the name implies, this property can only be used if the internal logger is enabled.

LISTENER_CLASSPATH

Specifies the listener classpath for workflows in this namespace. This property specifies the listener classpath for workflows, which is used to load listener classes in Java Actions and Dynamic Java Actions.

PRIORITY

Specifies the priority for workflows in this namespace. Priorities are used to give precedence to certain workflows when there are more workflows available to run than concurrency slots open to run them.

RUN_AS_USER

The username of a user whose permissions will be used to run this workflow. This can only be used when using the built-in Flux security mechanism.

Tree of Configuration Properties

The runtime configuration is set up like a tree of nodes (following the normal tree structure). Runtime configuration settings are applied on a per-namespace level, so the runtime configuration tree will mimic the workflow namespace tree. A property set on a branch in the runtime configuration tree is applied to the branch of the same name in the workflow namespace tree.

Runtime configuration namespaces are not case-sensitive. For example, the namespace “/MyNamespace” is equivalent to setting “/mynamespace” in the runtime configuration.

Each configuration property consists of a key and a value. The key must be a string. The value must be a persistent variable.

Although the runtime configuration tree can contain any kind of property, it includes pre-defined properties: concurrency throttle, concurrency throttle cluster-wide, default workflow error handler, and priority.

Workflow names are hierarchical. They form a tree of workflows. For example, the two workflows, “/lightweights/workflow 1” and “/heavyweights/workflow 2”, form a tree with two branches off the root, “/lightweights” and “/heavyweights”.

You can annotate this tree of workflows using properties in the runtime configuration. For example, to set a concurrency throttle for heavyweight workflows, set an appropriate concurrency throttle in the “/heavyweights” branch of the runtime configuration tree. To set a default workflow error handler for lightweight workflows, set an appropriate error handler in the “/lightweights” branch of the runtime configuration tree.

Properties set on a child namespace will override any properties set on a parent. For example, if a workflow runs in the namespace “/parent/child/”, it will first check the namespace “/parent/child/” for any runtime properties. Properties for the “/parent” namespace would only be applied if they were not set in “/parent/child”.

In summary, the tree of configuration properties mirrors the tree of workflows. A workflow’s behavior can be further configured based on the branches in the runtime configuration tree that correspond to that workflow’s fully qualified name.

Runtime Configuration Properties for Namespaces that contain a space

If your namespace contains a space character, like:

/Folder A/MyWorkflow

You will need to make sure the space is escaped in your runtime configuration by adding a backslash (‘') character before the space in the configuration. For example, to set a runtime configuration property for a workflow with the name and namespace above, you could add the following line to your runtime configuration file:

/Folder\ A/CONCURRENCY_THROTTLE=0

Memory Usage

It is assumed that your entire runtime configuration tree can be stored in memory. A very large runtime configuration can cause memory problems in Flux, so take care to ensure that you are not setting any unnecessary or extraneous data in the tree.

Storing Encrypted Passwords in the Runtime Configuration

If you have a password value that is used in several workflows across the engine, it often makes sense to store the password in one central location where all of the workflows can easily access it.

In this case, you would normally not want to store the password in plain text in the runtime configuration either (where any user with access to the runtime configuration file or the logs might access it). To help with this, Flux allows you to store an encrypted value in the runtime configuration, protecting your password values and ensuring that they are not available in plain text anywhere on the system.

To store an encrypted password in the runtime configuration and access it within your workflows, follow these steps:

  1. Follow the Command Line Interface for generating an encrypted password. This will create an encrypted password that looks something like Gc4f0oRkPJ4=.
  2. Edit your runtime configuration to store the encrypted password. The encrypted password doesn’t require any special syntax and can be set like any other variable:
    /MyEncryptedPassword=Gc4f0oRkPJ4=
  3. Use runtime substitution to access the value in the password field on your action:
    ${runtime MyEncryptedPassword}

That’s it! This technique can be used for any action in Flux that requires a password, including file actions and triggers that use a password for authentication to a remote host.

The Runtime Configuration File and Java API

When using the runtime configuration file, it is automatically refreshed from the file system to pick up any changes that might be made. For this reason, it is important that you do not make any changes to the runtime configuration using the API if you are using a runtime configuration file (the engine has no way of knowing whether the file or API changes are the “correct” version, so the automatic file refresh assumes that it is correct and will overwrite any changes you have made in the API).

If you are using the runtime configuration file, any changes to the runtime configuration should be performed in the file alone. If you are using the API, the runtime configuration should be managed entirely that way, avoiding the runtime configuration file altogether.