Flux 8.1 Features

Flux orchestrates file transfers and the scheduling of batch processes. Flux then manages batch processes performed against this data, including conversion of data, data compression, encryption, enterprise-specific processes as well as secure transfer for backup, replication, and distribution of files. IT organizations realize that not all enterprise processing is interactive and web driven. Flux orchestrates the transfer of high volumes of data and business transactions across different platforms and systems automatically. As enterprises expand the sharing of data across disparate platforms, operating systems, and database management systems, reliable file transfer and batch process orchestration becomes indispensable.

Flux Features Overview

  • Flux is used to execute a sequence of processing actions and manage the transfer of files automatically.
  • These process sequences are called Flux workflows and they are constructed using a flowchart-like metaphor graphically, or they can be constructed using Java code.
  • Flux supports complex dependencies, parallel and multi-threaded processing, and work distributed across multiple computers.

Key Functions

  • Launch executable files and shell scripts automatically. Within Flux these are referred to as Process Actions.
  • Jobs (i.e., the execution of a Flux workflow) can be started based on triggering events. These events include: timed calendar (regular or business calendar) events, monitoring of incoming files, results of database queries, status of other workflows, incoming mail, and API events initiated by external applications.
  • Flux jobs can submit mail, perform database queries and updates, run stored procedures, submit web service requests, and even initiate other workflows.

Key Features

  • Flux can run standalone as a scheduler and workflow engine or be embedded within a Java application.
  • Job activities can be limited to time slots. Flux supports any number of time slots (called Business Intervals), which can be configured according to individual requirements. Support for multiple business calendars provided.
  • Flux allows the assignment of workflow priorities to prioritize the scheduling of execution of workflows.
  • Job history is stored in a database.
  • Standard job packages e.g. for log rotation and cleanup, sanity checking, job execution by remote Flux engines or agents, FTP file transfer.
  • Notifications for job execution results per e-mail, configurable logging and monitoring of logs.
  • API to implement workflows and workflow scripts, e.g. for complex conditional processing.
  • Restrictable graphical user interfaces: A built-in interface for job control and a GUI for managing configurations for several Flux instances on different server systems.
  • Support for users and groups via LDAP and Active Directory.
  • Flux engine and user interface can be extended with application-specific features by customer-developed custom code.

Mode of Operation

  • Used for batch scheduling, Flux runs as a Unix daemon or Windows service in the background. Flux can also be run in the foreground from a command line.
  • Job control is carried out by the command line, REST calls via Curl scripts, or by the built-in graphical user interface - The Flux Operations Console.
  • A graphical editor - The Flux Designer - for workflow design is provided. A Java API can also be used to construct workflows programmatically.

Job Execution

  • Within Flux, jobs are individual running instances of workflows.
  • Workflows are the basic unit for the processing of executable files, shell scripts, file transfers and other workflow features of Flux.
  • Workflows can be executed independently from one another. Depending on the execution result, i.e. exit code signaling success, failure or a specific exit code, of a workflow any number of successor workflows can be started.
  • Workflows can be executed in parallel up to a configurable number of simultaneous tasks.
  • Workflow dependencies based on execution results of prior workflows can be configured.
  • Monitoring of directories can trigger workflow starts, this allows the integration of legacy applications into the business workflow.
  • Workflows execute file transfers based on the built-in FTP and SFTP capabilities as a means to integrate applications.
  • Workflow starts are triggered by built-in calendar, by other workflows, by file watchers, by email messages, by database conditions, by command line or by web interface. These triggers can be combined to support complex dependencies.
  • Other applications can start jobs or otherwise control the Flux environment via APIs including command line, Java, and REST.

Flux Engine Features

  • API to implement workflows and workflow scripts, e.g. for complex conditional processing.
  • Restrictable graphical user interfaces: A built-in interface for job control and a GUI for managing configurations for several Flux instances on different server systems.
  • Secured and filtered access for users and groups via LDAP and Active Directory.

Additional Features

  • High-availability Cluster: a Flux cluster is 2 or more Flux engines sharing their state information in a shared database. No additional software beyond multiple Flux licenses and a shared database is required to create a Flux cluster. A Flux cluster ensures fail-safe operation and fault tolerance with automatic fail-over. A fail-safe system consists of multiple Flux engines running on different computers.
  • Database Reconnect: Automatically suspends operation, waits, and attempt reconnection in the event of a database failure or loss of network connectivity to the database.
  • Load Sharing: For a high volume of data with time consuming processing using multiple Flux engines will speed up the processing time considerably and provide higher availability. In load sharing mode the processing tasks are shared between multiple Flux engines that are handling distributed workflows on more than one host. Use of Flux agents (see below) can further distribute the workload.
  • Reporting: Embedded support for executing Jasper Reports.
  • Secure Shell: Support provided for remote execution, SCP Secure Copy and SFTP Secure File Transfer.
  • Message Queues: Can read and write JMS message queues.

Flux User Interface Features

  • User interface is run within a browser. Chrome, Safari, Firefox, and certain versions of Internet Explorer are supported.

Flux Workflows

  • Workflows are saved into the Flux Repository database and into files as XML.
  • Workflows are constructed using the Flux Designer user interface and the Flux Java API.
  • Workflows consist of actions and triggers. Actions perform tasks and triggers listen for events.
  • Workflows can be designed with variable at design time, and provisioned with values at initiation and during workflow execution. Variables can be passed internally within workflows between actions and triggers to configure actions and triggers and control workflow processing.
  • Workflows can be templated - and then submitted to the Flux engine under different namespaces. Different namespaces can be provisioned with variables at runtime so that the same workflow template can be reused in many different namespaces with different variable values.
  • Workflows can have embedded forms (created using a simple JSON text format) that request information when the workflow is submitted (aka provisioning) and when the workflow is executing (aka claiming). Any information entered into the workflow is then available downstream in the workflow for processing or routing decisions.

Implementation

  • Flux is written entirely in Java for Java 8, 9, 10, and 11.

Supported Platforms

  • Flux can be operated in 32 bit and 64 bit environments that have a Java virtual machine (JVM) installed.
  • Support for JDK 1.8 through 11.

Security

  • User access can be authenticated to LDAP or Active Directory.
  • Flux provides an internal security database if external authentication is not required.
  • Database, file server, mail server, and other password protected resources can use an encrypted password in all dialogs and configuration files. Passwords are encrypted using a Flux-provided command line utility.
  • Users assigned to roles. Roles can restrict access to specific feature and functions.
  • Roles can filter access to specific groups of workflows and limit access to read-only, partial operations, or full edit capabilities.
  • User operations on workflows is audited.

Reporting

  • Audit and execution reports provided.
  • Report execution is done via provided Flux workflows. Frequency of reports and dependency on execution is easily configurable as standard Flux workflows.
  • Reports can be extended using Jasper reports.

Process Action Features

  • The Flux Process Action, also known as the “Command Line Action”, can execute command line programs, batch files, scripts (including Powershell), and, more generally, any native program.

  • Processes can execute in the foreground (meaning that the workflow will wait for the Process Action to finish running before moving on), or asynchronously in the background (the workflow will move on immediately while the Process Action runs).

  • If the process prints output to standard in (stdin), standard out (stdout), or standard error (stderr), the Process Action can be configured to store that output in a file, or simply direct that output to the engine’s standard out / in / error.

  • The process’s working directory and its environment can be set in the Process Action control when and how the process is executed.
  • The Process Action can be configured to terminate the process if a user interrupts the workflow, if a signal is raised, or if the action times out.
  • The Process Action can be configured as to if (and when) the Process Action should invoke Flux’s Error Handling mechanism by setting an exit code or codes that indicates that an error has occurred.
  • The Flux SSH Command Action, used for the remote command execution on Linux and Unix hosts, provides many of the same features as the Flux Process Action.

Error Handling

  • Flux provides an error handling mechanism that allows defining remediation steps either directly within a workflow, or through a general series of steps that can attempt to recover from the error, notify a user that the error has occurred, or display a message in the Operations Console.

  • Error handling is done in one of two ways: through an error flow, a specialized flow that allows you to take immediate and specific action for any errors that occur on a given step; or through an error handler, a specialized workflow that allows you to define how workflows react to errors where no error flows are available.

  • Different error handlers can be defined for different namespaces (grouping of workflows), that react to errors differently depending on the type of workflow that’s running, the priority of a workflow, and the impact that the workflow will have on the overall application.

Basic File Features

  • Send and receive, e.g. move and/or copy, files by FTP to/from a target host.
  • A wide range of protocols:
    • FTP, SFTP, FPTS,
    • File System, UNC names, Samba (SMB)
  • ASCII and binary file transfer modes *Host to host file transfer without touch down, using different protocols, e.g. transfer from SFTP to FTP.
  • Execute commands by SSH on the source and/or the target host at the beginning or the end or during transmission.
  • Send files by FTP or SFTP to a jump host and forward them by FTP or SFTP to a target host.
  • Different transfer protocols can be used between localhost and jump_host and between jump_host and target host.
  • Receive files from a remote host by FTP or SFTP to a jump host and forward them by FTP or SFTP to the localhost.
  • Different protocols can be used for transfer between the hosts.
  • Password and public key authentication are supported for SFTP.
  • File transfer parameterization by command line parameters and/or by configuration files.
  • There is a common syntax for parameters, no matter what transfer type is configured.
  • All parameters are specified on the client exclusively.
  • Logging and error handling are provided; errors are detected on a per-file basis.
  • A test mode

Advanced File Features

  • Selection of files whose names are matched by a regular expression.
  • Polling (waiting) for files. *Automatic reconnect in event of loss of connection.
  • Checking the steady state of all files before starting transfer.
  • Recursive transfer of files in subfolders.
  • Atomic Transfer to avoid any partially transferred file being picked up by directory watchers.
  • Transactional Behavior for file transfers.
  • Check transferred files for completeness.
  • Handling of empty files.
  • Renaming of files.
  • Support for UNC paths in directory names.
  • Track record of file transfers.
  • Parallel file transfer (multi-threading).
  • Creating and checking hash values (e.g. MD5).
  • Zip files or groups of files.
  • Rename files without transfer.
  • Delete files without transfer.
  • Pre- and post-processing of file transfers.
  • PGP encrypt and decrypt of files.

Database Features

  • Support for Oracle, MySQL, MariaDB, SQL Server, Postgres, and DB2. Also support HSQL, H2 for in-memory use. Apache Derby for non-production use.
  • Support for stored procedures.
  • Perform queries, updates, and deletes using standard SQL.
  • Can access multiple databases, on multiple platforms from different database vendors, within the same workflow.
  • Polls to trigger workflow actions based on database conditions via SQL query.
  • Each database row can be processed through a set of tasks within the workflow or pulled in bulk into a list for downstream processing.
  • Database query results can be pushed in its entirety into a list for later processing in downstream actions.

Mail Features

  • Support for POP3 and IMAP mail servers such as Exchange, Gmail, and Office 365.
  • Can listen for incoming emails to a specific user and inbox.
  • Download and extract attachments.
  • Support for TLS and SSL connections.
  • Support for text and HTML formats.
  • Can send emails from multiple email servers within a single workflow.

Agent Features

  • Support for Windows, Linux, and AIX Flux Java Agents.
  • Commonly used to delegate compute intense processes to remote machines, or to perform file transfers in a DMZ.
  • Agents communicate via HTTPS/REST to Flux engines for work assignments.
  • Support for dozens of agents against a Flux engine (or a Flux cluster).
  • Agents can run processes and file actions.
  • Agents can automatically failover to another Flux engine if their primary engine goes down.
  • Agents can run multiple actions concurrently.
  • Stdout, Stderr, and action results conveyed back to the engine for use in workflow decisions.

Variable Management

  • Flux provides support for three levels of variable management.
    • Variables within the scope of an entire workflow
    • Variables within the scope of an individual flow within a workflow.
    • Variables within the scope of an individual action or trigger within a workflow.
  • Variables can be inserted and retrieved via Java code, scripting code within actions and triggers, and via the user interface for Flux.
  • Variables are stored in the Flux database at defined transaction breaks defined within workflows.
  • Variables with large amounts of data can be marked as transient, and not stored in the database.

Java Code Integration Features

  • Support for executing Java code written to the Flux listener interface, or using reflection to execute third-party Java libraries.
  • Support for calling Java via RMI.
  • Documented support for adding custom actions and triggers to extend Flux.
  • Support for reading and writing JMS queues.

Web Service Features

  • Support for submitting web service requests to remote services.
  • REST and SOAP requests supported.

Version Control Support

  • Writes and overwrites workflows into a VCS managed directory for VCS control, or creates a backup copy of the workflow with timestamp and user name performing the save appended to it.

  • This event occurs after a successful save to the Repository.