File Triggers and Actions

The pages in this section discuss specific details related to the individual file triggers and actions, including the file move action, file exist trigger, file copy action, and more.

The following concepts and properties are shared across all file triggers and actions. If you are looking for more specific information about a particular type of file trigger or action, refer to the page for that trigger or action.

System Clocks with Remote Servers

If you are using a file trigger or action with a remote server (UNC, FTP, FTPS, SFTP or Amazon S3), take care to ensure that the system clock on the machine where your engine is running matches the clock on the remote machine.

This is especially important if you are using a stable period with a file trigger, or in general, if you are performing any operation using the timestamp of the file. Timestamps are compared against the local date (the date and time from the system clock on the engine server), so if the local date and time do not match the date and time from the remote server, any operations involving timestamps are likely to be compromised.

In short, it is vital that the date, time, and time zone are matched from the engine server to the remote server in order to ensure the integrity of any time-based calculations. We recommend using a Network Time Protocol (NTP) to synchronized system clocks across multiple servers.

File Criteria

Most file triggers and actions can use a File Criteria. The File Criteria specifies what files will participate in file triggers and actions. They also specify the hosts where these files reside. These hosts can be the localhost or any remote host.

Include

File Criteria includes specify the pattern that Flux should use to locate files on the system. Includes should contain both path and filename (or wildcard file pattern) information.

For example:

/path/to/myfile.txt

This include will locate any file with the literal name and extension myfile.txt that is located in the folder /path/to. Any file matching this name and location is included in the file operation.

You can also use the “*” character to include any files that match a particular pattern:

/path/to/*.txt

This include will allow all files with the extension .txt to be included in the file operation.

Similarly, you can simply set the include to “*” to include every file in the directory.

/path/to/*

You can also use a wildcard in the middle of a pattern:

/path/to/a*z.txt

This will include any .txt file that begins with the letter “a” and ends with the letter “z”.

Flux also supports another wildcard character, “?”. Unlike “*” (which matches any number of characters), the “?” will only match a single character.

For example, the include:

/path/to/?.txt

Will match any .txt file whose name contains just a single character. This means the include would match a.txt, but not aa.txt.

Sometimes, you’ll want to extend your include to files not just in a directory, but also any sub-directories under the base directory. There are two ways to do this in a file include.

First, you can use the special syntax “*/”, which indicates that Flux should check the base directory and any sub-directories that are exactly one level below the base directory. This means that the include:

/path/to/*/*.txt

Will match any .txt files in the /path/to/ directory, and any sub-directories exactly one level below the base.

You can also mix-and-match this with other include concepts – for example, to get all .txt files that end with the character “z” that are in the base directory and sub-directories one level lower than the base, you would use the include:

/path/to/*/*z.txt

You can also match all sub-directories lower than the base, even if they are more than one level down. To do this, use the special syntax “**/”. For example:

/path/to/**/*.txt

This will match all .txt files in the base directory and all sub-directories that are at any level below the base.

In Flux File Criteria, the forward-slash (“/”) and backslash (“") characters can be used interchangeably on both Windows and Unix systems. Flux is capable of automatically converting these characters internally to match your file system, so you do not need to worry about adjusting your File Criteria to match the particular file system it will operate on.

The Following Table lists all of the acceptable file criteria formatters and wildcards and their meanings:

Symbol	Meaning
*	Include all file names in the current directory. Directory names are not included. File and directory names in any sub-directories are not included.
**	Include all file and directory names off the current directory and in all sub-directories indirectly reachable from the current directory.
*.txt	Includes all file names ending with the literal “.txt” in the current directory. No files in any sub-directories are included.
?.txt	Includes all file names ending with the literal “.txt” in the current directory that have exactly one character in front of the “.txt” suffix. No files in any sub-directories are included.
*/*.txt	Includes all file names ending with the literal “.txt” in each sub-directory directly beneath the current directory. No deeper sub-directories are included. Does not include any “.txt” files in the current directory.
**/*.txt	Includes all file names ending with the literal “.txt” in the current directory and in all subdirectories.
myfile	Includes the literal name myfile. If the literal name myfile already exists as a file or directory, myfile refers to that file or directory. If myfile does not exist, myfile refers to a file, not a directory.

Whenever a trailing slash is used in the context of searching for files, “**” is silently appended to the symbol. For example, if “mydirectory/” is specified, then “mydirectory/” is silently translated to “mydirectory/**”, which means that all files indirectly reachable from “mydirectory” are included.

If multiple includes are specified on a file criteria, Flux will use “OR” logic to match the criteria. For example, if a file criteria contains the following two includes:

a.txt
b.txt

The file criteria will match if either a.txt OR b.txt exists.

To emulate “AND” logic with file criteria, you can place each include on a separate action or trigger, then have the actions / triggers flow into a join point that waits for all to complete.

Using a Runtime Data Map to pass the RESULT from a File Trigger to the Source of a File Action

When a File Exist trigger executes, you can tell Flux to take the list of files it retrieves and passes it on to the source of a File Action (File Copy, File Move, etc.) for processing. This can easily be accomplished using Runtime Data Mapping.

Double-click on a flow that connects the File Exist Trigger and the File Action, and add a new Runtime Data Map by clicking on the ‘+ Mapping’ button under ‘Runtime Data Map’:

A new menu will appear. In this new menu, you need to set the Runtime Data Map’s source and targets. You’ll need to map the ‘RESULT.url_string_matches’ source to the target property ‘File Criteria Sources’:

Save all after editing and you’ll be good to go.

Exclude

To exclude files from the group of files that will participate in a file trigger or action, call FileCriteria.exclude(). This method accepts a file pattern of files to exclude. This file pattern follows the same rules as defined in the above section. All the included file patterns are processed first, followed by the excluded file patterns.

Suppose that you have three text files in your /path/to/ directory: “a.txt”, “b.txt”, and “c.txt”. Now suppose you want to include all text files except “b.txt”. The following exclude would allow “a.txt” and “c.txt” to be picked up while ignoring “b.txt”:

/path/to/b.txt

Like includes, excludes must contain both path and filename information.

Regex Filter

You can also exclude files by specifying a regular expression filter. Regular expressions are powerful, compact expressions that are used to match strings against patterns. A Regex Filter is simply a regular expression that excludes matching files from the group of files that will participate in a file trigger or action.

Further information about regular expressions can be found online:

http://java.sun.com/docs/books/tutorial/essential/regex/

For example:

fileCriteria.include("*.txt");
fileCriteria.addRegexFilter(".*a\.txt");

The above method calls first to include all text files in the current directory. Then any file ending with “a.txt” is filtered out, using the regular expression “.a.txt”. Note that the regular expression filter is matched against the full path name of each file. Consequently, a leading “.” is needed in front of “a.txt” to match against, for example, c:\temp\a.txt.

Base Directory

All relative file and directory names described in a File Criteria object are relative to a base directory. Relative files require a base directory component to completely specify their location in the file system. Absolute files, on the other hand, do not. For example, c:\temp is an absolute directory name while mydir/myfile.txt is a relative file name. The relative file mydir/myfile.txt needs a base directory in order to completely specify its location in the file system.

By default, the base directory is the current directory. The current directory is the directory where the JVM was started. You can retrieve the name of this directory by accessing the system property user.dir.

You can change the base directory to anything you like.

For example:

fileCriteria.setBaseDirectory("c:\temp");

In the above example, all relative files and directories included in the File Criteria object are relative to the c:\temp directory. However, the base directory is not used by any absolute files and directories in the File Criteria object.

File Copy and File Move Actions

File Copy and File Move Actions have some additional options available when using a File Criteria. Since these options are only used when transferring a file to a new location, they are not available on other action types (like the File Delete Action) that do not transfer files during their operation.

Renamer

Renamers allow you to change the name of a file at the target location during a copy or move operation. There are two renamers available:

Global Renamer

The Global Renamer is used to rename files matching a particular pattern using simple wildcard expressions. The wildcard character, ‘*’, can be used to match zero or more characters within a filename.

Typically Global Renamers are used for changing the extension of a file or group of files. For example, to convert all .java files to .txt, you could use a Global Renamer with the following settings:

From File Pattern

*.received

To File Pattern

*.validated

The renamer “From File Pattern” and “To File Pattern” must contain one wildcard ‘’ each. The characters matched in the From Pattern will replace the “” character in the To Pattern when the file action runs (for example, if the From Pattern is “*.zip” and the To Pattern “.txt”, and the action matched a file “my.zip”, the file would be renamed to “my.txt” when the action executed).

Regular Expression Renamer

The Regular Expression Renamer is used to rename files matching a particular regular expression pattern. For example, to modify all filenames ending with abc.txt and make them end instead with xyz.txt, you could use the following settings:

From File Pattern

(.*)abc\.txt

To File Pattern

\1xyz.txt

The pattern “<number>” syntax can be used to access a matched portion from the regular expression in the From File Pattern (represented by “(.*)”). “\1” retrieves the first matched portion, “\2” the second, and so on.

For more information on regular expression usage, see Oracle’s guide to regular expressions.

Preserve Directory Structure

The Preserve Directory Structure flag indicates whether Flux should attempt to recreate the directory structure leading to a file after it is moved or copied.

In order to use this flag, the include pattern that matches a file must contain the sub-directory matching pattern, “**”.

When this flag is enabled, a new directory structure is taken on the target, beginning from the base directory of the include and ending with the directory where the source file actually resides. The base directory is the directory path of the include that occurs before the “**” pattern.

For example, consider a File Criteria with a source Base Directory of:

/home/flux/**/*.*

That matches a file named:

/home/flux/path/to/myfile.txt

And has a target Base Directory of:

/mytargetfolder/

If Preserve Directory Structure is enabled, then the final path of the file on the target will be:

/mytargetfolder/path/to/myfile.txt

On the other hand, if Preserve Directory Structure is disabled, the final path of the file on the target will be:

/mytargetfolder/myfile.txt

Network Hosts

FTP Hosts

The files and directories specified in a File Criteria object normally refer to files and directories on the local file system. However, you can specify that these files are located on a remote FTP server. This functionality allows you to specify files and directories on a remote FTP server that should be used by the different file triggers and actions.

For example, by setting an FTP host, you can copy files to and from an FTP server. You can also have a File Trigger fire when a file on an FTP server changes its state.

To create and initialize an FTP host, first, create a FtpHost object from a FileFactory object. The FtpHost object contains methods for setting the name of the FTP host, the port number, and username and password information for logging in. Finally, the FTP host is set by calling:

fileCriteria.setHost(ftpHost);

After calling this method, the File Criteria’s included files and directories are relative to the default base directory on this FTP host. Furthermore, calling fileCriteria.setHost() resets the base directory to the default base directory for FTP servers, “/”. To change the base directory to a different directory on the FTP server, call fileCriteria.setBaseDirectory() after calling fileCriteria.setHost(). Note that in this case, the base directory is relative to the FTP server, not the local file system. This way, files and directories on the FTP server can be included and excluded in the usual way using File Criteria objects.

If your FTP server does not display directory listings in the standard format, you can substitute your own parser to adapt non-standard FTP directory listings to the scheduler. You need to create a class with a default constructor that implements the flux.file.FtpFileListParser interface and provide that class to your FtpHost object using the setFileListParser() method.

Non-standard FTP Servers

Some FTP servers, such as special-purpose FTP servers used for EDI purposes, return non-standard file listings when responding to FTP commands. Flux supports these non-standard FTP servers by allowing the workflow to supply its own Java class that is responsible for parsing the non-standard file listings. Thus, Flux can monitor and manipulate files that reside on non-standard FTP servers.

Use the FtpHost.setFileListParser() method to register your custom file listing parser. This class must implement the flux.file.FtpFileListParser interface.

Secure FTP Host

A Secure FTP Host is exactly like an FTP Host, except that it operates on secure FTP servers instead of standard FTP servers. Secure FTP servers encrypt the communications traffic for security purposes.

When using Secure FTP (SFTP) with the Flux Operations Console, workflows and error messages may have problems being displayed inside of the web application due to classpath issues. To prevent this problem from occurring either place the flux.jar file inside your invoking JVM’s lib/ext directory, or include flux.jar in the system classpath.

Using a Username / Password with a Private Key

When using a Secure FTP Host, you can set a username/password while using a private key. In the editor, just set the username/password as normal, then click the “Set Private Key” button to set the private key. Your username and password credentials will be saved in addition to the key settings.

FTP Over SSL

FTP over SSL (FTPS) encompasses both FTP and SFTP functionality. FTPS provides a way to perform secure file transfers using an SSL/TLS layer. When connecting to FTPS servers using Flux, the connection may take longer than usual to initialize depending upon which type of connection strategy the FTPS server is utilizing. The possibility of an extended connection wait is due to a series of connection strategies tried until the proper strategy is found. Connection via the TLS layer is first attempted, the second attempt is a connection over SSL, and lastly, an attempt is made to connect via implicit SSL. Each attempt is given five seconds to connect. If the connection can not be made within the time frame, the next connection strategy is attempted.

Supported FTP Servers in Flux

Flux explicitly supports two File Transfer Protocol (FTP) servers, although other FTP servers are known to work with Flux. These servers are listed below.

• Microsoft’s Internet Information Services (IIS) FTP server version 5.0+
• Very Secure FTP server (VSFTP) 2.0+

Although the above servers have been thoroughly tested against Flux and are fully supported, other FTP servers have been known to work correctly. If you are using any other FTP servers other than what is listed above, note that they have not been tested and are not officially supported by Flux.

IIS 5.0 NOTE: When using IIS 5.0, a problem occurs when changing into directories with spaces in their names. A workaround is to either enable the “Use CD Commands” option in the Flux Designer under the FTP Host section within the “File Criteria” editor or set the “issueCdCommandsMode” flag to true on a flux.file.FtpHost object in Java code.

Passive Mode

Passive network connections are firewall-friendly but require support from the FTP server. The passive mode option is available on FTP, Secure FTP, and FTP over SSL hosts. Connections established with these hosts are passive by default. If passive mode is disabled, active mode will be used.

Issue Cd Commands

The Issue Cd Commands option specifies how Flux changes directories when using an FTP, Secure FTP, or FTP over SSL host. If issue cd commands are enabled, Flux will issue a CD command to the server for each directory. If the option is disabled, Flux will attempt to issue a single CD command for the entire path. This option is useful for decreasing the number of commands Flux issues to your server. Issue CD commands are disabled by default.

NOTE: Certain FTP servers do not support specifying an entire path with the CD command.

Binary/ASCII Transfer Mode

File transfers with Flux can be done in two modes, Binary or ASCII. Binary mode is for byte-oriented files while ASCII mode is used for text-oriented files. This option can be set on an FTP, Secure FTP, or FTP over SSL host and will be used for all files transferred by that host. Binary transfer mode is enabled by default.

Universal Naming Convention (UNC)

Flux also supports connection to network devices via UNC. When using a UNC host in Flux, the user may specify an optional domain, username, password, and port. The name of the server and file being used are required fields.

Specify a Default Username, Password, and Domain for UNC Hosts

You can specify default username, password, and domain settings by setting the following JVM options on the JVM where the Flux engine runs:

-Djcifs.smb.client.username=<username> -Djcifs.smb.client.password=<password> -Djcifs.smb.client.domain=<domain>

Where <username>, <password>, and <domain> are replaced (respectively) with the username, password and domain settings for your UNC server.

Once these properties are in place, any file trigger or action that uses a UNC host and does not have a username, password, or domain specified will use the settings defined above.

A value set directly on the trigger or action will override the default value, allowing you to configure specific values for particular actions while retaining the default settings for other actions in your workflows.

These settings are applied system-wide and are not dependent on workflow namespace.

Amazon S3

Flux supports connections to Amazon S3 Cloud Object Storage buckets. The user would need to specify the Region, Bucket, Access ID, and Secret Access ID.

Specifying Defaults when Flux Runs as a Windows Service

You can specify these default settings by adding the following lines to the engine.conf file for your Windows service:

wrapper.java.additional.13=-Djcifs.smb.client.username=<username>
wrapper.java.additional.14=-Djcifs.smb.client.password=<password>
wrapper.java.additional.15=-Djcifs.smb.client.domain=<domain>

Zip File Criteria

A Zip File Criteria is the same as a File Criteria, except that the included and excluded files are relative to the files contained in a given Zip file. The base directory is also relative to the root of the Zip file itself.

After creating a ZipFileCriteria object, you must specify the actual name of the Zip file by calling ZipFileCriteria.setZipFilename().

For example, suppose a Zip file a.zip contains the files /myfile.txt and /mydir/myOtherFile.txt. You can specify these two text files in a Zip File Criteria as follows:

zipFileCriteria.setZipFilename("a.zip");
zipFileCriteria.include("*/.txt");

These two method calls include the text files /myfile.txt and /mydir/myOtherFile.txt from inside the Zip file for use in a file trigger or action.

Zip File Criteria objects can be used for creating Zip files, copying files into existing Zip files, and extracting files from Zip files.

File Actions and Transaction Breaks

If you are using file actions (including the file copy, move, delete, and create actions) within a workflow, and the workflow fails or must be rolled back to the last transaction break for any reason, it is important to note that any work performed on the file system will not be rolled back along with the action itself.

This means that, although the state and execution of the workflow are rolled back in the database as normal, any file operations — meaning any file that was deleted, moved, copied, or created — are retained on the file system. If and when the workflow begins re-executing, the file action will simply perform its operation again, overwriting any files (or partial data) that were written to the system on the previous rolled-back attempt.

If instead, you need the file system to be cleaned up or “reset” to a different state, you will need to design an error handler or error flow that performs the file cleanup. For example, if your workflow contains a file move action that might be rolled back, your error handler could contain a corresponding file move action to reverse the transfer in the event of an error or rollback.

File Infos

Some file triggers and actions return results that contain File Infos. This is a special data type that contains information about a file found or operated on by the trigger or action.

The file info is a complex data map, meaning it has several properties available containing specific information about the file it represents. If you have a File Info variable, you can use variable substitution to access each property (also called a “field”). For example, if a File Info is stored as the variable “fileinfo”, you could use variable substitution to access the filename property: ${fileinfo.filename}.

The available properties for a File Info are:

Property Name	Description
filename	The name of this file (without path information).
lastModified	The date that the file was last modified on the file system.
parent	The system-dependant absolute path to the parent of this file (in other words, the absolute path of the directory where this file is stored).
read	A boolean (true/false) value indicating whether this file can be read.
size	The size of the file (in bytes).
url	The URL to the file.
write	A boolean (true/false) value indicating whether this file can be modified.