Overview

Expression libraries are files (.expr) that contain one or more expressions that can be imported into a Pipeline for use across all expression properties. Libraries can be useful if you have static metadata or common functions that you wish to reference in a Snap's expression properties, but may require a strong familiarity with JavaScript syntax.

When a Pipeline is executed, the contents of all of the expression library files are evaluated and the results are stored as properties in the lib global variable. Note that the values that make up the libraries are static and cannot be changed during Pipeline execution.

You can create a library file using your favorite text editor to build the expression. You can make an object literal containing all the values and functions to use.

Creating an Expression Library

Example 1: Basic Library with Function

{
    prefix: 'test',
    prefixer: x => this.prefix + x
}

Placing this content in a file named "helpers.expr" and uploading it to your project will make it available for use in Pipelines.

Importing a Library into a Pipeline

To import a library into a Pipeline:

1. Open the Pipeline Properties dialog in the Designer and find the Expression Libraries table near the bottom.
2. Add a row to that table and then click the file browser icon to select the file to import.
3. Click Save.

After saving, all imported libraries will be available for use through the lib variable in the expression language under the name of the file without the extension.

Using Expression Libraries

For example, to reference the "prefix" property in the "helpers.expr" library, you would write:

lib.helpers.prefix

And, to call the "prefixer" function, you would write:

lib.helpers.prefixer($.name)

If you would like to explicitly set the name of the library in the lib variable, you can fill out the "As" column in the Expression Libraries table in the Pipeline Properties dialog.

Advanced Example: Date Calculations

Example 2: Previous Month Start and End Dates

The following expression library can be used to generate the start date and end date of the previous month in multiple formats.

{
    // Convert the given Date-with-time object into a Date with only the year and month
    to_month_date: x => Date.UTC(x.getFullYear(), x.getMonth() - 1),

    // Get the start of the previous month based on the given date or the current time if no parameters are given
    prev_month_start: x => this.to_month_date((x || Date.now()).minusMonths(1)),

    // Get the end time of the previous month based on the given date or the current time if no parameters are given
    prev_month_end: x => this.to_month_date((x || Date.now()).withDayOfMonth(1)).minusSeconds(1),

    prev_month_start_string: x => this.prev_month_start(x).toString().split('T')[0],
    prev_month_end_string: x => this.prev_month_end(x).toString().split('T')[0],
    prev_month_start_epoch: x => this.prev_month_start(x).getTime(),
    prev_month_end_epoch: x => this.prev_month_end(x).getTime()
}

Variadic Functions with Spread Operator

Example 3: Variable Number of Parameters

In the following example, the spread operator ...other can be used to indicate that zero or more parameters can be accepted. As a result, the CONCAT function has two or more parameters. This function returns the concatenation of all parameters.

{
    CONCAT: (x, y, ...other) => other.reduce((accum, curVal) => accum + curVal, x.toString() + y.toString())
}

The 'this' Variable

You may have noticed in the example expression library listed above that the function references the this variable. When defining an object literal, the this variable is set to the object being constructed so that you can reference other functions and data values.

The overlay works like the "merge()" method on objects where existing properties are updated instead of replacing the entire object.

Built-in Variables

The following variables are available for use within the expressions in the library:

• __path__ - The path to this library as it was listed in the imports for the Pipeline.
• __name__ - The name of this library as it was derived or given in the imports for the Pipeline.

Overlaying Libraries

The values in one library can be overlaid over another by giving them the same name (that is, setting the "As" column in the import list to the same value). This feature allows you to create a library with default values/functions that can then be specialized by importing other libraries that override the default values.

As an example, if the following two code blocks were in separate files and imported with the name "stuff":

File 1:

{
  "num": 1,
  "str": "Hello, World"
}

File 2:

{
  "num": 2
}

The resulting value of library would be as follows since the "num" value in the second file would override the value in the first file:

{
  "num": 2,
  "str": "Hello, World"
}

Complete Example: Data Transformation

In this example, we have a set of status updates from certain account types. For example, let us assume the following JSON:

[
    {
        "id": "123792837",
        "status": "Done",
        "type": "Checking"
    },
    {
        "id": "543793437",
        "status": "Done",
        "type": "Savings"
    },
    {
        "id": "136792898",
        "status": "some-error",
        "type": "Home Loan"
    }
]

We want to move this data to another system that uses different values for status and account type, thus implementing a rule that:

• For attribute "status":
  • If status = "Done" then make the status = "COMPLETED"
  • If status contains "error" then make the status = "ERROR"
• For attribute "type":
  • If type = "Checking" then make the type = "CHK"
  • If type = "Savings" then make the type = "SAV"
  • If type = "Home Loan" then make the type = "MOR"

We create an expression library file, demo.expr, that contains the following:

{
    typeMap: {
        Checking: "CHK",
        Savings: "SAV",
        "Home Loan": "MOR"
    },

    convertStatus: x => (x == "Done" ? "COMPLETED" : (x.indexOf("error") != -1 ? "ERROR" : "UNKNOWN"))
}

Where lines 2-5 map the attribute type and line 7 sets the status.

The demo.expr file is added to our Pipeline through the Expression Libraries section of the Pipeline Properties.

In a Mapper Snap, we create the following expressions:

• lib.demo.convertStatus($status) to a Target path of $updated_status. This uses the lib variable to call convertStatus property in the demo library to use on the incoming $status field.
• lib.demo.typeMap[$type] to a Target path of $updated_type. This uses the lib variable to call typeMap property in the demo library to use on the incoming $type field.

Result:

[
    {
        "id": "123792837",
        "updated_status": "COMPLETED",
        "updated_type": "CHK"
    },
    {
        "id": "543793437",
        "updated_status": "COMPLETED",
        "updated_type": "SAV"
    },
    {
        "id": "136792898",
        "updated_status": "ERROR",
        "updated_type": "MOR"
    }
]

Best Practices

• Naming: Use descriptive names for library files and properties to make them easy to understand and use.
• Organization: Group related functions and data together in the same library file.
• Documentation: Use block comments (/* */) to document complex functions and their parameters.
• Reusability: Design libraries to be generic and reusable across multiple pipelines.
• Testing: Test library functions thoroughly before using them in production pipelines.
• Version Control: Keep expression library files in version control along with your pipelines.
• Static Data: Remember that library values are evaluated once at pipeline start and cannot be changed during execution.
• Overlaying: Use library overlaying to create environment-specific configurations (dev, test, prod).
• this References: Use this to reference other properties within the same library object.

Common Use Cases

• Data Mapping: Define mappings between different data formats or systems.
• Constants: Store configuration values, API endpoints, or other constants.
• Utility Functions: Create reusable functions for common transformations.
• Date Calculations: Define complex date/time manipulation functions.
• Validation Rules: Centralize validation logic for data quality checks.
• String Formatting: Create custom string formatting functions.
• Environment Configuration: Use overlaying to manage environment-specific settings.
• Business Logic: Encapsulate business rules that are used across multiple pipelines.