Using config modules for environment management

Tutorials
#1

In this post I’d like to showcase how to manage environment configuration using config modules.

Let’s say we have two environments that our solution needs to run in: dev and prod.

We need configuration like path locations and database connections to be specific to each environment.

The working example for this post is available here: config-modules.zip (5.1 KB)

Creating config modules

We’ll need config modules for each environment. Each module has the same set of vars and services defined, but with different values.

The dev environment module:

Screenshot 2020-08-30 at 12.48.29
Screenshot 2020-08-30 at 12.48.29917×490 32.5 KB

The prod environment module:

Screenshot 2020-08-30 at 12.49.07
Screenshot 2020-08-30 at 12.49.07921×476 32.1 KB

A config module makes its vars and services available in a global namespace named $config. We need to set each module’s role to be config, so the module will know to do that. If a module’s role is not set to config, it does not set up the global $config namespace, and it cannot function as a config module.

Now that we have two environments to choose from, we must specify which one is the currently active one.

Selecting the active module in the workbench

You can pick the currently active config module in one of several ways:

  • Pick “Choose Config Module…” from the File menu
    Screenshot 2020-08-30 at 13.01.18

  • If a config module files is open, you can right click its file tab and pick “Set as config module”
    Screenshot 2020-08-30 at 12.59.03

You can see which module is currently active in the lower right corner of the status bar.

env_arrow

Right clicking the status bar display of the current module brings up a context menu that allows you to edit, clear, or change the currently active module.

You can switch the currently active config module at any time.

Referencing environment configuration

The variables and services of an active config module are available in all flows as $config.vars.<varname> and $config.services.<servicename> respectively.

To keep a good overview of config dependencies it’s best to import them into flow properties, and reference them as flow vars and services throughout the flow.

This approach allows you to move config options from locally defined, to config module defined, to parameter derived, etc. without affecting the steps that use the values.

To make this approach simple to follow, there are import buttons below flow variables and services. They appear when there is an active config module selected.

importing_config_values
importing_config_values902×906 731 KB

Config modules on command line

When running on the command line, you pass the active config module via the -g switch like this:

$ bin/engine.sh -g conf/dev.tsm flow.cfl

15:31:39 ¦ INFO  ¦ main     ¦ loading flow flow.cfl
...
15:31:40 ¦ INFO  ¦ Logger.0 ¦ data path from $config: /var/incoming/data/
15:31:40 ¦ INFO  ¦ flow.cfl ¦ flow finished successfully
1 Like