Environments

As part of Meltano’s vision to enable data teams to operate with best practices, Environments allows you to define custom layers of configuration within your project. That way, You can run the same commands against multiple environments, by passing a single environment variable or CLI option.

This reduces the number of environment variables you need to set (and switch between), and at the same time, eliminates the need for managing and toggling between multiple .env files.

A set of environment definitions looks like this within meltano.yml:

environments:
  - name: prod
    config:
      plugins:
        extractors:
          - name: tap-github
            config:
              organizations: [Meltano]
            select: ["*.*"]
        loaders:
          - name: target-snowflake
            config:
              dbname: prod
              warehouse: prod_wh
              batch_size_rows: 100000
    env:
      SOME_PROD_ONLY_SETTING: abc
  - name: dev
    config:
      plugins:
        extractors:
          - name: tap-github
            config:
              organizations: [MeltanoLabs]
            select: ["repositories.*"]
        loaders:
          - name: target-snowflake
            config:
              dbname: dev
              warehouse: dev_wh
              batch_size_rows: 1000

Environments vs Python Virtual Environments

For installable Python plugins (i.e. those with a pip_url property) configured across multiple Environments, the same Python virtual environment and executable are reused.

To install different versions of the same plugin, you can use plugin inheritance and set a different pip_url in the inherited plugin.

Inheritance #

Environments are most powerful when inheriting from a base plugin definition. Configuration that is set in an environment can be used to add or override configuration set in the base plugin configuration. This enables reuse of configuration common to multiple environments while making it easy to switch configuration for a specific environment.

The env mapping #

An Environment can define an env mapping that will be injected into the plugin(s) environment at runtime. Only project-set environment variables referenced in the env mapping will be expanded appropriately. That is, MELTANO_PROJECT_ROOT and MELTANO_ENVIRONMENT.

In the below example, the $MELTANO_PROJECT_ROOT/path/to/a/file.json value will properly read the $MELTANO_PROJECT_ROOT environment variable and inject the full value as $MY_ENV_VAR into the environment.

environments:
  - name: dev
    env:
      MY_ENV_VAR: $MELTANO_PROJECT_ROOT/path/to/a/file.json

Activation #

To use an environment, you can pass the option --environment=<ENV> to the CLI command, or set the MELTANO_ENVIRONMENT=<ENV> variable.

# Using the CLI option
meltano --environment=dev elt tap-github target-sqlite

# Using env vars
export MELTANO_ENVIRONMENT=dev
meltano elt tap-github target-sqlite

Once activated, Plugins and other processes invoked by Meltano can access the current environment via the MELTANO_ENVIRONMENT environment variable available in every Plugins execution environment. If no environment is active, the MELTANO_ENVIRONMENT is populated with an empty string.

Default Environments #

If you have an environment that you’d like to use by default without having to pass the --environment=<ENV> option or set the MELTANO_ENVIRONMENT=<ENV> variable you can set it as the default_environment in your meltano.yml.

default_environment: <ENV>

Example #

When activating an environment, you will observe different configurations are injected by Meltano whenever you invoke an operation in the command line:

$ meltano --environment=prod config target-snowflake
{
  "dbname": "prod",
  "warehouse": "prod_wh",
  "batch_size_rows": 100000
}
$ meltano --environment=dev config target-snowflake
{
  "dbname": "dev",
  "warehouse": "dev_wh",
  "batch_size_rows": 1000
}

Commands with support for --environment #

See the CLI Reference.