Meltano 2.0 Migration Guide

Note: This document is still a work in progress. Expect further changes, coming soon.

The following list includes all recommended migration tasks as well as breaking changes in Meltano version 2.0.

Migrate to an Adapter-Specific dbt Transformer #

If you previously used the dbt Transformer, we recommend migrating to an adapter-specific installation as per the dbt available adapters documentation.

Install dbt #

This is easy to do! Following the instructions from above to discover and install your chosen adapter:

# list available transformer plugins
meltano discover transformers

# install adapter-specific dbt, e.g. for snowflake
meltano add transformer dbt-snowflake

Update your dbt_project.yml #

Installation of a new Transformer will introduce two important files to your transform/ directory:

  • A new profiles.yml file in transform/profiles/<adapter name>/profiles.yml
  • A new dbt_project.yml file in transform/dbt_project (<adapter name>).yml

The new profiles.yml will only be used by adapter-specific dbt executions (e.g. dbt-snowflake), and can be customized to meet your requirements. Your existing profiles.yml will remain in use by your existing dbt Transformer plugin (via elt and invoke).

It is likely that the new dbt_project (<adapter name>).yml will contain changes from your previous dbt_project.yml file, especially if you haven’t already upgraded to dbt v1.0. To complete your migration, consolidate dbt_project.yml and dbt_project (<adapter name>).yml into a single file called dbt_project.yml. As this project file will be used by both dbt and dbt-<adapter> Transformer plugins by default, you must ensure you are running an up-to-date installation of plugin dbt if you intend to use both adapter-specific and legacy dbt installs together (not recommended).

If you make use of Transform plugins, these will continue to work as regular dbt packages. However adding new Transform plugins will currently (tracking at #3304) re-add the legacy dbt Transformer plugin. To avoid this we recommend adding Transforms as regular packages directly via dbt as per the dbt Packages documentation.

Remove the dbt Transformer plugin and associated files #

To remove the legacy dbt Transformer plugin, run:

# remove the transformer `dbt`
meltano remove transformer dbt

# remove the file bundle `dbt`
meltano remove files dbt

Removing a file bundle does not remove any files from your transform/ directory. Manually remove transform/profile/profiles.yml to complete clean-up (as adapter-specific installs come with their own profiles.yml in transform/profiles/<adapter name>/profiles.yml).

Removed #

model and dashboard plugin types #

These plugin types provided very basic BI capabilities using Meltano UI. However there are already great 3rd party open source BI solutions in this space, such as the newly added Superset plugin. Meltano model and dashboard plugins have been removed in favour of existing and future 3rd party tools for the same purpose.

transform support in meltano elt #

Meltano 2.0 continues to support extract-load (EL) operations with meltano elt. However, for EL+T operations which also need to transform data, please use meltano run.

transform support in Meltano schedules #

Meltano 2.0 continues to support extract-load (EL) operations in schedules. However, for EL+T operations which also need to transform data, please use the new meltano job add command to create a job definition and then specify the new job name in your schedule.

env_aliases in Plugin config #

As part of our effort to streamline the configuration experience in Meltano, we are deprecating the env_aliases attribute of plugin definitions. Previously env_aliases provided two functions:

  1. Sourcing setting values from the terminal by a name other than the default environment variable (of the form <PLUGIN_NAME>_<SETTING_NAME>).
  2. Writing setting values into the plugins’ runtime environment under an environment variable name other than the default.

For sourcing setting values we encourage users going forward to make use of the default environment variables (of the form <PLUGIN_NAME>_<SETTING_NAME>) for settings in most cases. These can be conveniently found for a given plugin by running meltano config <plugin> list. In cases where an environment variable of a name other than the default must be used to source a setting value, Meltano supports referencing in meltano.yml. For example:

plugins:
  extractors:
    - name: tap-gitlab
      config:
        ultimate_license: $GITLAB_API_ULTIMATE_LICENSE

This will take the value from the environment variable GITLAB_API_ULTIMATE_LICENSE and use it configure the tap-gitlab setting named ultimate_license.

For writing setting values into the plugins’ runtime environment under an environment variable name other than the default, we support the env: key for setting definitions. These can be added or overridden in your meltano.yml file. For example:

plugins:
  extractors:
    - name: tap-gitlab
      settings:
        - name: ultimate_license
          env: GITLAB_API_ULTIMATE_LICENSE

This will create an environment variable called GITLAB_API_ULTIMATE_LICENSE in the plugins’ runtime environment with the configured value of the setting ultimate_license.

Updating your Project #

Before v2.0 Meltano made use of env_aliases internally and in several common plugins. To ensure Meltano and those plugins continue to work as expected, references to the deprecated environment variables in your own project should be replaced. The table below contains details of the env_aliases that have been removed, and the corresponding default setting environment variable to use in each case. If for any reason you wish to keep sourcing or writing setting values to deprecated aliases, we recommend using the techniques detailed above to replace the functionality formally provided by env_aliases.

Plugin Variant Deprecated Replacement
meltano MELTANO_API_HOSTNAME MELTANO_UI_BIND_HOST
MELTANO_API_PORT MELTANO_UI_BIND_PORT
PORT MELTANO_UI_BIND_PORT
WORKERS MELTANO_UI_WORKERS
WEB_CONCURRENCY MELTANO_UI_WORKERS
FORWARDED_ALLOW_IPS MELTANO_UI_FORWARDED_ALLOW_IPS
MELTANO_READONLY MELTANO_UI_READONLY
MELTANO_AUTHENTICATION MELTANO_UI_AUTHENTICATION
MELTANO_NOTIFICATION MELTANO_UI_NOTIFICATION
TAP_ADWORDS_OAUTH_CLIENT_ID MELTANO_OAUTH_SERVICE_GOOGLE_ADWORDS_CLIENT_ID
TAP_ADWORDS_OAUTH_CLIENT_SECRET MELTANO_OAUTH_SERVICE_GOOGLE_ADWORDS_CLIENT_SECRET
OAUTH_GITLAB_APPLICATION_ID MELTANO_OAUTH_GITLAB_CLIENT_ID
OAUTH_GITLAB_SECRET MELTANO_OAUTH_GITLAB_CLIENT_SECRET
MELTANO_CLI_TRACKING_ID MELTANO_TRACKING_IDS_CLI
MELTANO_UI_TRACKING_ID MELTANO_TRACKING_IDS_UI
MELTANO_EMBED_TRACKING_ID MELTANO_TRACKING_IDS_UI_EMBED
dbt-bigquery meltano DBT_PROFILES_DIR DBT_BIGQUERY_PROFILES_DIR
dbt-postgres meltano DBT_PROFILES_DIR DBT_POSTGRES_PROFILES_DIR
dbt-redshift meltano DBT_PROFILES_DIR DBT_REDSHIFT_PROFILES_DIR
dbt-snowflake meltano DBT_PROFILES_DIR DBT_SNOWFLAKE_PROFILES_DIR
tap-adwords meltano OAUTH_GOOGLE_ADWORDS_CLIENT_ID TAP_ADWORDS_OAUTH_CLIENT_ID
OAUTH_GOOGLE_ADWORDS_CLIENT_SECRET TAP_ADWORDS_OAUTH_CLIENT_SECRET
OAUTH_GOOGLE_ADWORDS_DEVELOPER_TOKEN TAP_ADWORDS_DEVELOPER_TOKEN
singer-io OAUTH_GOOGLE_ADWORDS_CLIENT_ID TAP_ADWORDS_OAUTH_CLIENT_ID
OAUTH_GOOGLE_ADWORDS_CLIENT_SECRET TAP_ADWORDS_OAUTH_CLIENT_SECRET
OAUTH_GOOGLE_ADWORDS_DEVELOPER_TOKEN TAP_ADWORDS_DEVELOPER_TOKEN
tap-bigquery anelendata GOOGLE_APPLICATION_CREDENTIALS TAP_BIGQUERY_CREDENTIALS_PATH
tap-bing-ads singer-io OAUTH_BING_ADS_CLIENT_ID TAP_BING_ADS_OAUTH_CLIENT_ID
OAUTH_BING_ADS_CLIENT_SECRET TAP_BING_ADS_OAUTH_CLIENT_SECRET
OAUTH_BING_ADS_DEVELOPER_TOKEN TAP_BING_ADS_DEVELOPER_TOKEN
tap-csv meltano TAP_CSV_FILES_DEFINITION TAP_CSV_CSV_FILES_DEFINITION
meltanolabs TAP_CSV_FILES_DEFINITION TAP_CSV_CSV_FILES_DEFINITION
tap-gitlab meltano GITLAB_API_GROUPS TAP_GITLAB_GROUPS
GITLAB_API_PROJECTS TAP_GITLAB_PROJECTS
GITLAB_API_START_DATE TAP_GITLAB_START_DATE
GITLAB_API_TOKEN TAP_GITLAB_PRIVATE_TOKEN
GITLAB_API_ULTIMATE_LICENSE TAP_GITLAB_ULTIMATE_LICENSE
meltanolabs GITLAB_API_GROUPS TAP_GITLAB_GROUPS
GITLAB_API_PROJECTS TAP_GITLAB_PROJECTS
GITLAB_API_START_DATE TAP_GITLAB_START_DATE
GITLAB_API_TOKEN TAP_GITLAB_PRIVATE_TOKEN
GITLAB_API_ULTIMATE_LICENSE TAP_GITLAB_ULTIMATE_LICENSE
tap-google-analytics meltano GOOGLE_ANALYTICS_API_CLIENT_SECRETS TAP_GOOGLE_ANALYTICS_KEY_FILE_LOCATION
GOOGLE_ANALYTICS_API_END_DATE TAP_GOOGLE_ANALYTICS_END_DATE
GOOGLE_ANALYTICS_API_OAUTH_ACCESS_TOKEN TAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_ACCESS_TOKEN
GOOGLE_ANALYTICS_API_OAUTH_CLIENT_ID TAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_CLIENT_ID
GOOGLE_ANALYTICS_API_OAUTH_CLIENT_SECRET TAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_CLIENT_SECRET
GOOGLE_ANALYTICS_API_OAUTH_REFRESH_TOKEN TAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_REFRESH_TOKEN
GOOGLE_ANALYTICS_API_REPORTS TAP_GOOGLE_ANALYTICS_REPORTS
GOOGLE_ANALYTICS_API_START_DATE TAP_GOOGLE_ANALYTICS_START_DATE
GOOGLE_ANALYTICS_API_VIEW_ID TAP_GOOGLE_ANALYTICS_VIEW_ID
tap-pendo singer-io TAP_PENDO_INTEGRATION_KEY TAP_PENDO_X_PENDO_INTEGRATION_KEY
tap-stripe meltano STRIPE_ACCOUNT_ID TAP_STRIPE_ACCOUNT_ID
STRIPE_API_KEY TAP_STRIPE_CLIENT_SECRET
prratek STRIPE_API_KEY TAP_STRIPE_API_KEY
singer-io STRIPE_ACCOUNT_ID TAP_STRIPE_ACCOUNT_ID
STRIPE_API_KEY TAP_STRIPE_CLIENT_SECRET
target-bigquery adswerve GOOGLE_APPLICATION_CREDENTIALS TARGET_BIGQUERY_CREDENTIALS_PATH
target-postgres datamill-co PG_ADDRESS TARGET_POSTGRES_POSTGRES_HOST
PG_DATABASE TARGET_POSTGRES_POSTGRES_DATABASE
PG_PASSWORD TARGET_POSTGRES_POSTGRES_PASSWORD
PG_PORT TARGET_POSTGRES_POSTGRES_PORT
PG_SCHEMA TARGET_POSTGRES_POSTGRES_SCHEMA
PG_USERNAME TARGET_POSTGRES_POSTGRES_USERNAME
TARGET_POSTGRES_DATABASE TARGET_POSTGRES_POSTGRES_DATABASE
TARGET_POSTGRES_HOST TARGET_POSTGRES_POSTGRES_HOST
TARGET_POSTGRES_PASSWORD TARGET_POSTGRES_POSTGRES_PASSWORD
TARGET_POSTGRES_PORT TARGET_POSTGRES_POSTGRES_PORT
TARGET_POSTGRES_SCHEMA TARGET_POSTGRES_POSTGRES_SCHEMA
TARGET_POSTGRES_SSLCERT TARGET_POSTGRES_POSTGRES_SSLCERT
TARGET_POSTGRES_SSLCRL TARGET_POSTGRES_POSTGRES_SSLCRL
TARGET_POSTGRES_SSLKEY TARGET_POSTGRES_POSTGRES_SSLKEY
TARGET_POSTGRES_SSLMODE TARGET_POSTGRES_POSTGRES_SSLMODE
TARGET_POSTGRES_SSLROOTCERT TARGET_POSTGRES_POSTGRES_SSLROOTCERT
TARGET_POSTGRES_USERNAME TARGET_POSTGRES_POSTGRES_USERNAME
meltano PG_ADDRESS TARGET_POSTGRES_HOST
PG_DATABASE TARGET_POSTGRES_DBNAME
PG_PASSWORD TARGET_POSTGRES_PASSWORD
PG_PORT TARGET_POSTGRES_PORT
PG_SCHEMA TARGET_POSTGRES_SCHEMA
PG_URL TARGET_POSTGRES_URL
PG_USERNAME TARGET_POSTGRES_USER
POSTGRES_DBNAME TARGET_POSTGRES_DBNAME
POSTGRES_HOST TARGET_POSTGRES_HOST
POSTGRES_PASSWORD TARGET_POSTGRES_PASSWORD
POSTGRES_PORT TARGET_POSTGRES_PORT
POSTGRES_SCHEMA TARGET_POSTGRES_SCHEMA
POSTGRES_URL TARGET_POSTGRES_URL
POSTGRES_USER TARGET_POSTGRES_USER
transferwise PG_ADDRESS TARGET_POSTGRES_HOST
PG_DATABASE TARGET_POSTGRES_DBNAME
PG_PASSWORD TARGET_POSTGRES_PASSWORD
PG_PORT TARGET_POSTGRES_PORT
PG_SCHEMA TARGET_POSTGRES_DEFAULT_TARGET_SCHEMA
PG_USERNAME TARGET_POSTGRES_USER
target-redshift transferwise AWS_ACCESS_KEY_ID TARGET_REDSHIFT_AWS_ACCESS_KEY_ID
AWS_PROFILE TARGET_REDSHIFT_AWS_PROFILE
AWS_SECRET_ACCESS_KEY TARGET_REDSHIFT_AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN TARGET_REDSHIFT_AWS_SESSION_TOKEN
TARGET_REDSHIFT_SCHEMA TARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA
target-snowflake datamill-co SF_ACCOUNT TARGET_SNOWFLAKE_SNOWFLAKE_ACCOUNT
SF_DATABASE TARGET_SNOWFLAKE_SNOWFLAKE_DATABASE
SF_PASSWORD TARGET_SNOWFLAKE_SNOWFLAKE_PASSWORD
SF_ROLE TARGET_SNOWFLAKE_SNOWFLAKE_ROLE
SF_SCHEMA TARGET_SNOWFLAKE_SNOWFLAKE_SCHEMA
SF_USER TARGET_SNOWFLAKE_SNOWFLAKE_USERNAME
SF_WAREHOUSE TARGET_SNOWFLAKE_SNOWFLAKE_WAREHOUSE
TARGET_SNOWFLAKE_ACCOUNT TARGET_SNOWFLAKE_SNOWFLAKE_ACCOUNT
TARGET_SNOWFLAKE_DATABASE TARGET_SNOWFLAKE_SNOWFLAKE_DATABASE
TARGET_SNOWFLAKE_PASSWORD TARGET_SNOWFLAKE_SNOWFLAKE_PASSWORD
TARGET_SNOWFLAKE_ROLE TARGET_SNOWFLAKE_SNOWFLAKE_ROLE
TARGET_SNOWFLAKE_SCHEMA TARGET_SNOWFLAKE_SNOWFLAKE_SCHEMA
TARGET_SNOWFLAKE_USERNAME TARGET_SNOWFLAKE_SNOWFLAKE_USERNAME
TARGET_SNOWFLAKE_USERNAME TARGET_SNOWFLAKE_SNOWFLAKE_USERNAME
TARGET_SNOWFLAKE_WAREHOUSE TARGET_SNOWFLAKE_SNOWFLAKE_WAREHOUSE
meltano SF_ACCOUNT TARGET_SNOWFLAKE_ACCOUNT
SF_DATABASE TARGET_SNOWFLAKE_DATABASE
SF_PASSWORD TARGET_SNOWFLAKE_PASSWORD
SF_ROLE TARGET_SNOWFLAKE_ROLE
SF_SCHEMA TARGET_SNOWFLAKE_SCHEMA
SF_USER TARGET_SNOWFLAKE_USERNAME
SF_WAREHOUSE TARGET_SNOWFLAKE_WAREHOUSE
SNOWFLAKE_ACCOUNT TARGET_SNOWFLAKE_ACCOUNT
SNOWFLAKE_DATABASE TARGET_SNOWFLAKE_DATABASE
SNOWFLAKE_PASSWORD TARGET_SNOWFLAKE_PASSWORD
SNOWFLAKE_ROLE TARGET_SNOWFLAKE_ROLE
SNOWFLAKE_SCHEMA TARGET_SNOWFLAKE_SCHEMA
SNOWFLAKE_USERNAME TARGET_SNOWFLAKE_USERNAME
SNOWFLAKE_WAREHOUSE TARGET_SNOWFLAKE_WAREHOUSE
transferwise SF_ACCOUNT TARGET_SNOWFLAKE_ACCOUNT
SF_DATABASE TARGET_SNOWFLAKE_DBNAME
SF_PASSWORD TARGET_SNOWFLAKE_PASSWORD
SF_ROLE TARGET_SNOWFLAKE_SNOWFLAKE_ROLE
SF_SCHEMA TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA
SF_USER TARGET_SNOWFLAKE_USER
SF_WAREHOUSE TARGET_SNOWFLAKE_WAREHOUSE
TARGET_SNOWFLAKE_DATABASE TARGET_SNOWFLAKE_DBNAME
TARGET_SNOWFLAKE_SCHEMA TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA
TARGET_SNOWFLAKE_USERNAME TARGET_SNOWFLAKE_USER
target-sqlite meltano SQLITE_DATABASE TARGET_SQLITE_DATABASE
meltanolabs SQLITE_DATABASE TARGET_SQLITE_DATABASE

CLI and API Changes #

Use --state-id instead of --job_id #

In 2.0, many references to “Job ID” in our code and docs were changed to the more accurate name of “State ID”.

If you are explicitly providing a --job_id option to any scripted or otherwise automated CLI workflows, these commands should be updated to use the --state-id option instead.

Schedule list format changes #

If you have custom orchestrator integrations based on the meltano schedule list command, you will need to make adjustments to handle a new output format.

With the addition of support for scheduled jobs in 2.0, the schema output of meltano schedule list --format=json has changed. It now includes a top level field schedules and two nested array fields job and elt which hold and describe their respective schedules.

ex:

{
  "schedules": {
    "job": [
      {
        "name": "daily-doit",
        "interval": "@daily",
        "cron_interval": "0 0 * * *",
        "env": {},
        "job": {
          "name": "simple-demo",
          "tasks": [
            "tap-gitlab hide-gitlab-secrets target-jsonl",
            "tap-gitlab target-csv"
          ]
        }
      }
    ],
    "elt": [
      {legacy elt schedule entry remains unchanged}, ...
    ]
  }
}