Troubleshooting

If you have a question about Meltano, are having trouble getting it to work, or have any kind of feedback, you can:

Check out the Meltano issue tracker to see if someone else has already reported the same issue or made the same request. Feel free to weigh in with extra information, or just to make sure the issue gets the attention it deserves.

Join the Meltano Slack workspace, which is frequented by the core team and thousands of community members and data experts. You can ask any questions you may have in here or just chat with fellow users.

Common Troubleshooting Issues #

Log Level Debug #

When you are trying to troubleshoot an issue the Meltano logs should be your first port of call.

If you have a failure using Meltano’s execution commands (invoke, elt, run, or test) or you’re experienced general unexpected behavior, you can learn more about what’s happening behind the scenes by setting Meltano’s cli.log_level setting to debug, using the MELTANO_CLI_LOG_LEVEL environment variable or the --log-level CLI option:

export MELTANO_CLI_LOG_LEVEL=debug

meltano --log-level=debug <command> ...

In debug mode, Meltano will log additional information about the environment and arguments used to invoke your components (Singer taps and targets, dbt, Airflow, etc.), including the paths to the generated config, catalog, state files, etc. for you to review.

Here is an example with meltano elt:

$ meltano --log-level=debug elt tap-gitlab target-jsonl --state-id=gitlab-to-jsonl
meltano            | INFO Running extract & load...
meltano            | INFO Found state from 2020-08-05 21:30:20.487312.
meltano            | DEBUG Invoking: ['demo-project/.meltano/extractors/tap-gitlab/venv/bin/tap-gitlab', '--config', 'demo-project/.meltano/run/tap-gitlab/tap.config.json', '--state', 'demo-project/.meltano/run/tap-gitlab/state.json']
meltano            | DEBUG Env: {'TAP_GITLAB_API_URL': 'https://gitlab.com', 'GITLAB_API_TOKEN': '', 'GITLAB_API_GROUPS': '', 'GITLAB_API_PROJECTS': 'meltano/meltano', 'GITLAB_API_ULTIMATE_LICENSE': 'False', 'GITLAB_API_START_DATE': '2021-03-01'}
meltano            | DEBUG Invoking: ['demo-project/.meltano/loaders/target-jsonl/venv/bin/target-jsonl', '--config', 'demo-project/.meltano/run/target-jsonl/target.config.json']
meltano            | DEBUG Env: {'MELTANO_EXTRACTOR_NAME': 'tap-gitlab', 'MELTANO_EXTRACTOR_NAMESPACE': 'tap_gitlab', 'MELTANO_EXTRACT_API_URL': 'https://gitlab.com', 'MELTANO_EXTRACT_PRIVATE_TOKEN': '', 'MELTANO_EXTRACT_GROUPS': '', 'MELTANO_EXTRACT_PROJECTS': 'meltano/meltano', 'MELTANO_EXTRACT_ULTIMATE_LICENSE': 'False', 'MELTANO_EXTRACT_START_DATE': '2021-03-01', 'TAP_GITLAB_API_URL': 'https://gitlab.com', 'GITLAB_API_TOKEN': '', 'GITLAB_API_GROUPS': '', 'GITLAB_API_PROJECTS': 'meltano/meltano', 'GITLAB_API_ULTIMATE_LICENSE': 'False', 'GITLAB_API_START_DATE': '2021-03-01', 'TARGET_JSONL_DESTINATION_PATH': 'output', 'TARGET_JSONL_DO_TIMESTAMP_FILE': 'False'}

Dump Files Generated by Running Meltano Commands to STDOUT #

The --dump flag can be passed to the meltano invoke and meltano elt commands to dump the content of a pipeline-specific generated file to STDOUT instead of actually running the pipeline. Note that adding support for meltano run is tracked in this GitHub issue.

This can aid in debugging extractor catalog generation, incremental replication state lookup, and pipeline environment variables.

Supported values are:

  • catalog: Dump the extractor catalog file that would be passed to the tap’s executable using the --catalog option.
  • state: Dump the extractor state file that would be passed to the tap’s executable using the --state option.
  • extractor-config: Dump the extractor config file that would be passed to the tap’s executable using the --config option.
  • loader-config: Dump the loader config file that would be passed to the target’s executable using the --config option.

Like any standard output, the dumped content can be redirected to a file using >, e.g. meltano elt ... --dump=state > state.json.

Examples #

meltano elt tap-gitlab target-postgres --transform=run --state-id=gitlab-to-postgres

meltano elt tap-gitlab target-postgres --state-id=gitlab-to-postgres --full-refresh

meltano elt tap-gitlab target-postgres --catalog extract/tap-gitlab.catalog.json
meltano elt tap-gitlab target-postgres --state extract/tap-gitlab.state.json

meltano elt tap-gitlab target-postgres --select commits
meltano elt tap-gitlab target-postgres --exclude project_members

meltano elt tap-gitlab target-postgres --state-id=gitlab-to-postgres --dump=state > extract/tap-gitlab.state.json