Onboarding

Meltano Cloud is currently in Beta.

While in Beta, functionality is not guaranteed and subject to change.
If you're interested in using Meltano Cloud please join our waitlist.

Prereqs

Prereq #1: Sign up for the waitlist

While Meltano Cloud is in Beta, you must sign up via the waitlist in order to gain access.

Waitlist members are added to Meltano Cloud on a rolling basis. You'll receive an email once you've been added.

Prereq #2: Provide access to your repo

Click the following link to install the Meltano Cloud GitHub App to your organization. When asked "Where do you want to install this app?" you must select either your GitHub organization or personal account depending on which contains the repository you wish to provide access. When asked which repositories to provide the app access to, select your GitHub project repo.

If you do not yet have a Meltano project in GitHub, you can follow the steps in our Getting Started guide.

Prereq #3: Define your schedules

Meltano Cloud uses the schedules defined in your meltano.yml project file to run workloads.

If your project does not yet have schedules defined - for instance, if you are running workloads via an external orchestrator - you'll want to create new schedules for use by Meltano Cloud.

Schedules should be specified in UTC if providing a cron expression.

If you don't yet have an existing schedule, you can get started quickly by copy-pasting this snippet to the bottom of your meltano.yml file:

schedules:
- name: daily-refresh
  job: daily-refresh-job
  interval: '@daily'  # Can be @daily, @hourly, etc., or a cron-based interval
jobs:
- name: daily-refresh-job
  tasks:
  # Update this section to include any EL jobs or other
  # commands that you'd like to run:
  - tap-gitlab target-snowflake
  - dbt-snowflake:run
  - dbt-snowflake:test

Onboarding Steps

Step 1: Install (or update) the Cloud CLI

To install from scratch using pipx:

pipx install 'git+https://github.com/meltano/meltano.git@cloud'

To upgrade to the latest version:

pipx reinstall meltano

The above commands will install a meltano-cloud CLI command into your workstation.

Step 2: Login to Meltano Cloud

Login to Meltano Cloud by running the meltano-cloud login command in your local terminal. You'll have to sign in with the same GitHub user you provided when you signed up via the waitlist.

The login command will open a browser window which you can use to access your account. In the Beta, your identification and authorization will be driven by your GitHub login identity. No Meltano-specific passwords or usernames are needed, and Meltano Cloud does not have access to your personal GitHub credentials.

While Meltano Cloud is in Beta, it may take up to one business day for your user to be fully provisioned and added to your Meltano Cloud organization. You'll be notified via email once your newly created user has been added to your organization, at which time you can logout via meltano-cloud logout and then log back in to have full CLI access to your organization's Meltano Cloud resources.

Installing the Meltano Cloud GitHub App to your organization is a separate process from granting Meltano Cloud access to use your GitHub profile for login purposes. Both the GitHub App installation for your organization and the OAuth grant flow for your profile must be performed in order to have full access to Meltano Cloud functionality.

Step 3: Create a Project

After logging in you can explore the interface with a few different commands. The full list of CLI commands is in the Cloud Docs.

To create a project, use the meltano-cloud project create command:

meltano-cloud project create --name example-project --repo-url https://github.com/meltano/squared.git --root-path "data/"
⠋ Creating project - this may take several minutes...
Project 'example-project' created successfully.

Step 4: Set default project and validate access and functionality

To see Meltano Cloud projects for your organizations, run:

meltano-cloud project list

Running this command will verify that your project is connected and you're properly authenticated with Meltano Cloud.

You should select a project to use as default for all commands. You can do this by running:

meltano-cloud project use --name <project name>

Step 5: Create deployments

In order for pipelines to run, they must have a deployment to run in.

To deploy a named Meltano Environment to Meltano Cloud, run the following command:

meltano-cloud deployment create --name <deployment name> --environment <Meltano Environment name> --git-rev <the git revision to use for this deployment>

For example, if you wanted to deploy the prod Meltano Environment as defined in your meltano.yml in the main branch of your git repo and you wanted the Meltano Cloud deployment to be named production, you would run:

meltano-cloud deployment create --name production --environment prod --git-rev main

If your deployment is failing you can try running `meltano compile` to confirm that your configuration files are valid. Also double check that you have schedules configured, otherwise the deployment will throw an error.

To confirm that your deployment was created, you can view all of your Meltano Cloud deployments by running:

meltano-cloud deployment list

Step 6: Initialize secrets

Secrets allow you to pass environment variables to your workloads without needing to expose them within your meltano.yml. Setting a secret in Meltano Cloud is equivalent to using a .env file to store environment variables at runtime. Secrets are shared across all deployments in your project and can be referenced in your meltano.yml file just as you would reference environment variables locally.

Secrets are configured using the Cloud CLI.

meltano-cloud config env set --key TAP_GITHUB_AUTH_TOKEN --value 'my_super_secret_auth_token'
> Secret value: ****

meltano-cloud config env list
> TAP_GITLAB_FOOBAR
> TAP_GITLAB_PASSWORD

Step 7: Run your workloads

You can invoke a schedule on-demand with the meltano-cloud run command:

meltano-cloud run --deployment=staging SCHEDULE_NAME

Within 1-2 minutes, the running workload will appear in history:

meltano-cloud history

To view logs for any completed or still-running job, you can use:

meltano-cloud logs print --execution-id=ASDF1234...

Step 8: Enable schedules

Your Meltano Cloud project's schedules are disabled by default. Use the meltano-cloud schedule enable <SCHEDULE NAME> command to enable schedules.