Cloud Command Line
Meltano Cloud is currently in Beta.
While in Beta, functionality is not guaranteed and subject to change.
If you're interested in using Meltano Cloud please join our waitlist.
config
The config
command provides an interface for managing project configuration and secrets.
The config
command supports the setting of environment variables via the env
command group.
env
Values set via this interface will be injected as Environment Variables into tasks run within the associated project.
Once set
, values cannot be viewed.
If you are unsure of the current value of an env var, use the set
command to set a known value.
The list
subcommand provides an interface to see existing set env var keys:
meltano-cloud config env list --limit 5
TAP_GITHUB_AUTH_TOKEN
TAP_GITHUB_USER_AGENT
The set
subcommand provides an interface to set new, or override existing, env var values.
meltano-cloud config env set --key TAP_GITHUB_AUTH_TOKEN --value 'my_super_secret_auth_token'
The delete
subcommand provides an interface to delete env vars:
meltano-cloud config env delete TAP_GITHUB_AUTH_TOKEN
Reserved Variables
See the reserved variables docs for more details on variables that are reserved for use by Meltano Cloud.
notification
The notification
subcommand provides a way for managing your Meltano Cloud notifications.
list
List all notifications configured for your current default project or organization.
$ meltano-cloud config notification list
╭──────────────────────────┬─────────┬──────────┬─────────────────────────────────────────╮
│ Recipient │ Type │ Status │ Filters │
├──────────────────────────┼─────────┼──────────┼─────────────────────────────────────────┤
│ dragon@meltano.com │ email │ active │ N/A │
│ https://melty │ webhook │ active │ [{'events': ['job_run'], 'status': []}] │
╰──────────────────────────┴─────────┴──────────┴─────────────────────────────────────────╯
Notification System Limitations
The notification system has a few limitations that should be taken into consideration:
- Unique Notifications: Currently, our system permits only a single notification per recipient within each scope, 'organization' or 'project'.
- Notification Overwrite: If you attempt to set another notification for the same recipient within the same scope, the new notification will overwrite the existing one. In other words, setting a second notification won't create a new notification, but rather modify the existing one.
set
Set a new notification or override an existing one.
Case-Sensitivity in Values
Take note that all the values in the commands and configuration settings are case-sensitive. This means job_created, Job_Created, and JOB_CREATED would be considered as different events. Please ensure to use the correct casing to avoid configuration issues or unexpected behaviors.
Interactive mode
meltano-cloud config notification set <type>
Non-interactive mode:
meltano-cloud config notification set <type> --recipient <target_recipient> --filter <filter>
options
--recipient
(Required): The recipient to send the notification to.
--filter
: An optional filter you want to check against, you can supply multiple filters which will all be joined as an OR expression.
Values within the filter are validated as an AND Expression.
Job Failures
To subscribe only to job failures you would run
meltano-cloud config notification set webhook --recipient http://test+melty --filter '{"events": ["job_run"], "status": ["failed"]}'
update
Update an existing notification.
meltano-cloud config notification update webhook --recipient <old_url> --new <new_recipient> --status <status>
options
--recipient
(Required): The recipient of the notification to update.
--new
: The new recipient value
--status
: The new status of the notification (either 'active' or 'inactive')
meltano-cloud config notification update --recipient http://test+melty --new http://test+better --status inactive
The above example updates the notification that was configured to send to http://test+melty
. It changes the recipient to http://test+better
and sets the status to inactive.
delete
Delete an existing notification
meltano-cloud config notification delete webhook --recipient <webhook_url>
options
--recipient
(Required): The recipient of the notification you are trying to delete
deployment
The deployment
command provides an interface for managing Meltano Cloud deployments for your projects.
Create a new deployment interactively:
meltano-cloud deployment create
Create a new deployment non-interactively:
meltano-cloud deployment create --name 'my-dev-deployment' --environment 'dev' --git-rev 'develop'
The above example creates a new deployment named my-dev-deployment
for the Meltano environment named dev
, using the develop
branch of the project's git repository. Note that the Meltano environment name must match what is in meltano.yml
.
If your deployment is failing you can try running `meltano compile` to confirm that your configuration files are valid. Also double check that you have schedules configured, otherwise the deployment will throw an error.
List deployments:
$ meltano-cloud deployment list
╭───────────┬──────────┬───────────────┬───────────────────┬────────────────────┬───────────────────────╮
│ Default │ Name │ Environment │ Tracked Git Rev │ Current Git Hash │ Last Deployed (UTC) │
├───────────┼──────────┼───────────────┼───────────────────┼────────────────────┼───────────────────────┤
│ │ prod │ prod │ main │ 0fa3aab │ 2023-05-30 16:52:44 │
│ X │ staging │ staging │ main │ a3268dd │ 2023-05-31 11:14:34 │
│ │ 1234-xyz │ dev │ feat/1234-xyz │ d105f18 │ 2023-06-01 03:57:31 │
╰───────────┴──────────┴───────────────┴───────────────────┴────────────────────┴───────────────────────╯
Delete a deployment:
meltano-cloud deployment delete --name 'my-dev-deployment'
Use a deployment as the default deployment for other commands:
meltano-cloud deployment use --name 'my-dev-deployment'
Selecting a default deployment can also be done interactively:
meltano-cloud deployment use
Currently Meltano Cloud doesn't automatically sync updates to schedules stored in your meltano.yml
file or changes to your tracked branch.
If you've made a change to your schedules configuration or tracked branch and would like them to be re-deployed to Meltano Cloud you can run the following:
meltano-cloud deployment update --name prod
You can then confirm the deployment is on the correct revision of your tracked branch by running the following:
meltano-cloud deployment list
docs
Opens the Meltano Cloud documentation in the system browser.
meltano-cloud docs
history
Display the history of executions for a project.
$ meltano-cloud history --limit 3
╭──────────────────────────────────┬─────────────────┬──────────────┬─────────────────────┬──────────┬────────────╮
│ Execution ID │ Schedule Name │ Deployment │ Executed At (UTC) │ Result │ Duration │
├──────────────────────────────────┼─────────────────┼──────────────┼─────────────────────┼──────────┼────────────┤
│ 15e1cbbde6b2424f86c04b237291d652 │ daily │ sandbox │ 2023-03-22 00:04:49 │ Success │ 00:05:08 │
│ ad2b34087e7c4332a1398321552f2a82 │ daily │ sandbox │ 2023-03-22 00:03:23 │ Failed │ 00:10:13 │
│ 695de7b041b445f5a46a7aac1d0879b9 │ daily │ sandbox │ 2023-03-21 15:44:55 │ Failed │ 00:08:09 │
╰──────────────────────────────────┴─────────────────┴──────────────┴─────────────────────┴──────────┴────────────╯
# Display the last 12 hours of executions
$ meltano-cloud history --lookback 12h
# Display the last week of executions
$ meltano-cloud history --lookback 1w
# Display the last hour and a half of executions
$ meltano-cloud history --lookback 1h30m
# Display the executions for the schedule named "daily"
$ meltano-cloud history --schedule daily
# Display the executions for schedules starting with "da"
$ meltano-cloud history --schedule-prefix da
# Display the executions for schedules containing the word "slack"
# Can also use `--filter`
$ meltano-cloud history --schedule-contains slack
# Display the executions for the deployment named "prod"
$ meltano-cloud history --deployment prod
# Display the executions for deployments starting with "pr"
$ meltano-cloud history --deployment-prefix pr
# Display the executions for deployments containing the string "pr"
$ meltano-cloud history --deployment-contains pr
# Display the executions for executions that failed.
# Options are success, failed, and running.
$ meltano-cloud history --result failed
# Display the execution history in json format
# Options are terminal, markdown, and json
$ meltano-cloud history --format json
login
Logging into Meltano Cloud via the CLI stores a token locally which is used by the CLI to take actions that require authentication.
Logging in will open a browser tab where you may be asked to authenticate yourself using GitHub.
# Login to Meltano Cloud
meltano-cloud login
logout
Logging out of Meltano Cloud invalidates your login token, and deletes the local copy that was saved when meltano-cloud login
was run.
# Logout from Meltano Cloud
meltano-cloud logout
logs
# Print logs for an execution
meltano-cloud logs print --execution-id <execution_id>
project
The project
command provides an interface for Meltano Cloud projects.
The list
subcommand shows all of the projects you have access to, and can use with other commands:
meltano-cloud project list
╭───────────┬───────────────────────────────┬──────────────────────────────────────────────────────────╮
│ Default │ Name │ Git Repository │
├───────────┼───────────────────────────────┼──────────────────────────────────────────────────────────┤
│ │ Meltano Squared │ https://github.com/meltano/squared.git │
│ │ MDS-in-a-box │ https://github.com/aaronsteers/meltano-demo-in-a-box.git │
╰───────────┴───────────────────────────────┴──────────────────────────────────────────────────────────╯
The create
subcommand creates a new Meltano Cloud project.
In order to create a project, you'll first need to grant access to the Meltano Cloud GitHub app as described in our onboarding documentation.
meltano-cloud project create --name example-project --repo-url https://github.com/meltano/squared.git --root-path "data/"
⠋ Creating project - this may take several minutes...
Project 'example-project' created successfully.
When you run the login
command, if you only have a single project, it will be set as the default project to use for future commands. Otherwise, you will need to run meltano-cloud project use
to specify which Meltano Cloud project the other meltano-cloud
commands should operate on.
When meltano-cloud project use
is not provided any argument, it will list the available projects, and have you select one interactively using the arrow keys. To select a project as the default non-interactively, use the --name
argument:
meltano-cloud project use --name 'Meltano Squared'
Set 'Meltano Squared' as the default Meltano Cloud project for future commands
meltano-cloud project list
╭───────────┬───────────────────────────────┬──────────────────────────────────────────────────────────╮
│ Default │ Name │ Git Repository │
├───────────┼───────────────────────────────┼──────────────────────────────────────────────────────────┤
│ X │ Meltano Squared │ https://github.com/meltano/squared.git │
│ │ MDS-in-a-box │ https://github.com/aaronsteers/meltano-demo-in-a-box.git │
╰───────────┴───────────────────────────────┴──────────────────────────────────────────────────────────╯
When specifying a project to use as the default for future command, its name must be exactly as shown when running meltano-cloud project list
. If there are spaces or special characters in the name, then it must be quoted.
run
Run a schedule immediately specifying the schedule name and deployment.
meltano-cloud run daily --deployment sandbox
Running a Meltano project in Meltano Cloud.
The running workload will appear in the history
within 1-2 minutes.
job
# Stop a job
meltano-cloud job stop --execution-id 15e1cbbde6b2424f86c04b237291d652
At the moment, executions stopped manually will be marked as Failed in the output of meltano-cloud history
. See #7697 for more details.
schedule
Prior to enabling or disabling a schedule, the project that schedule belongs to must be deployed.
Schedules are disabled when initially deployed, and must be enabled using the enable
command.
Currently, updating a schedule requires a redeployment. In the future it will be possible to update a schedule without a redeployment.
# Enable a schedule
meltano-cloud schedule enable --deployment <deployment name> --schedule <schedule name>
# Disable a schedule
meltano-cloud schedule disable --deployment <deployment name> --schedule <schedule name>
Schedules can be listed using the list
command:
meltano-cloud schedule list
╭──────────────┬────────────┬──────────────────────┬──────────────┬───────────╮
│ Deployment │ Schedule │ Interval │ Runs / Day │ Enabled │
├──────────────┼────────────┼──────────────────────┼──────────────┼───────────┤
│ staging │ schedule_1 │ 0 6 * * 1,3,5 │ < 1.0 │ False │
│ staging │ schedule_2 │ 0 */6 * * * │ 4.0 │ True │
│ prod │ schedule_3 │ 0 6 * * * │ 1.0 │ True │
│ prod │ schedule_4 │ 15,45 */2 * * 1,3,5 │ 10.3 │ False │
╰──────────────┴────────────┴──────────────────────┴──────────────┴───────────╯
meltano-cloud schedule list --deployment prod
╭──────────────┬────────────┬──────────────────────┬──────────────┬───────────╮
│ Deployment │ Schedule │ Interval │ Runs / Day │ Enabled │
├──────────────┼────────────┼──────────────────────┼──────────────┼───────────┤
│ prod │ schedule_3 │ 0 6 * * * │ 1.0 │ True │
│ prod │ schedule_4 │ 15,45 */2 * * 1,3,5 │ 10.3 │ False │
╰──────────────┴────────────┴──────────────────────┴──────────────┴───────────╯
Individual schedules can be more thoroughly described using the describe
command:
meltano-cloud schedule describe --deployment staging --schedule schedule_4 --num-upcoming 5
Deployment name: prod
Schedule name: schedule_4
Interval: 15,45 */2 * * 1,3,5
Enabled: True
Approximate starting date and time (UTC) of next 5 schedule runs:
2023-03-24 20:45
2023-03-24 22:15
2023-03-24 22:45
2023-03-27 00:15
2023-03-27 00:45
The --only-upcoming
option can be used to have the command only output the upcoming scheduled run start dates and times:
meltano-cloud schedule describe --deployment staging --schedule schedule_4 --num-upcoming 5 --only-upcoming
2023-03-24 20:45
2023-03-24 22:15
2023-03-24 22:45
2023-03-27 00:15
2023-03-27 00:45
If a schedule is disabled, it will never have any upcoming scheduled runs.
state
The state
command provides a subset of functionality from the Meltano Core state command to manage job state for Cloud jobs.
The list
subcommand prints State IDs for which state is being stored in Meltano Cloud.
meltano-cloud state list
dev:tap-github-to-target-postgres
prod:tap-github-to-target-postgres
dev:tap-hubspot-to-target-snowflake
The get
subcommand prints Singer state for a provided State ID.
meltano-cloud state get --state-id dev:tap-github-to-target-postgres
{"completed": {"singer_state": "bookmarks": [{"stream-1": "bookmark-1"}, {"stream-2": "bookmark-2"}]}}
The set
subcommand creates or replaces Singer state for a provided State ID with the contents of a provided .json
file.
meltano-cloud state set --state-id dev:tap-github-to-target-postgres --file-path local_state.json
Successfully set state for dev:tap-github-to-target-postgres
The delete
subcommand clears Singer state for the provided State ID.
meltano-cloud state delete --state-id dev:tap-github-to-target-postgres
Successfully deleted state for dev:tap-github-to-target-postgres
Note that State IDs are of the form {Meltano environment}:{extractor name}-to-{loader name}
.