If you’ve added a custom plugin (or variant) to your project that could be discoverable and supported out of the box for new users, please contribute its description to Meltano Hub to save the next user the hassle of setting up the custom plugin. GitHub makes it easy to contribute changes without requiring you to leave your browser.
The format and further requirements are laid out in more detail in the Meltano Hub plugin definition syntax document.
When the maintainer of the default variant of a discoverable plugin becomes unresponsive to issues and contributions filed by the community, that plugin is considered up for adoption, which means that we are looking for a different variant of the plugin with a more engaged maintainer to become the new default.
This new variant can either be a fork of the original default variant, or an alternative implementation for the same source or destination, as long as it is actively maintained.
As a plugin’s primary maintainer, you do not have to spend a lot of time improving the plugin yourself. In fact, attracting more users and getting the community involved is likely to recude your personal maintenance burden, since you’ll receive contributions with bug fixes and new features that you will only be expected to review, not build yourself.
When you need to make changes to
discovery.yml, these changes are not automatically detected inside of the
meltano repo during development. While there are a few ways to solve this problem, it is recommended to create a symbolic link in order ensure that changes made inside of the
meltano repo appear inside the Meltano project you initialized and are testing on.
discovery.ymlin the repo
Open your Meltano project in your terminal
Create a symbolic link by running the following command:
ln -s $YOUR_DISCOVERY_YML_PATH
Now, when you run the
ls -l command, you should see something like:
bencodezen staff 72 Nov 19 09:19 discovery.yml -> /Users/bencodezen/Projects/meltano/src/meltano/core/bundle/discovery.yml
Now, you can see your changes in
discovery.yml live in your project during development! 🎉
Whenever new functionality is introduced that changes the schema of
discovery.yml (the exact properties it supports and their types), the
src/meltano/core/bundle/discovery.yml and the
VERSION constant in
src/meltano/core/plugin_discovery_service.py must be incremented, so that older instances of Meltano don’t attempt to download and parse a
discovery.yml its parser is not compatible with.
discovery.yml that only use existing properties do not constitute schema changes and do not require
version to be incremented.
Watch “How taps are built” for an explanation of how Singer taps (which form the basis for Meltano extractors) work, and what goes into building new ones or verifying and modifying existing ones for various types of APIs.
Then watch “How transforms are built” for an explanation of how DBT transforms work, and what goes into building new ones for new data sources.
We should be good citizen about these, and use the default workflow to contribute. Most of these are on GitHub so:
We qualify taps with the capabilities it supports:
--propertiesformat for the catalog
--catalogformat for the catalog
You should look at the tap’s documentation to see which one is supported.
Try to run the tap with the
--discover switch, which should output a catalog on STDOUT.
target-postgres, then validate that:
This might be a configuration issue with the catalog file that is sent to the tap. Take a look at the tap’s documentation and look for custom metadata on the catalog.
When you need to expose data to the user through Meltano UI, this often will require updating the transforms and models. At a high level:
You can test local transforms in a project by adding them in a Meltano project’s
Every transform file is a SQL file that will determine how the table is created. Some caveats include:
analytics.gitlab_issues), the syntax uses
refto refer to tables
*seems to crash dbt. Instead, you should explicitly define every column. For example:
users.user_id as user_id, users.user_name as user_name, issues.month_closed as month_closed, issues.year_closed as year_closed,
Once you’ve created your transforms, you can run it with the following command:
# Replace your extractors / targets with the appropriate ones meltano elt tap-gitlab target-postgres --transform only
To create a file bundle plugin like https://github.com/meltano/files-dbt, follow these steps:
LICENSEfrom https://github.com/meltano/files-dbt and edit these files as appropriate.
bundledirectory with an empty
bundledirectory. All of these files will be copied over into the Meltano project directory when the file bundle is added to the project.
pip_urlas appropriate, and if applicable, set
namespaceof the plugin the file bundle is related to (e.g.
meltano upgradeis run, add an
[file path]: Trueentries for each file.
discovery.yml(and an appropriate
CHANGELOGitem, of course).