Dataplugins
Dataplugins simply define a source of data from a given repository. Meltano Cloud provides a number of pre-configured platform-wide dataplugins out-the-box, as well as the ability to create custom dataplugins through the API. From these, pipeline jobs can be run to inject data into a workspace.
Objects
Dataplugin
| Path | JSON Type | Format | Description |
|---|---|---|---|
id | string | Version 4 UUID | The dataplugin ID |
name | string | The dataplugin name | |
description | string | A description of the dataplugin | |
repositoryUrl | string | URL | The dataplugin repository URL |
settings | object[] | Array of Settings | The dataplugin settings |
{
"id" : "cbbdeccb-c82d-4cde-b378-30ff809b343c",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/pipelinewise-target-postgres.git@v0.2.1",
"repo" : "https://github.com/Matatika/pipelinewise-target-postgres",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c",
"type" : "DELETE"
}
}
}
Setting
| Path | JSON Type | Format | Description |
|---|---|---|---|
name | string | The setting name | |
value | string | The setting default value | |
label | string | The setting label | |
protected | bool | The setting protection status | |
kind | string | Setting Kind | The setting kind |
description | string | A description of the setting | |
placeholder | string | The setting placeholder text | |
envAliases | string[] | Environment variable aliases for the setting | |
documentation | string | URL | The setting documentation URL |
oauth | OAuth | The setting OAuth configuration | |
env | string |
OAuth
| Path | JSON Type | Format | Description |
|---|---|---|---|
provider | string | The OAuth provider |
Formats
Setting Kind
string
| Value | Description |
|---|---|
STRING | String setting |
INTEGER | Integer setting |
PASSWORD | Password setting |
HIDDEN | Hidden setting |
BOOLEAN | Boolean setting |
DATE_ISO8601 | ISO 8601 date setting |
EMAIL | Email setting |
OAUTH | OAuth setting |
FILE | File setting |
ARRAY | Array setting |
Requests
View all supported dataplugins
GET /api/dataplugins
Returns all dataplugins supported by Meltano Cloud.
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/dataplugins' -i -X GET
import requests
url = "https://app.matatika.com/api/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "1149bda6-c93f-4db6-a22c-f95afd60d575",
"pluginType" : "FILE",
"name" : "analyze-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-sit.git",
"repo" : "https://github.com/Matatika/analyze-sit",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"analyze/datasets/tap-matatika-sit/user-ages.yml" : "true",
"analyze/datasets/tap-matatika-sit/user-genders.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : true,
"requires" : [ {
"id" : "931124c6-882f-4f0d-b0ca-6db09f1e1948",
"pluginType" : "EXTRACTOR",
"name" : "tap-matatika-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"label" : "Matatika SIT",
"description" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs",
"logoUrl" : "/assets/images/datasource/tap-matatika-sit.svg",
"hidden" : false,
"docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
"pipUrl" : "git+https://github.com/Matatika/tap-spreadsheets-anywhere@v0.4.1 google-api-core<2.28.0",
"repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
"executable" : "tap-spreadsheets-anywhere",
"capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "tables",
"aliases" : [ ],
"label" : "Tables",
"value" : "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]",
"options" : [ ],
"kind" : "ARRAY",
"description" : "A setting in Matatika SIT that allows users to view and manage tables of data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : true,
"requires" : [ ],
"fullDescription" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs\n\n## Settings\n\n\n### Tables\n\nA setting in Matatika SIT that allows users to view and manage tables of data."
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/1149bda6-c93f-4db6-a22c-f95afd60d575"
}
}
}, {
"id" : "2b48567d-5b9d-4018-9b6f-a9015963f53b",
"pluginType" : "LOADER",
"name" : "target-s3-avro",
"namespace" : "target_s3_avro",
"variant" : "faumel",
"label" : "S3 Avro",
"description" : "S3 Avro is a software tool for converting data between Avro and JSON formats in Amazon S3.\n\nS3 Avro is a software tool that allows users to easily convert data between Avro and JSON formats in Amazon S3. This tool is particularly useful for those who work with large amounts of data and need to quickly and efficiently convert between these two formats. With S3 Avro, users can easily upload Avro files to S3, convert them to JSON, and then download the converted files back to their local machine. This tool is designed to be user-friendly and intuitive, making it accessible to users of all skill levels.",
"logoUrl" : "/assets/logos/loaders/s3-avro.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3-avro/",
"pipUrl" : "git+https://github.com/faumel/target-s3-avro.git",
"repo" : "https://github.com/faumel/target-s3-avro",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "verify",
"aliases" : [ ],
"label" : "Verify",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Boolean value indicating whether to verify SSL certificates for HTTPS requests.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "Aws Session Token",
"options" : [ ],
"kind" : "STRING",
"description" : "Temporary session token for AWS authentication.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "api_version",
"aliases" : [ ],
"label" : "Api Version",
"options" : [ ],
"kind" : "STRING",
"description" : "Version of the S3 Avro API to use.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "endpoint_url",
"aliases" : [ ],
"label" : "Endpoint Url",
"options" : [ ],
"kind" : "STRING",
"description" : "URL for the S3 Avro API endpoint.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "Aws Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "Secret access key for AWS authentication.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "Aws Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "Access key ID for AWS authentication.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "flatten_delimiter",
"aliases" : [ ],
"label" : "Flatten Delimiter",
"options" : [ ],
"kind" : "STRING",
"description" : "Delimiter to use when flattening nested Avro records.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "region_name",
"aliases" : [ ],
"label" : "Region Name",
"options" : [ ],
"kind" : "STRING",
"description" : "Name of the AWS region where the S3 bucket is located.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "tmp_dir",
"aliases" : [ ],
"label" : "Tmp Dir",
"options" : [ ],
"kind" : "STRING",
"description" : "Directory to use for temporary files during Avro serialization.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "use_ssl",
"aliases" : [ ],
"label" : "Use SSL",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Boolean value indicating whether to use SSL for HTTPS requests.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema_bucket_key",
"aliases" : [ ],
"label" : "Target Schema Bucket Key",
"options" : [ ],
"kind" : "STRING",
"description" : "Key for the Avro schema file in the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "config",
"aliases" : [ ],
"label" : "Config",
"options" : [ ],
"kind" : "STRING",
"description" : "Additional configuration options for the S3 Avro API connection.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_bucket_key",
"aliases" : [ ],
"label" : "Target Bucket Key",
"options" : [ ],
"kind" : "STRING",
"description" : "Key for the target object in the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 Avro is a software tool for converting data between Avro and JSON formats in Amazon S3.\n\nS3 Avro is a software tool that allows users to easily convert data between Avro and JSON formats in Amazon S3. This tool is particularly useful for those who work with large amounts of data and need to quickly and efficiently convert between these two formats. With S3 Avro, users can easily upload Avro files to S3, convert them to JSON, and then download the converted files back to their local machine. This tool is designed to be user-friendly and intuitive, making it accessible to users of all skill levels.\n\n## Settings\n\n\n### Verify\n\nBoolean value indicating whether to verify SSL certificates for HTTPS requests.\n\n### Aws Session Token\n\nTemporary session token for AWS authentication.\n\n### Api Version\n\nVersion of the S3 Avro API to use.\n\n### Endpoint Url\n\nURL for the S3 Avro API endpoint.\n\n### Aws Secret Access Key\n\nSecret access key for AWS authentication.\n\n### Aws Access Key Id\n\nAccess key ID for AWS authentication.\n\n### Flatten Delimiter\n\nDelimiter to use when flattening nested Avro records.\n\n### Region Name\n\nName of the AWS region where the S3 bucket is located.\n\n### Tmp Dir\n\nDirectory to use for temporary files during Avro serialization.\n\n### Use SSL\n\nBoolean value indicating whether to use SSL for HTTPS requests.\n\n### Target Schema Bucket Key\n\nKey for the Avro schema file in the S3 bucket.\n\n### Config\n\nAdditional configuration options for the S3 Avro API connection.\n\n### Target Bucket Key\n\nKey for the target object in the S3 bucket.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/2b48567d-5b9d-4018-9b6f-a9015963f53b"
}
}
}, {
"id" : "4f3acdb4-898b-4ddf-a70f-1141f7b73129",
"pluginType" : "TRANSFORM",
"name" : "dbt-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-solarvista@v0.4.0",
"repo" : "https://github.com/Matatika/dbt-tap-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "81ca6a43-b7bf-4e3d-b01f-7c9fff39b962",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "secret_git_credential",
"aliases" : [ ],
"options" : [ ],
"kind" : "STRING",
"description" : "A convenient and simple way to supply a git credential for packages resolve by dbt:deps",
"hidden" : false,
"sensitive" : true,
"env" : "DBT_ENV_SECRET_GIT_CREDENTIAL",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts\n\n### secret_git_credential\n\nA convenient and simple way to supply a git credential for packages resolve by dbt:deps"
}, {
"id" : "33444aa0-a5e9-4edb-927a-d0c15707baa0",
"pluginType" : "EXTRACTOR",
"name" : "tap-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"label" : "Solarvista Live",
"description" : "Solarvista Live is a software platform for field service management.\n\nSolarvista Live is a cloud-based software platform designed to help businesses manage their field service operations more efficiently. It provides a range of tools and features to help businesses schedule and dispatch technicians, track work orders, manage inventory, and more. With Solarvista Live, businesses can streamline their field service operations, reduce costs, and improve customer satisfaction. The platform is highly customizable and can be tailored to meet the specific needs of each business. It is also designed to be easy to use, with a user-friendly interface that makes it simple for technicians and other field service personnel to access the information they need to do their jobs effectively. Overall, Solarvista Live is a powerful tool for businesses looking to optimize their field service operations and improve their bottom line.\n### Prerequisites\n- Datasources: The datasources required to connect to Solarvista Live are specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Account: The account information required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.\n- Client ID: The client ID required to connect to Solarvista Live is specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Code: The code required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.",
"logoUrl" : "/assets/images/datasource/tap-solarvista.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-solarvista/",
"pipUrl" : "git+https://github.com/Matatika/tap-solarvista@v0.3.1",
"repo" : "https://github.com/Matatika/tap-solarvista",
"capabilities" : [ "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "datasources",
"aliases" : [ ],
"label" : "Datasources",
"options" : [ ],
"kind" : "STRING",
"description" : "The data sources to connect to in Solarvista Live.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "account",
"aliases" : [ ],
"label" : "Account",
"options" : [ ],
"kind" : "STRING",
"description" : "The account name to use for authentication.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "clientId",
"aliases" : [ ],
"label" : "Client ID",
"options" : [ ],
"kind" : "STRING",
"description" : "The client ID to use for authentication.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "code",
"aliases" : [ ],
"label" : "Code",
"options" : [ ],
"kind" : "STRING",
"description" : "The code to use for authentication.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The date to start retrieving data from.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "force_start_date",
"aliases" : [ ],
"label" : "Force Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "A flag indicating whether to force the start date even if data already exists for that date.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Solarvista Live is a software platform for field service management.\n\nSolarvista Live is a cloud-based software platform designed to help businesses manage their field service operations more efficiently. It provides a range of tools and features to help businesses schedule and dispatch technicians, track work orders, manage inventory, and more. With Solarvista Live, businesses can streamline their field service operations, reduce costs, and improve customer satisfaction. The platform is highly customizable and can be tailored to meet the specific needs of each business. It is also designed to be easy to use, with a user-friendly interface that makes it simple for technicians and other field service personnel to access the information they need to do their jobs effectively. Overall, Solarvista Live is a powerful tool for businesses looking to optimize their field service operations and improve their bottom line.\n### Prerequisites\n- Datasources: The datasources required to connect to Solarvista Live are specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Account: The account information required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.\n- Client ID: The client ID required to connect to Solarvista Live is specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Code: The code required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.\n\n## Settings\n\n\n### Datasources\n\nThe data sources to connect to in Solarvista Live.\n\n### Account\n\nThe account name to use for authentication.\n\n### Client ID\n\nThe client ID to use for authentication.\n\n### Code\n\nThe code to use for authentication.\n\n### Start Date\n\nThe date to start retrieving data from.\n\n### Force Start Date\n\nA flag indicating whether to force the start date even if data already exists for that date."
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/4f3acdb4-898b-4ddf-a70f-1141f7b73129"
}
}
}, {
"id" : "14518e68-ecda-48c9-9c93-155453d89ef2",
"pluginType" : "FILE",
"name" : "analyze-auth0",
"namespace" : "tap_auth0",
"variant" : "matatika",
"label" : "Auth0 Insights",
"description" : "Instant insights on users, logins and quotas from Auth0.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-auth0@v0.5.0",
"repo" : "https://github.com/Matatika/analyze-auth0",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "6c5a07d0-8580-4bf3-a56e-fb87f7c24c09",
"pluginType" : "EXTRACTOR",
"name" : "tap-auth0",
"namespace" : "tap_auth0",
"variant" : "matatika",
"label" : "Auth0",
"description" : "Auth0 is an identity and access management platform.\n\nAuth0 is a cloud-based platform that provides a comprehensive set of tools and services for managing user authentication and authorization in web and mobile applications. It allows developers to easily add authentication and authorization capabilities to their applications, without having to build and maintain their own identity management system. Auth0 supports a wide range of authentication methods, including social login, multi-factor authentication, and passwordless authentication. It also provides features such as user management, role-based access control, and integration with third-party identity providers. With Auth0, developers can focus on building their applications, while leaving the complex task of identity management to the experts.\n### Prerequisites\nTo obtain the Client ID, Client Secret, and Domain for connecting to Auth0, you need to follow these steps:\n\n1. Log in to your Auth0 account.\n2. From the dashboard, click on the \"Applications\" tab.\n3. Click on the \"Create Application\" button.\n4. Choose the type of application you want to create (Single Page Application, Regular Web Application, etc.).\n5. Give your application a name and click on the \"Create\" button.\n6. Once your application is created, you will be redirected to the \"Settings\" tab.\n7. Here, you will find the Client ID and Client Secret.\n8. To obtain the Domain, go to the \"Settings\" tab of your Auth0 account and copy the value of the \"Domain\" field.\n\nNote: The exact steps may vary slightly depending on the version of Auth0 you are using.",
"logoUrl" : "/assets/images/datasource/tap-auth0.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-auth0/",
"pipUrl" : "git+https://github.com/Matatika/tap-auth0@v0.6.0",
"repo" : "https://github.com/Matatika/tap-auth0",
"capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "client_id",
"aliases" : [ ],
"label" : "Client ID",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier for the client application that is registered with Auth0.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "client_secret",
"aliases" : [ ],
"label" : "Client Secret",
"options" : [ ],
"kind" : "STRING",
"description" : "A secret string that is used to authenticate the client application with Auth0.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "domain",
"aliases" : [ ],
"label" : "Domain",
"options" : [ ],
"kind" : "STRING",
"description" : "The Auth0 domain associated with the tenant.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "job_poll_interval_ms",
"aliases" : [ ],
"label" : "Job poll interval ms",
"value" : "2000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The interval in milliseconds at which to poll for the status of a long-running job.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "job_poll_max_count",
"aliases" : [ ],
"label" : "Job poll max count",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of times to poll for the status of a long-running job.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Auth0 is an identity and access management platform.\n\nAuth0 is a cloud-based platform that provides a comprehensive set of tools and services for managing user authentication and authorization in web and mobile applications. It allows developers to easily add authentication and authorization capabilities to their applications, without having to build and maintain their own identity management system. Auth0 supports a wide range of authentication methods, including social login, multi-factor authentication, and passwordless authentication. It also provides features such as user management, role-based access control, and integration with third-party identity providers. With Auth0, developers can focus on building their applications, while leaving the complex task of identity management to the experts.\n### Prerequisites\nTo obtain the Client ID, Client Secret, and Domain for connecting to Auth0, you need to follow these steps:\n\n1. Log in to your Auth0 account.\n2. From the dashboard, click on the \"Applications\" tab.\n3. Click on the \"Create Application\" button.\n4. Choose the type of application you want to create (Single Page Application, Regular Web Application, etc.).\n5. Give your application a name and click on the \"Create\" button.\n6. Once your application is created, you will be redirected to the \"Settings\" tab.\n7. Here, you will find the Client ID and Client Secret.\n8. To obtain the Domain, go to the \"Settings\" tab of your Auth0 account and copy the value of the \"Domain\" field.\n\nNote: The exact steps may vary slightly depending on the version of Auth0 you are using.\n\n## Settings\n\n\n### Client ID\n\nA unique identifier for the client application that is registered with Auth0.\n\n### Client Secret\n\nA secret string that is used to authenticate the client application with Auth0.\n\n### Domain\n\nThe Auth0 domain associated with the tenant.\n\n### Job poll interval ms\n\nThe interval in milliseconds at which to poll for the status of a long-running job.\n\n### Job poll max count\n\nThe maximum number of times to poll for the status of a long-running job."
} ],
"fullDescription" : "Instant insights on users, logins and quotas from Auth0.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/14518e68-ecda-48c9-9c93-155453d89ef2"
}
}
}, {
"id" : "e5e317b4-ddfe-4617-8228-966feeb124ed",
"pluginType" : "EXTRACTOR",
"name" : "tap-autopilot",
"namespace" : "tap_autopilot",
"variant" : "singer-io",
"label" : "Autopilot",
"description" : "Autopilot is a marketing automation software. \n\nAutopilot is a cloud-based marketing automation software that helps businesses automate their marketing tasks and workflows, such as lead generation, email marketing, and customer journey mapping, to improve customer engagement and drive revenue growth. It offers a visual canvas for creating personalized customer journeys, as well as integrations with popular CRM and marketing tools. Autopilot also provides analytics and reporting features to track campaign performance and optimize marketing strategies.",
"logoUrl" : "/assets/logos/extractors/autopilot.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-autopilot/",
"pipUrl" : "tap-autopilot",
"repo" : "https://github.com/singer-io/tap-autopilot",
"capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Autopilot is a marketing automation software. \n\nAutopilot is a cloud-based marketing automation software that helps businesses automate their marketing tasks and workflows, such as lead generation, email marketing, and customer journey mapping, to improve customer engagement and drive revenue growth. It offers a visual canvas for creating personalized customer journeys, as well as integrations with popular CRM and marketing tools. Autopilot also provides analytics and reporting features to track campaign performance and optimize marketing strategies.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/e5e317b4-ddfe-4617-8228-966feeb124ed"
}
}
}, {
"id" : "dbf87b80-6eb6-483a-90bb-b7a8c094fb3a",
"pluginType" : "FILE",
"name" : "analyze-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"label" : "Solarvista Insights",
"description" : "Instant insights on revenue, projects, work items, and engineer performance from Solarvista Live.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-solarvista@v0.5.0",
"repo" : "https://github.com/Matatika/analyze-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "4f3acdb4-898b-4ddf-a70f-1141f7b73129",
"pluginType" : "TRANSFORM",
"name" : "dbt-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-solarvista@v0.4.0",
"repo" : "https://github.com/Matatika/dbt-tap-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : ""
} ],
"fullDescription" : "Instant insights on revenue, projects, work items, and engineer performance from Solarvista Live.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/dbf87b80-6eb6-483a-90bb-b7a8c094fb3a"
}
}
}, {
"id" : "ffd26c88-aa25-4e04-913c-8dd0b22762d1",
"pluginType" : "FILE",
"name" : "analyze-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"label" : "Trello Insights",
"description" : "Instant insights on members, cards, boards, and actions from Trello.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-trello@v0.5.0",
"repo" : "https://github.com/Matatika/analyze-trello",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "512c097b-df0e-4437-ba9a-3374557a30d9",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-trello@v0.5.0",
"repo" : "https://github.com/Matatika/dbt-tap-trello",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : ""
} ],
"fullDescription" : "Instant insights on members, cards, boards, and actions from Trello.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/ffd26c88-aa25-4e04-913c-8dd0b22762d1"
}
}
}, {
"id" : "c5c84dde-1880-494d-95c4-7c71f43528f5",
"pluginType" : "EXTRACTOR",
"name" : "tap-aftership",
"namespace" : "tap_aftership",
"variant" : "harrystech",
"label" : "AfterShip",
"description" : "AfterShip is a shipment tracking platform for online retailers and customers.\n\nAfterShip allows online retailers to track and manage their shipments across multiple carriers and provides customers with real-time updates on the status of their orders. The platform integrates with over 700 carriers worldwide and offers features such as branded tracking pages, delivery notifications, and analytics to help businesses improve their shipping performance. AfterShip also offers a mobile app for customers to track their packages on-the-go.",
"logoUrl" : "/assets/logos/extractors/aftership.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-aftership/",
"pipUrl" : "git+https://github.com/harrystech/tap-aftership.git",
"repo" : "https://github.com/harrystech/tap-aftership",
"capabilities" : [ "ABOUT", "STREAM_MAPS", "SCHEMA_FLATTENING", "CATALOG", "STATE", "DISCOVER" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "api_key",
"aliases" : [ ],
"label" : "Api Key",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier used to authenticate and authorize API requests.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The earliest date for which shipment tracking information should be retrieved.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The latest date for which shipment tracking information should be retrieved.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A list of stream maps that define the structure of the response data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Additional configuration settings for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "A boolean value indicating whether or not the response data should be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth to which the response data should be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "AfterShip is a shipment tracking platform for online retailers and customers.\n\nAfterShip allows online retailers to track and manage their shipments across multiple carriers and provides customers with real-time updates on the status of their orders. The platform integrates with over 700 carriers worldwide and offers features such as branded tracking pages, delivery notifications, and analytics to help businesses improve their shipping performance. AfterShip also offers a mobile app for customers to track their packages on-the-go.\n\n## Settings\n\n\n### Api Key\n\nA unique identifier used to authenticate and authorize API requests.\n\n### Start Date\n\nThe earliest date for which shipment tracking information should be retrieved.\n\n### End Date\n\nThe latest date for which shipment tracking information should be retrieved.\n\n### Stream Maps\n\nA list of stream maps that define the structure of the response data.\n\n### Stream Map Config\n\nAdditional configuration settings for the stream maps.\n\n### Flattening Enabled\n\nA boolean value indicating whether or not the response data should be flattened.\n\n### Flattening Max Depth\n\nThe maximum depth to which the response data should be flattened.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c5c84dde-1880-494d-95c4-7c71f43528f5"
}
}
}, {
"id" : "c0598af4-f633-4d21-8f56-80a60aea9140",
"pluginType" : "LOADER",
"name" : "target-s3-csv",
"namespace" : "pipelinewise_target_s3_csv",
"variant" : "transferwise",
"label" : "S3 CSV",
"description" : "S3 CSV is a tool for managing CSV files in Amazon S3.\n\nS3 CSV is a software tool that allows users to easily manage CSV files stored in Amazon S3. It provides features such as importing, exporting, and transforming CSV files, as well as querying and filtering data. S3 CSV also offers advanced functionality such as data validation, data cleansing, and data enrichment. With S3 CSV, users can streamline their CSV file management processes and improve the accuracy and quality of their data.",
"logoUrl" : "/assets/logos/loaders/pipelinewise-s3-csv.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3-csv/",
"pipUrl" : "git+https://github.com/transferwise/pipelinewise-target-s3-csv.git",
"repo" : "https://github.com/transferwise/pipelinewise-target-s3-csv",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "S3 Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "S3 Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "AWS Session token",
"options" : [ ],
"kind" : "STRING",
"description" : "The session token for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_endpoint_url",
"aliases" : [ ],
"label" : "AWS endpoint URL",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint URL for the AWS service.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_profile",
"aliases" : [ ],
"label" : "AWS profile",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the AWS profile to use.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_bucket",
"aliases" : [ ],
"label" : "S3 Bucket name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the S3 bucket to connect to.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_key_prefix",
"aliases" : [ ],
"label" : "S3 Key Prefix",
"options" : [ ],
"kind" : "STRING",
"description" : "The prefix to use when searching for files in the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "delimiter",
"aliases" : [ ],
"label" : "Delimiter",
"options" : [ ],
"kind" : "STRING",
"description" : "The delimiter used in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "quotechar",
"aliases" : [ ],
"label" : "Quote Char",
"options" : [ ],
"kind" : "STRING",
"description" : "The character used to quote fields in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the output.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "encryption_type",
"aliases" : [ ],
"label" : "S3 Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The encryption key to use for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "encryption_key",
"aliases" : [ ],
"label" : "Encryption Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The compression algorithm to use for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "compression",
"aliases" : [ ],
"label" : "Compression",
"options" : [ ],
"kind" : "STRING",
"description" : "The naming convention to use for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "naming_convention",
"aliases" : [ ],
"label" : "Naming Convention",
"options" : [ ],
"kind" : "STRING",
"description" : "(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports \"folders\" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the \"filename\". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "S3 Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 CSV is a tool for managing CSV files in Amazon S3.\n\nS3 CSV is a software tool that allows users to easily manage CSV files stored in Amazon S3. It provides features such as importing, exporting, and transforming CSV files, as well as querying and filtering data. S3 CSV also offers advanced functionality such as data validation, data cleansing, and data enrichment. With S3 CSV, users can streamline their CSV file management processes and improve the accuracy and quality of their data.\n\n## Settings\n\n\n### S3 Access Key Id\n\nThe access key ID for the AWS account.\n\n### S3 Secret Access Key\n\nThe secret access key for the AWS account.\n\n### AWS Session token\n\nThe session token for the AWS account.\n\n### AWS endpoint URL\n\nThe endpoint URL for the AWS service.\n\n### AWS profile\n\nThe name of the AWS profile to use.\n\n### S3 Bucket name\n\nThe name of the S3 bucket to connect to.\n\n### S3 Key Prefix\n\nThe prefix to use when searching for files in the S3 bucket.\n\n### Delimiter\n\nThe delimiter used in the CSV file.\n\n### Quote Char\n\nThe character used to quote fields in the CSV file.\n\n### Add Metadata Columns\n\nWhether or not to add metadata columns to the output.\n\n### S3 Access Key Id\n\nThe encryption key to use for the CSV file.\n\n### Encryption Key\n\nThe compression algorithm to use for the CSV file.\n\n### Compression\n\nThe naming convention to use for the CSV file.\n\n### Naming Convention\n\n(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports \"folders\" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the \"filename\". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv\n\n### S3 Access Key Id\n\n(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c0598af4-f633-4d21-8f56-80a60aea9140"
}
}
}, {
"id" : "24072fe8-2f1f-4a0c-be4a-97df8c5e5be7",
"pluginType" : "LOADER",
"name" : "target-s3-parquet",
"namespace" : "target_s3_parquet",
"variant" : "gupy-io",
"label" : "S3 Parquet",
"description" : "S3 Parquet is a file format for storing and processing large amounts of data in a distributed computing environment.\n\nS3 Parquet is a columnar storage format that allows for efficient compression and encoding of data, making it ideal for storing and processing large amounts of data in a distributed computing environment. It is designed to work seamlessly with Amazon S3 and other big data processing tools such as Apache Spark and Hadoop. S3 Parquet allows for faster data processing and analysis, as well as reduced storage costs, making it a popular choice for big data applications.",
"logoUrl" : "/assets/logos/loaders/s3-parquet.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3-parquet/",
"pipUrl" : "git+https://github.com/gupy-io/target-s3-parquet.git",
"repo" : "https://github.com/gupy-io/target-s3-parquet",
"capabilities" : [ "ABOUT", "STREAM_MAPS", "RECORD_FLATTENING" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "s3_path",
"aliases" : [ ],
"label" : "S3 Path",
"options" : [ ],
"kind" : "STRING",
"description" : "The path to the S3 bucket and object where the Parquet data is stored.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "AWS Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account that has permission to access the S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "AWS Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account that has permission to access the S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "athena_database",
"aliases" : [ ],
"label" : "Athena Database",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the Athena database where the Parquet data will be queried.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_record_metadata",
"aliases" : [ ],
"label" : "Add Record Metadata",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata to each record in the Parquet data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stringify_schema",
"aliases" : [ ],
"label" : "Stringify Schema",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert the schema of the Parquet data to a string format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A mapping of column names to stream names for the Parquet data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Configuration options for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flatten nested structures in the Parquet data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth to which nested structures will be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 Parquet is a file format for storing and processing large amounts of data in a distributed computing environment.\n\nS3 Parquet is a columnar storage format that allows for efficient compression and encoding of data, making it ideal for storing and processing large amounts of data in a distributed computing environment. It is designed to work seamlessly with Amazon S3 and other big data processing tools such as Apache Spark and Hadoop. S3 Parquet allows for faster data processing and analysis, as well as reduced storage costs, making it a popular choice for big data applications.\n\n## Settings\n\n\n### S3 Path\n\nThe path to the S3 bucket and object where the Parquet data is stored.\n\n### AWS Access Key Id\n\nThe access key ID for the AWS account that has permission to access the S3 bucket.\n\n### AWS Secret Access Key\n\nThe secret access key for the AWS account that has permission to access the S3 bucket.\n\n### Athena Database\n\nThe name of the Athena database where the Parquet data will be queried.\n\n### Add Record Metadata\n\nWhether or not to add metadata to each record in the Parquet data.\n\n### Stringify Schema\n\nWhether or not to convert the schema of the Parquet data to a string format.\n\n### Stream Maps\n\nA mapping of column names to stream names for the Parquet data.\n\n### Stream Map Config\n\nConfiguration options for the stream maps.\n\n### Flattening Enabled\n\nWhether or not to flatten nested structures in the Parquet data.\n\n### Flattening Max Depth\n\nThe maximum depth to which nested structures will be flattened.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/24072fe8-2f1f-4a0c-be4a-97df8c5e5be7"
}
}
}, {
"id" : "931124c6-882f-4f0d-b0ca-6db09f1e1948",
"pluginType" : "EXTRACTOR",
"name" : "tap-matatika-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"label" : "Matatika SIT",
"description" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs",
"logoUrl" : "/assets/images/datasource/tap-matatika-sit.svg",
"hidden" : false,
"docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
"pipUrl" : "git+https://github.com/Matatika/tap-spreadsheets-anywhere@v0.4.1 google-api-core<2.28.0",
"repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
"executable" : "tap-spreadsheets-anywhere",
"capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "tables",
"aliases" : [ ],
"label" : "Tables",
"value" : "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]",
"options" : [ ],
"kind" : "ARRAY",
"description" : "A setting in Matatika SIT that allows users to view and manage tables of data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : true,
"requires" : [ ],
"fullDescription" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs\n\n## Settings\n\n\n### Tables\n\nA setting in Matatika SIT that allows users to view and manage tables of data.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/931124c6-882f-4f0d-b0ca-6db09f1e1948"
}
}
}, {
"id" : "4fa46eaa-9d17-42c1-9f59-8998bf10a71e",
"pluginType" : "EXTRACTOR",
"name" : "tap-anaplan",
"namespace" : "tap_anaplan",
"variant" : "matthew-skinner",
"label" : "Anaplan",
"description" : "Anaplan is a cloud-based platform for enterprise planning and performance management.\n\nAnaplan provides a centralized platform for businesses to plan, forecast, and analyze their financial and operational data in real-time. It allows users to create and customize models for budgeting, forecasting, sales planning, workforce planning, and more. Anaplan's platform is designed to be flexible and scalable, allowing businesses to adapt to changing market conditions and make data-driven decisions. It also offers collaboration tools, data visualization, and reporting capabilities to help teams work together more efficiently and effectively.",
"logoUrl" : "/assets/logos/extractors/anaplan.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-anaplan/",
"pipUrl" : "git+https://github.com/matthew-skinner/tap-anaplan.git",
"repo" : "https://github.com/matthew-skinner/tap-anaplan",
"capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Anaplan is a cloud-based platform for enterprise planning and performance management.\n\nAnaplan provides a centralized platform for businesses to plan, forecast, and analyze their financial and operational data in real-time. It allows users to create and customize models for budgeting, forecasting, sales planning, workforce planning, and more. Anaplan's platform is designed to be flexible and scalable, allowing businesses to adapt to changing market conditions and make data-driven decisions. It also offers collaboration tools, data visualization, and reporting capabilities to help teams work together more efficiently and effectively.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/4fa46eaa-9d17-42c1-9f59-8998bf10a71e"
}
}
}, {
"id" : "cb74863b-07d2-4b9a-912f-c7f8172ffc36",
"pluginType" : "LOADER",
"name" : "target-s3csv",
"namespace" : "pipelinewise_target_s3_csv",
"variant" : "transferwise",
"label" : "S3 CSV",
"description" : "S3 CSV is a file format used for storing data in Amazon S3.\n\nAmazon S3 is a cloud-based storage service that allows users to store and retrieve data from anywhere on the web. S3 CSV is a file format used for storing data in S3 that is organized in rows and columns, similar to a spreadsheet. This format is commonly used for storing large amounts of data that can be easily accessed and analyzed using various tools and applications. S3 CSV files can be easily imported and exported to other applications, making it a popular choice for data storage and analysis in the cloud.\n### Prerequisites\nTo obtain the AWS Access Key Id and AWS Secret Access Key, you need to go to the AWS Management Console, navigate to the IAM service, and create an IAM user with programmatic access. During the user creation process, you will be provided with the Access Key Id and Secret Access Key.\n\nTo obtain the S3 Bucket name, you need to navigate to the S3 service in the AWS Management Console and select the bucket that contains the CSV file you want to connect to. The name of the bucket will be displayed in the bucket details page.",
"logoUrl" : "/assets/logos/extractors/s3-csv.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3csv/",
"pipUrl" : "git+https://github.com/transferwise/pipelinewise-target-s3-csv.git",
"repo" : "https://github.com/transferwise/pipelinewise-target-s3-csv",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "AWS Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "AWS Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "AWS Session token",
"options" : [ ],
"kind" : "STRING",
"description" : "The session token for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_endpoint_url",
"aliases" : [ ],
"label" : "AWS endpoint URL",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint URL for the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_profile",
"aliases" : [ ],
"label" : "AWS profile",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the AWS profile to use.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_bucket",
"aliases" : [ ],
"label" : "S3 Bucket name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the S3 bucket to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "s3_key_prefix",
"aliases" : [ ],
"label" : "S3 Key Prefix",
"options" : [ ],
"kind" : "STRING",
"description" : "The prefix for the S3 keys to read.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "delimiter",
"aliases" : [ ],
"label" : "delimiter",
"options" : [ ],
"kind" : "STRING",
"description" : "The delimiter used in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "quotechar",
"aliases" : [ ],
"label" : "Quote Char",
"options" : [ ],
"kind" : "STRING",
"description" : "The character used to quote fields in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to add metadata columns to the output.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "encryption_type",
"aliases" : [ ],
"label" : "Encryption Type",
"options" : [ ],
"kind" : "STRING",
"description" : "The type of encryption used for the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "compression",
"aliases" : [ ],
"label" : "Compression",
"options" : [ ],
"kind" : "STRING",
"description" : "The compression type used for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "naming_convention",
"aliases" : [ ],
"label" : "Naming Convention",
"options" : [ ],
"kind" : "STRING",
"description" : "The naming convention used for the output files.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 CSV is a file format used for storing data in Amazon S3.\n\nAmazon S3 is a cloud-based storage service that allows users to store and retrieve data from anywhere on the web. S3 CSV is a file format used for storing data in S3 that is organized in rows and columns, similar to a spreadsheet. This format is commonly used for storing large amounts of data that can be easily accessed and analyzed using various tools and applications. S3 CSV files can be easily imported and exported to other applications, making it a popular choice for data storage and analysis in the cloud.\n### Prerequisites\nTo obtain the AWS Access Key Id and AWS Secret Access Key, you need to go to the AWS Management Console, navigate to the IAM service, and create an IAM user with programmatic access. During the user creation process, you will be provided with the Access Key Id and Secret Access Key.\n\nTo obtain the S3 Bucket name, you need to navigate to the S3 service in the AWS Management Console and select the bucket that contains the CSV file you want to connect to. The name of the bucket will be displayed in the bucket details page.\n\n## Settings\n\n\n### AWS Access Key Id\n\nThe access key ID for the AWS account.\n\n### AWS Secret Access Key\n\nThe secret access key for the AWS account.\n\n### AWS Session token\n\nThe session token for the AWS account.\n\n### AWS endpoint URL\n\nThe endpoint URL for the S3 bucket.\n\n### AWS profile\n\nThe name of the AWS profile to use.\n\n### S3 Bucket name\n\nThe name of the S3 bucket to connect to.\n\n### S3 Key Prefix\n\nThe prefix for the S3 keys to read.\n\n### delimiter\n\nThe delimiter used in the CSV file.\n\n### Quote Char\n\nThe character used to quote fields in the CSV file.\n\n### Add Metadata Columns\n\nWhether to add metadata columns to the output.\n\n### Encryption Type\n\nThe type of encryption used for the S3 bucket.\n\n### Compression\n\nThe compression type used for the CSV file.\n\n### Naming Convention\n\nThe naming convention used for the output files.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/cb74863b-07d2-4b9a-912f-c7f8172ffc36"
}
}
}, {
"id" : "0879ca90-e5ba-49b9-8435-c68676133ac7",
"pluginType" : "FILE",
"name" : "analyze-meltano",
"namespace" : "tap_meltano",
"variant" : "matatika",
"label" : "Meltano Insights",
"description" : "Instant insights on jobs from Meltano.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-meltano@v0.4.0",
"repo" : "https://github.com/Matatika/analyze-meltano",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "8688dd6b-e9b9-48f9-b1ae-747ef53b071b",
"pluginType" : "TRANSFORM",
"name" : "dbt-meltano",
"namespace" : "tap_meltano",
"variant" : "matatika",
"description" : " Meltano is an open source project that manages data plugins and python virtual environments.\nMatatika extracts the data from Meltano and creates out of the box insights with this plugin.\n",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-meltano@v0.4.0",
"repo" : "https://github.com/Matatika/dbt-tap-meltano",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Meltano is an open source project that manages data plugins and python virtual environments.\nMatatika extracts the data from Meltano and creates out of the box insights with this plugin.\n"
} ],
"fullDescription" : "Instant insights on jobs from Meltano.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/0879ca90-e5ba-49b9-8435-c68676133ac7"
}
}
}, {
"id" : "9483915e-18cd-48a7-a804-ae0123da2931",
"pluginType" : "EXTRACTOR",
"name" : "tap-msaccess",
"namespace" : "tap_msaccess",
"variant" : "matatika",
"label" : "Microsoft Access",
"description" : "Database management system from Microsoft that combines the relational Access Database Engine with a graphical user interface and software-development tools",
"logoUrl" : "/assets/logos/extractors/msaccess.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-msaccess/",
"pipUrl" : "git+https://github.com/Matatika/tap-msaccess.git@v0.2.0",
"repo" : "https://github.com/Matatika/tap-msaccess",
"capabilities" : [ "ABOUT", "STREAM_MAPS", "BATCH", "SCHEMA_FLATTENING", "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "database_file",
"aliases" : [ ],
"label" : "Database file",
"options" : [ ],
"kind" : "STRING",
"description" : "Local path or URL to a Microsoft Access database `.mdb` or `.accdb` file",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Database management system from Microsoft that combines the relational Access Database Engine with a graphical user interface and software-development tools\n\n## Settings\n\n\n### Database file\n\nLocal path or URL to a Microsoft Access database `.mdb` or `.accdb` file",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/9483915e-18cd-48a7-a804-ae0123da2931"
}
}
}, {
"id" : "a9ba6541-32a3-47ab-bb96-8c4aef3c4ab4",
"pluginType" : "TRANSFORM",
"name" : "dbt-google-analytics",
"namespace" : "tap_google_analytics__legacy",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-google-analytics@v0.4.0",
"repo" : "https://github.com/Matatika/dbt-tap-google-analytics",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "81ca6a43-b7bf-4e3d-b01f-7c9fff39b962",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "secret_git_credential",
"aliases" : [ ],
"options" : [ ],
"kind" : "STRING",
"description" : "A convenient and simple way to supply a git credential for packages resolve by dbt:deps",
"hidden" : false,
"sensitive" : true,
"env" : "DBT_ENV_SECRET_GIT_CREDENTIAL",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts\n\n### secret_git_credential\n\nA convenient and simple way to supply a git credential for packages resolve by dbt:deps"
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/a9ba6541-32a3-47ab-bb96-8c4aef3c4ab4"
}
}
}, {
"id" : "bc91e7c0-6ade-43f3-987e-56083ce3f834",
"pluginType" : "EXTRACTOR",
"name" : "tap-anvil",
"namespace" : "tap_anvil",
"variant" : "svinstech",
"label" : "Anvil",
"description" : "Anvil is a web-based platform for building full-stack web apps with nothing but Python.\n\nAnvil allows users to build full-stack web applications using only Python code, without the need for front-end development skills or knowledge of HTML, CSS, or JavaScript. The platform provides a drag-and-drop interface for building user interfaces, as well as a built-in Python editor for writing server-side code. Anvil also includes a range of pre-built components and integrations, such as databases, authentication, and APIs, to help users build complex applications quickly and easily. With Anvil, developers can create web applications for a variety of use cases, from simple data entry forms to complex business applications.",
"logoUrl" : "/assets/logos/extractors/anvil.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-anvil/",
"pipUrl" : "git+https://github.com/svinstech/tap-anvil.git",
"repo" : "https://github.com/svinstech/tap-anvil",
"capabilities" : [ "ABOUT", "STREAM_MAPS", "SCHEMA_FLATTENING", "CATALOG", "STATE", "DISCOVER" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "api_key",
"aliases" : [ ],
"label" : "Api Key",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier used to authenticate and authorize API requests.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A mapping of input and output streams used to transform data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Configuration settings for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "A boolean value indicating whether or not to flatten nested data structures.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth of nested data structures to flatten.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Anvil is a web-based platform for building full-stack web apps with nothing but Python.\n\nAnvil allows users to build full-stack web applications using only Python code, without the need for front-end development skills or knowledge of HTML, CSS, or JavaScript. The platform provides a drag-and-drop interface for building user interfaces, as well as a built-in Python editor for writing server-side code. Anvil also includes a range of pre-built components and integrations, such as databases, authentication, and APIs, to help users build complex applications quickly and easily. With Anvil, developers can create web applications for a variety of use cases, from simple data entry forms to complex business applications.\n\n## Settings\n\n\n### Api Key\n\nA unique identifier used to authenticate and authorize API requests.\n\n### Stream Maps\n\nA mapping of input and output streams used to transform data.\n\n### Stream Map Config\n\nConfiguration settings for the stream maps.\n\n### Flattening Enabled\n\nA boolean value indicating whether or not to flatten nested data structures.\n\n### Flattening Max Depth\n\nThe maximum depth of nested data structures to flatten.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/bc91e7c0-6ade-43f3-987e-56083ce3f834"
}
}
}, {
"id" : "2331e1dd-8c46-4f76-b2d6-34ab1fa1e86b",
"pluginType" : "EXTRACTOR",
"name" : "tap-apachelog",
"namespace" : "tap_apachelog",
"variant" : "omelark",
"label" : "Apache Log Files",
"description" : "Apache Log Files is a tool for logging and analyzing web server activity.\n\nApache Log Files is a software tool that records all activity on a web server, including requests for pages, images, and other resources, as well as errors and other events. This data can then be analyzed to gain insights into how the server is being used, which pages are most popular, and where errors are occurring. The tool can also be used to monitor server performance and identify potential security issues. Apache Log Files is widely used by web developers, system administrators, and security professionals to optimize server performance and ensure the smooth operation of web applications.",
"logoUrl" : "/assets/logos/extractors/apachelog.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-apachelog/",
"pipUrl" : "git+https://github.com/omelark/tap-apachelog.git",
"repo" : "https://github.com/omelark/tap-apachelog",
"capabilities" : [ "ABOUT", "STREAM_MAPS", "SCHEMA_FLATTENING", "CATALOG", "STATE", "DISCOVER" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "files",
"aliases" : [ ],
"label" : "Files",
"options" : [ ],
"kind" : "ARRAY",
"description" : "The file path or paths to the Apache log files.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "apachelog_files_definition",
"aliases" : [ ],
"label" : "Apachelog Files Definition",
"options" : [ ],
"kind" : "STRING",
"description" : "The format of the Apache log files being read.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A mapping of fields in the log files to fields in the output stream.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Additional configuration options for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flatten nested JSON objects in the log files.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth to which nested JSON objects will be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Apache Log Files is a tool for logging and analyzing web server activity.\n\nApache Log Files is a software tool that records all activity on a web server, including requests for pages, images, and other resources, as well as errors and other events. This data can then be analyzed to gain insights into how the server is being used, which pages are most popular, and where errors are occurring. The tool can also be used to monitor server performance and identify potential security issues. Apache Log Files is widely used by web developers, system administrators, and security professionals to optimize server performance and ensure the smooth operation of web applications.\n\n## Settings\n\n\n### Files\n\nThe file path or paths to the Apache log files.\n\n### Apachelog Files Definition\n\nThe format of the Apache log files being read.\n\n### Stream Maps\n\nA mapping of fields in the log files to fields in the output stream.\n\n### Stream Map Config\n\nAdditional configuration options for the stream maps.\n\n### Flattening Enabled\n\nWhether or not to flatten nested JSON objects in the log files.\n\n### Flattening Max Depth\n\nThe maximum depth to which nested JSON objects will be flattened.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/2331e1dd-8c46-4f76-b2d6-34ab1fa1e86b"
}
}
}, {
"id" : "512c097b-df0e-4437-ba9a-3374557a30d9",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-trello@v0.5.0",
"repo" : "https://github.com/Matatika/dbt-tap-trello",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "81ca6a43-b7bf-4e3d-b01f-7c9fff39b962",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "secret_git_credential",
"aliases" : [ ],
"options" : [ ],
"kind" : "STRING",
"description" : "A convenient and simple way to supply a git credential for packages resolve by dbt:deps",
"hidden" : false,
"sensitive" : true,
"env" : "DBT_ENV_SECRET_GIT_CREDENTIAL",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts\n\n### secret_git_credential\n\nA convenient and simple way to supply a git credential for packages resolve by dbt:deps"
}, {
"id" : "a2ac4e7b-b259-43ba-bf95-19227c398f51",
"pluginType" : "EXTRACTOR",
"name" : "tap-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"label" : "Trello",
"description" : "Trello is a web-based project management tool.\n\nTrello is a web-based project management tool that allows users to create boards, lists, and cards to organize and prioritize tasks and projects. Users can add team members, due dates, attachments, and comments to cards, and move them between lists as they progress. Trello also offers integrations with other tools such as Google Drive, Slack, and Jira, making it a versatile and customizable solution for teams of all sizes and industries. With a simple and intuitive interface, Trello is a popular choice for individuals and teams looking to streamline their workflows and increase productivity.\n### Prerequisites\nTo obtain a Developer API Key and Access Token for Trello, follow these steps:\n\n1. Go to the Trello Developer API Key website: https://trello.com/app-key\n2. Log in to your Trello account if you haven't already.\n3. Click the \"Generate API Key\" button to generate your Developer API Key.\n4. Copy the Developer API Key to a safe location.\n5. Click the \"Token\" link next to the Developer API Key to generate your Access Token.\n6. Follow the prompts to authorize the token for your Trello account.\n7. Copy the Access Token to a safe location.\n\nNote: Keep your Developer API Key and Access Token secure, as they provide access to your Trello account.",
"logoUrl" : "/assets/images/datasource/tap-trello.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-trello/",
"pipUrl" : "git+https://github.com/Matatika/tap-trello@v0.8.0",
"repo" : "https://github.com/Matatika/tap-trello",
"capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "developer_api_key",
"aliases" : [ ],
"label" : "Developer API Key",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier used to authenticate and authorize API requests.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "A token that grants access to a user's Trello account and resources.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "STRING",
"description" : "The date from which to start retrieving data from Trello.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Trello is a web-based project management tool.\n\nTrello is a web-based project management tool that allows users to create boards, lists, and cards to organize and prioritize tasks and projects. Users can add team members, due dates, attachments, and comments to cards, and move them between lists as they progress. Trello also offers integrations with other tools such as Google Drive, Slack, and Jira, making it a versatile and customizable solution for teams of all sizes and industries. With a simple and intuitive interface, Trello is a popular choice for individuals and teams looking to streamline their workflows and increase productivity.\n### Prerequisites\nTo obtain a Developer API Key and Access Token for Trello, follow these steps:\n\n1. Go to the Trello Developer API Key website: https://trello.com/app-key\n2. Log in to your Trello account if you haven't already.\n3. Click the \"Generate API Key\" button to generate your Developer API Key.\n4. Copy the Developer API Key to a safe location.\n5. Click the \"Token\" link next to the Developer API Key to generate your Access Token.\n6. Follow the prompts to authorize the token for your Trello account.\n7. Copy the Access Token to a safe location.\n\nNote: Keep your Developer API Key and Access Token secure, as they provide access to your Trello account.\n\n## Settings\n\n\n### Developer API Key\n\nA unique identifier used to authenticate and authorize API requests.\n\n### Access Token\n\nA token that grants access to a user's Trello account and resources.\n\n### Start date\n\nThe date from which to start retrieving data from Trello."
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/512c097b-df0e-4437-ba9a-3374557a30d9"
}
}
}, {
"id" : "07736115-e671-4723-a2fa-abe3bd8fe9f4",
"pluginType" : "FILE",
"name" : "analyze-googleads",
"namespace" : "tap_googleads",
"variant" : "matatika",
"label" : "Google Ads Insights",
"description" : "Instant insights on campaigns, devices, costs, and clicks from Google Ads.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-googleads@v0.5.0",
"repo" : "https://github.com/Matatika/analyze-googleads",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "53c122a6-8924-49d9-a9cb-11a2c0b11ebb",
"pluginType" : "TRANSFORM",
"name" : "dbt-googleads",
"namespace" : "tap_googleads",
"variant" : "matatika",
"description" : " Google Ads is an online advertising platform that allows businesses to create and display ads to potential customers.\nMatatika extract the data from the Google Ads API and creates out of the box insights including:\n\n***Google Ads campaigns***\n- Campaign Name\n- Campaign Status\n- Total Cost\n- Average CPC per Day\n- Total Clicks\n- Total Impressions\n- Popularity\n\n***Google Ads click locations***\nAnswer questions such as *What are my top 10 locations by clicks across all my campaigns?*\n\n***Google Ads campaign clicks***\nAnswer questions such as *How many clicks have all my campaigns had over the last 30 days?*\n\n",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/dbt-tap-googleads@v0.6.0",
"repo" : "https://github.com/Matatika/dbt-tap-googleads",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : "{{ env_var('DBT_SOURCE_SCHEMA') }}"
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Google Ads is an online advertising platform that allows businesses to create and display ads to potential customers.\nMatatika extract the data from the Google Ads API and creates out of the box insights including:\n\n***Google Ads campaigns***\n- Campaign Name\n- Campaign Status\n- Total Cost\n- Average CPC per Day\n- Total Clicks\n- Total Impressions\n- Popularity\n\n***Google Ads click locations***\nAnswer questions such as *What are my top 10 locations by clicks across all my campaigns?*\n\n***Google Ads campaign clicks***\nAnswer questions such as *How many clicks have all my campaigns had over the last 30 days?*\n\n"
} ],
"fullDescription" : "Instant insights on campaigns, devices, costs, and clicks from Google Ads.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/07736115-e671-4723-a2fa-abe3bd8fe9f4"
}
}
} ]
},
"_links" : {
"first" : {
"href" : "https://app.matatika.com/api/dataplugins?page=0&size=20"
},
"self" : {
"href" : "https://app.matatika.com/api/dataplugins?page=0&size=20"
},
"next" : {
"href" : "https://app.matatika.com/api/dataplugins?page=1&size=20"
},
"last" : {
"href" : "https://app.matatika.com/api/dataplugins?page=27&size=20"
}
},
"page" : {
"size" : 20,
"totalElements" : 546,
"totalPages" : 28,
"number" : 0
}
}
View all workspace dataplugins
GET /api/workspaces/{workspace-id}/dataplugins
Returns all dataplugins available to the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins' -i -X GET
import requests
url = "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "cbbdeccb-c82d-4cde-b378-30ff809b343c",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/pipelinewise-target-postgres.git@v0.2.1",
"repo" : "https://github.com/Matatika/pipelinewise-target-postgres",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c",
"type" : "DELETE"
}
}
}, {
"id" : "0c32a431-c718-414a-9c7a-0c7054b405c2",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "secret_git_credential",
"aliases" : [ ],
"options" : [ ],
"kind" : "STRING",
"description" : "A convenient and simple way to supply a git credential for packages resolve by dbt:deps",
"hidden" : false,
"sensitive" : true,
"env" : "DBT_ENV_SECRET_GIT_CREDENTIAL",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/files-dbt@v1.3",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts\n\n### secret_git_credential\n\nA convenient and simple way to supply a git credential for packages resolve by dbt:deps",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/0c32a431-c718-414a-9c7a-0c7054b405c2"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/0c32a431-c718-414a-9c7a-0c7054b405c2",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/0c32a431-c718-414a-9c7a-0c7054b405c2",
"type" : "DELETE"
}
}
}, {
"id" : "65cfa27d-57cf-4824-8979-a35635aa29bc",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"options" : [ ],
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"options" : [ ],
"placeholder" : "example@domain.com",
"kind" : "EMAIL",
"description" : "The email login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "A dataplugin created during an SIT run\n\n## Settings\n\n\n### Username\n\nThe username login credential.\n\n### Email\n\nThe email login credential.\n\n### Start Date\n\nThe data to begin extracting data from, in ISO 8601 format.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "DELETE"
}
}
}, {
"id" : "e227b574-8a7d-4930-9fb7-e156bc92c36e",
"pluginType" : "EXTRACTOR",
"name" : "tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/e227b574-8a7d-4930-9fb7-e156bc92c36e"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/e227b574-8a7d-4930-9fb7-e156bc92c36e",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/e227b574-8a7d-4930-9fb7-e156bc92c36e",
"type" : "DELETE"
}
}
}, {
"id" : "a2b94c6f-dcdf-47aa-a2c3-f17f0ea46415",
"pluginType" : "LOADER",
"name" : "target-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/a2b94c6f-dcdf-47aa-a2c3-f17f0ea46415"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/a2b94c6f-dcdf-47aa-a2c3-f17f0ea46415",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/a2b94c6f-dcdf-47aa-a2c3-f17f0ea46415",
"type" : "DELETE"
}
}
}, {
"id" : "fe895f43-adaf-490c-8bca-d511ca72e53e",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/fe895f43-adaf-490c-8bca-d511ca72e53e"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/fe895f43-adaf-490c-8bca-d511ca72e53e",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/fe895f43-adaf-490c-8bca-d511ca72e53e",
"type" : "DELETE"
}
}
}, {
"id" : "31594b21-1280-4afa-af3a-1e3c79504589",
"pluginType" : "FILE",
"name" : "analyze-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/31594b21-1280-4afa-af3a-1e3c79504589"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/31594b21-1280-4afa-af3a-1e3c79504589",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/31594b21-1280-4afa-af3a-1e3c79504589",
"type" : "DELETE"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins?page=0&size=20"
}
},
"page" : {
"size" : 20,
"totalElements" : 7,
"totalPages" : 1,
"number" : 0
}
}
View a dataplugin
GET /api/dataplugins/{dataplugin-id}
Returns the dataplugin {dataplugin-id}.
Prerequisites
- Dataplugin
{dataplugin-id}must exist
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c' -i -X GET
import requests
url = "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin with HAL links.
{
"id" : "cbbdeccb-c82d-4cde-b378-30ff809b343c",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/pipelinewise-target-postgres.git@v0.2.1",
"repo" : "https://github.com/Matatika/pipelinewise-target-postgres",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/cbbdeccb-c82d-4cde-b378-30ff809b343c",
"type" : "DELETE"
}
}
}
Initialise a new dataplugin
POST /api/dataplugins
Initialises a new dataplugin.
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins' -i -X POST \
-H 'Content-Type: application/json'
import requests
url = "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin with HAL links.
{
"id" : "65cfa27d-57cf-4824-8979-a35635aa29bc",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
},
"create dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "PUT"
}
}
}
Create a dataplugin
PUT /api/dataplugins/{dataplugin-id}
Creates the dataplugin {dataplugin-id}.
Body
Dataplugin resource.
{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "example@domain.com",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc' -i -X PUT \
-H 'Content-Type: application/json' \
-d '{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "example@domain.com",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}'
import requests
url = "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
data = {
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "example@domain.com",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
201 Created
Dataplugin with HAL links.
{
"id" : "65cfa27d-57cf-4824-8979-a35635aa29bc",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"placeholder" : "example@domain.com",
"kind" : "EMAIL",
"description" : "The email login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "A dataplugin updated during an SIT run\n\n## Settings\n\n\n### Username\n\nThe username login credential.\n\n### Email\n\nThe email login credential.\n\n### Start Date\n\nThe data to begin extracting data from, in ISO 8601 format.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "DELETE"
}
}
}
Update a dataplugin
PUT /api/dataplugins/{dataplugin-id}
Updates the dataplugin {dataplugin-id}.
Prerequisites
- Dataplugin
{dataplugin-id}must exist
Body
Dataplugin resource.
{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "example@domain.com",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
| Path | JSON Type | Format | Description | Constraints |
|---|---|---|---|---|
description | string | A description of the dataplugin | ||
repositoryUrl | string | A URL to the dataplugin repository | ||
settings | object[] | Array of Settings | The dataplugin settings |
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc' -i -X PUT \
-H 'Content-Type: application/json' \
-d '{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "example@domain.com",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}'
import requests
url = "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
data = {
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "example@domain.com",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin with HAL links.
{
"id" : "65cfa27d-57cf-4824-8979-a35635aa29bc",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"placeholder" : "example@domain.com",
"kind" : "EMAIL",
"description" : "The email login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "A dataplugin updated during an SIT run\n\n## Settings\n\n\n### Username\n\nThe username login credential.\n\n### Email\n\nThe email login credential.\n\n### Start Date\n\nThe data to begin extracting data from, in ISO 8601 format.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/72957e0e-9c4b-4dd7-9e1c-307ea982367d/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc",
"type" : "DELETE"
}
}
}
Delete a dataplugin
DELETE /api/dataplugins/{dataplugin-id}
Deletes the dataplugin {dataplugin-id}.
Examples
- cURL
- Python (requests)
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc' -i -X DELETE
import requests
url = "https://app.matatika.com/api/dataplugins/65cfa27d-57cf-4824-8979-a35635aa29bc"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("DELETE", url, headers=headers)
print(response.text.encode('utf8'))
Response
204 No Content
No response body provided.