What are Python Virtual Environments
Python Virtual Environments, also known as a venv, are a way to allow a Python application access to specific versions of the libraries it needs to run properly. In the Singer ecosystem you may need to install multiple Taps and Targets, which all have different dependencies.
If any of those dependecies are were in conflict with each other you would have a difficult development experience without Virtual Envrionments.
Python has a great technical write up at Virtual Environments and Packages.
We're going to focus on the Meltano use case for Virtual Environments which is use for a two main purposes:
- Installing Meltano
- Installing Plugins (Taps, Targets, Transformers, etc)
Why should I care about Virtual Environments?
Ideally, you don't have to to worry about Virtual Environments while using Meltano, we recommend using pipx to install Meltano which manages the creation of venvs for you. See our Installation Guide for more details.
However, if you ever need to customize or build your own production pipeline (or do anything else) you may need to understand how to install Meltano in an isolated way so that you don't have a conflict with dependecies within your own Operating System or other Python applications running on the same machine.
How do I use Virtual Environments?
We strongly suggest you create a directory where you want your virtual environments to be saved (e.g.
This can be any directory in your environment, but we recommend saving it in your Meltano project to make it easier to keep
Then create a new virtual environment inside that directory:
python -m venv .venv/meltano/
python command is calling the
venv package and it is creating a virtual environment in
And that's it! You've created a virtual environment.
Feel free to explore the directory and compare it to your global python directory!
You can navigate to the directory by running
Activating Your Virtual Environment
Activate the virtual environment, and upgrade pip using:
pip install --upgrade pip
Note that pip needs to be upgraded every time you make a new venv. Doing this helps avoid hard to troubleshoot issues with dependencies later.
If the virtual environment was activated successfully, you'll see a
(meltano) indicator added to
Once a virtual environment is activated, it stays active until the current shell is closed. In a new
shell, you must re-activate the virtual environment before interacting with the
that will be installed in the next step.
You can deactivate a virtual environment by typing
deactivate in your shell.
Install Meltano into a Virtual Environment
Now that you have your virtual environment set up and running, run the following command to install the Meltano package:
pip install meltano
As a reminder, we do generally recommend using `pipx` for installing python packages. Ensure you have `pipx` installed by reviewing the Install pipx instructions.
How does meltano use Virtual Environments internally?
Whenever you run
meltano install, Meltano creates a
.meltano/ directory in your project.
This directory has a number of sub-directories (subject to change at any point, as this is an
.meltano/ you may see an
extractors/ directory that was created (if you
have an extractor).
In that directory you'll see the name of a your tap and inside of that
directory is a virtual environment!
Each plugin is an application, so each of the steps we ran above to install Meltano is ran for you for each plugin when you run
Meltano manages all of your plugins virtual environments for you!