Python local-packages, à la npm node_modules

Kamal Mustafa - Nov 26 '19 - - Dev Community

Updates

Follow-up post where I made this into simple program called pipm.
**************** ## ***********

Firstly, this thing doesn't exists. Unlike npm or php composer, packaging tools in python by default will install to the global packages location first.

In npm or composer, if you don't specify npm install -g, then by default the package will be installed to folder node_modules in the project root. This quite intuitive, local first, then global.

In python's pip, if you don't want to install to the global packages directory, you can restrict it --user flag. For example:-

python3 -mpip install --user requests
Enter fullscreen mode Exit fullscreen mode

On linux, above command usually will install requests package to user's $HOME/.local/lib/python3.x/site-packages directory. You can see that this only "local" to the user, not project unlike npm node_modules above.

In python, the solution is to use virtualenv. To use virtualenv however, there are at least 2 schools of taught. The first, and also the majority is to put the virtualenv external to your project.

So you might create a directory in your $HOME/.virtualenvs to store all your virtual environment. Creating new virtualenv may look like this:-

python3 -mvenv $HOME/.virtualenvs/myproject
Enter fullscreen mode Exit fullscreen mode

Then you will 'activate' the environment in order to make it active:-

source $HOME/.virtualenvs/myproject/bin/activate
Enter fullscreen mode Exit fullscreen mode

From now on, when you run python it will actually invoke $HOME/.virtualenvs/myproject/bin/python. And any packages you install with pip, for example:-

python -mpip install requests
Enter fullscreen mode Exit fullscreen mode

will install the requests library to $HOME/.virtualenvs/myproject/lib/python3.x/site-packages/requests.

But doing all this manually found to be tedious, so there's tools like virtualenvwrapper, or modern tools like pipenv or poetry that automatically create virtualenv for you.

However I don't really like this approach. Forgetting to 'activate' the virtualenv or worst, activating wrong virtualenv for your project is one of the most painful debugging experience.

So I am in the second camp, who create the virtualenv inside our project dir. For example, if I have a project in $HOME/myproject, I'll do:-

cd $HOME/myproject
python -mvenv venv
Enter fullscreen mode Exit fullscreen mode

So above command will create the virtualenv in the project directory - $HOME/myproject/venv. From now on, whenever I need to invoke python, I'll run ./venv/bin/python from my project root. To install packages, I'll run:-

./venv/bin/python -mpip install requests
Enter fullscreen mode Exit fullscreen mode

That will install requests to ./venv/lib/python3.x/site-packages. Explicit is better than implicit.

If using poetry, you can achieve similar effect by having in your project's poetry.toml:-

[virtualenvs]
in-project = true
Enter fullscreen mode Exit fullscreen mode

Ok, back to the original topic. What if we don't want to use virtualenv? Fortunately, pip has a flag called --target where if specified it will install the package to the target's location. For example:-

python -mpip install -t local-packages requests
Enter fullscreen mode Exit fullscreen mode

(Using name local-packages as a play to site-packages)

Above command will instead install requests to the directory local-packages, which you can create in your project root, similar to node_modules.

Recent version of pip will also created the package's executable script (if they have one) in local-packages/bin directory. For example if we install package httpie, which provide executable script named http in local-packages/bin/http.

Unfortunately, if we try to run the executable script it will fail:-

./local-packages/bin/http 
Traceback (most recent call last):
  File "./local-packages/bin/http", line 6, in <module>
    from httpie.__main__ import main
ModuleNotFoundError: No module named 'httpie'
Enter fullscreen mode Exit fullscreen mode

This is not surprising as ./local-packages dir (which has our httpie packages) is not on python sys.path which is a list of paths where python look for modules to import. We can check it through the interactive console:-

python
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/kamal/.local/lib/python3.7/site-packages', '/usr/lib/python3.7/site-packages']
Enter fullscreen mode Exit fullscreen mode

To add additional path, we can specify PYTHONPATH environment variable:-

PYTHONPATH=$PWD/local-packages ./local-packages/bin/http https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Mon, 25 Nov 2019 22:36:19 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}
Enter fullscreen mode Exit fullscreen mode

So it works now. But there's one gotcha. If somehow we have httpie already installed in global or user site-packages, that one probably will get used instead of the one we have in our local-packages. This is bad and can cause another headache when debugging. And this is the reason we use virtualenv. But we don't want to use virtualenv, right?

Fortunately, python has another flag. The -S flag will disable processing site-packages. From the docs:-

Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. Also disable these manipulations if site is explicitly imported later (call site.main() if you want them to be triggered).

But we run into another problem if we invoke python with the -S flag:-

PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 18, in <module>
    main()
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 10, in main
    from .core import main
  File "/home/kamal/python/testpip/local-packages/httpie/core.py", line 23, in <module>
    from httpie.client import get_response
  File "/home/kamal/python/testpip/local-packages/httpie/client.py", line 8, in <module>
    from httpie import sessions
  File "/home/kamal/python/testpip/local-packages/httpie/sessions.py", line 11, in <module>
    from httpie.plugins import plugin_manager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/__init__.py", line 10, in <module>
    from httpie.plugins.manager import PluginManager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/manager.py", line 2, in <module>
    from pkg_resources import iter_entry_points
ModuleNotFoundError: No module named 'pkg_resources'
Enter fullscreen mode Exit fullscreen mode

Let's try copying the pkg_resources module into our local-packages dir:-

cp -a /usr/lib/python3.7/site-packages/pkg_resources local-packages/
Enter fullscreen mode Exit fullscreen mode

No luck. We will find out that we need these modules to be available:-

  • pkg_resources
  • six
  • appdirs
  • packaging
  • pyparsing

And it turned out that all these modules actually part of setuptools, another beast in python packaging. So in short, in order to make our local-packages works, we need to install setuptools as well:-

python -mpip install -t local-packages setuptools
python -mpip install -t local-packages httpie
Enter fullscreen mode Exit fullscreen mode

And now invoking python with -S flag should work:-

PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Tue, 26 Nov 2019 00:15:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}
Enter fullscreen mode Exit fullscreen mode

And that's it. We finally manage to invoke python, free from global site-packages and using our local-packages dir to place all dependencies for our project.

End notes: In practice, we're using buildout to achieve the same objectives.

Image
Why Injury Lawyer Is Fast Becoming The Trendiest Thing Of 2023
Image
Difficult Conversations

webdev, beginners, leadership, career
Image
Mieszkanie marzeń: Jak stworzyć przestrzeń pełną harmonii?
Image
Trik Bikin Situs Judi Online Supaya Sukses
Image
Buy Negative Google Reviews

webdev, javascript, beginners, programming
Image
7 Simple Tricks To Rocking Your Wall Mount For Electric Fireplace
Image
15 Tips Your Boss Wants You To Know About Wall Mounted Electric Fireplace You'd Known About Wall Mounted Electric Fireplace
Image
Gold Versus Paper
Image
10 Quick Tips For Car Ghost Immobiliser
Image
Single Oven Under Counter
Image
Who Is Responsible For An Asbestos Lawsuit Settlements Budget? 12 Top Ways To Spend Your Money
Image
Single Ovens With Grill
Image
Apakah ada Judi Online Tanpa Deposit Awalan?
Image
The Expert Guide To Mobility Scooters Near Me For Sale
Image
Nine Ways Create Better Gold With The Assistance Of Your Canine
Image
From Startup to Cloud Giant: The Evolution of Salesforce and Its Impact on CRM

salesforceevolution, salesforceorigin
Image
CV Builder

codepen
Image
Trik Cepat Kaya Dari Permainan Judi Online
Image
Day 28: Model Compression Techniques for Large Language Models (LLMs)

llm, 75daysofllm, nlp
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .