Installing python deps with pip -r requirements.txt fails, but installing one by one succeeds. Why?

I have the following requirements.txt file:

appdirs==1.4.3
decorator==4.3.0
numpy==1.15.4
pybind11==2.2.4
pyopencl==2018.2.2
pytools==2018.5.2
six==1.12.0

Running pip install -r requirements.txt fails with:

Collecting appdirs==1.4.3 (from -r reqs.txt (line 1)) Downloading
Collecting decorator==4.3.0 (from -r reqs.txt (line 2)) Downloading
Collecting numpy==1.15.4 (from -r reqs.txt (line 3)) Downloading (13.8MB) 100% |████████████████████████████████| 13.8MB 1.2MB/s
Collecting pybind11==2.2.4 (from -r reqs.txt (line 4)) Downloading (145kB) 100% |████████████████████████████████| 153kB 1.2MB/s
Collecting pyopencl==2018.2.2 (from -r reqs.txt (line 5)) Downloading (341kB) 100% |████████████████████████████████| 348kB 1.1MB/s Complete output from command python setup.py egg_info: --------------------------------------------------------------------------- Pybind11 is not installed. --------------------------------------------------------------------------- Very likely, the build process after this message will fail. Simply press Ctrl+C and type python -m pip install pybind11 to fix this. If you don't, the build will continue in a few seconds. [1] --------------------------------------------------------------------------- Continuing in 1 seconds... --------------------------------------------------------------------------- Mako is not installed. --------------------------------------------------------------------------- That is not a problem, as most of PyOpenCL will be just fine without it. Some higher-level parts of pyopencl (such as pyopencl.reduction) will not function without the templating engine Mako [1] being installed. If you would like this functionality to work, you might want to install Mako after you finish installing PyOpenCL. Simply type python -m pip install mako either now or after the installation completes to fix this. [1] --------------------------------------------------------------------------- Hit Ctrl-C now if you'd like to think about the situation. --------------------------------------------------------------------------- Continuing in 1 seconds... Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-d25rz2_5/pyopencl/setup.py", line 353, in <module> main() File "/tmp/pip-install-d25rz2_5/pyopencl/setup.py", line 320, in main language='c++', File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 41, in __init__ self._include_dirs = self.include_dirs File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 55, in get_include_dirs return self._include_dirs + self.get_additional_include_dirs() File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 52, in get_additional_include_dirs return [self.get_numpy_incpath()] File "/tmp/pip-install-d25rz2_5/pyopencl/aksetup_helper.py", line 47, in get_numpy_incpath file, pathname, descr = find_module("numpy") File "/home/app/.venv/lib/python3.5/imp.py", line 296, in find_module raise ImportError(_ERR_MSG.format(name), name=name) ImportError: No module named 'numpy' ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-d25rz2_5/pyopencl/

Whereas running:

pip install numpy==1.15.4
pip install pybind11==2.2.4
pip install pyopencl==2018.2.2

works like a charm. Any idea why? Thanks in advance.

--------- EDIT: ---------

I have also tried with pip-tools without success... In this case I created a file requirements.in:

numpy
pybind11
pyopencl

And ran pip-compile, which fails with:

Traceback (most recent call last): File "/home/app/.venv/bin/pip-compile", line 11, in <module> sys.exit(cli()) File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/app/.venv/lib/python3.5/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/app/.venv/lib/python3.5/site-packages/piptools/scripts/compile.py", line 196, in cli results = resolver.resolve(max_rounds=max_rounds) File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 101, in resolve has_changed, best_matches = self._resolve_one_round() File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 198, in _resolve_one_round for dep in self._iter_dependencies(best_match): File "/home/app/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 284, in _iter_dependencies dependencies = self.repository.get_dependencies(ireq) File "/home/app/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 217, in get_dependencies self._dependencies_cache[ireq] = self.resolve_reqs(download_dir, ireq, wheel_cache) File "/home/app/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 183, in resolve_reqs results = resolver._resolve_one(reqset, ireq) File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for self.require_hashes File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement abstract_dist.prep_for_dist(finder, self.build_isolation) File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 126, in prep_for_dist self.req.run_egg_info() File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 473, in run_egg_info command_desc='python setup.py egg_info') File "/home/app/.venv/lib/python3.5/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp0ykgmwk4build/pyopencl/

1 Answer

Running pip install -r requirements.txt fails ... Whereas running:

pip install numpy==1.15.4
pip install pybind11==2.2.4
pip install pyopencl==2018.2.2

works like a charm. Any idea why?

That's because the two commands are not identical. pip install supports multiple arguments (installing multiple packages in one pass), so the command equivalent to pip install -r requirements.txt would rather be

$ pip install numpy==1.15.4 pybind11==2.2.4 pyopencl==2018.2.2 # etc

and this one will fail. The reason for this is that pyopencl contains not only Python code, but also C code (so-called "C extension") that needs to be compiled with a C compiler before it can be installed. For the compilation, both numpy and pybind11 must already be installed when pyopencl starts building the extensions, but they are only scheduled for installation at that time. Thus, pip is unable to build pyopencl, thus can't prepare it for installation and in the end, fails the whole command, installing nothing at all.

This is unfortunately quite a common issue. There are two ways to avoid it:

1. Basically, this is the same approach as the one you've already discovered: install different groups of packages in separate pip install commands. Split the pip install -r requirements.txt command into two: create requirements-pre.txt (name is not relevant). Move numpy and pybind11 from requirements.txt to requirements-pre.txt. Now install the packages with

   $ pip install -r requirements-pre.txt && pip install -r requirements.txt

2. Use pipenv and Pipfiles instead of pip and "requirements" text files, as pipenv knows how to reinstall failed dependencies in a single pass. Packages persisted in the Pipfile

   [[source]]
   url = ""
   verify_ssl = true
   [packages]
   numpy = "==1.15.4"
   pybind11 = "==2.2.4"
   pyopencl = "==2018.2.2"

can be installed with a single pipenv install command.

And ran pip-compile, which fails with:

This is the same error as with pip install, only that the original traceback is hidden by pip-compile.