Human → Computer

Jonathan Stoppani - Managing dependencies of Python projects pyconde16

written by Patrick Gerken on 2016-10-30


Part 1

You start with::

pip install django
pip install boto...

Now lets deploy. SSH to server, copy dist files, copy source, talk done!

Second deployment:
O, missed dependencies. requirements.txt to the rescue

Third deployment:
Everything breaks again, because you did not pin your packages yet.

Fourth deployment:
Also pin dependencies of your dependencies:

pip freeze > requirements.txt

pip does not check that your dependencies are consistent to each other.

pip-tools helps with that.

pip-compile recursively compiles dependencies together and checks for consistency

Create a with top level requirements.
Run pip-compile, and the requirements.txt contains all pinned requirements.

Every time you run pip-compile, the second level dependencies get updated again. If after that, tests fail, you must update your to document the known incompatibility.

Part 2


pip compile takes time and can be platform specfic. pip install takes time to resolve distributions, resolves depdencies and install the distributions.

Our solution: wheelsproxy a pypi proxy, builds wheels, knows abouts platform.

Uses docker to build wheels for the different platforms.

pip install went down from 180 seconds to 15 seconds.

Demo time: file with 15 requirements. uploads to wheelsproxy, then pip compile runs on the server with cached data. pip installed django with 120 requirements in 40 seconds.


Questions: What about tools to see if there are newer versions.
Answer: you committed the reqs.txt, so run git-compile again and check the diff.