shiv - the Python Tool You Didn't Know You Wanted
Your code + a zip archive = single file deployments
I know that’s an incredibly enticing intro, but bear with me. We’ll get there.
Way back in the dark ages of 2008, Python 2.6 was released. This release was particularly exciting because it brought us such wondrous things as the
multiprocessing module, the
io module, and codified the
with statement as a keyword for doing better-controlled contextual processing. It also brought a lot of existential angst about the upcoming Python 3 release, but we’re skipping over that.
Buried in the release notes was a single paragraph that mentioned “oh hey by the way you can now run Python files from inside a zip archive”.
Imagine, being able to call a zip file with Python and have it actually work:
This was, somewhat predictably, completely missed by the vast majority of folks. In 2013, PEP 441 attempted to fix this by bringing the whole ‘package your stuff in a zip file’ thing into the public view by adding the
zipapp module to the standard library, adding a whole suite of tools to make working with zipapps much easier for the average Python developer.
This was such a revolutionary change that it got a whole three sentences in the release notes and was more or less completely overshadowed by the introduction of
typing as a core Python feature. Sorry, Daniel and Paul – you tried.
Now that we have reasonable support for making and using zipapps, we can cover the single huge glaring issue: no third-party dependencies.
Zipapps are fantastic! They let you bundle ALL of your Python code into a single zip archive, pass that archive off to someone else, and they can just run it! However, there’s one glaring issue: you can only rely on the standard library. Third party dependencies are right out unless you manually bundle them with your codebase, which is both a pain to do in the first place and an easy way to accidentally run afoul of licenses if you’re not paying close enough attention.
I don’t know about you, but I basically never write things that rely only on the standard library – third party dependencies are a fact of life for all the Python developers I know. So how can we get the power of third party dependencies with the pure awesomeness of being able to package an entire runnable program into a zip file?
The answer, as it turns out, was figured out by all those cool folks at Twitter who aren’t there anymore. They designed and built
pex, a system that solves the third party dependency problem by essentially zipping up the virtual environment (minus the interpreter) and extracting it prior to running the zipapp. The implementation used by
pex is functional, though a bit on the slow side.
shiv is LinkedIn’s attempt at solving the speed issues of
pex while still keeping all of the functionality. (You can find more information about the speed differences between
shiv here. You can also find more technical information on
shiv’s approach at that link as well.) You only need a few things installed to actually make a
- a recent version of Python
- an environment that includes
- a setup.py file that lists all the dependencies of your app
- your app code itself
That’s really it, and in fact, that’s all you need to have in your CI pipeline if you want to automatically build your
shiv apps for CI-controlled releases. Take a look at this example, starting from an empty directory:
We have a single file that calls a third-party dependency and attempts to print a reference. If we call it now with
python cli.py it’ll break, since we don’t have
httpx installed. However, we’ll add a reference to that in our setup.py file now.
Now that we have a pretty basic
setup.py file and everything else configured, we can make the app:
If we check our directory, we have a new file –
myapp.pyz. Let’s run it!
Now we have a single zip archive that includes a third party dependency that can be referenced and interacted with! It printed the reference to
httpx, so we know that it’s in there.
…actually, do we know it’s in there? Maybe it’s just picking that up from the environment that’s still activated. Let’s check.
There we go! We can see that it’s pulling the dependency from its own packaged virtual environment.
The Use Case
Being able to package your entire app into a zip file has some incredible benefits, namely that you can now run your entire app on any system that just has an appropriate version of Python installed. For applications where your app just needs to run without tweaking, this means that someone (or something) can just grab your compiled and finished zipapp and just… run it. No fuss and no stress.
I wouldn’t recommend that you check this out unless I’d given it a thorough test myself; the nonprofit I run, the Grafeas Group, has been running all of our production services as zipapps for the last 8 months. That’s:
- three Reddit bots
- a Discord bot
- a Slack bot
- a Django monolith that handles our website and APIs
Deployments have never been easier (and are completely controlled via Slack commands!) – it’s literally as easy as backing up the existing zipapp, downloading the new one, and running it. The Slack bot, Bubbles, updates itself by downloading the new version on top of the existing one and restarting its own service!
The benefit here is that if the new version fails, getting back to a working state just swapping out the older version of the zipapp and restarting the service. What’s that? You made changes to the virtual environment between versions? Doesn’t matter because each zipapp uses its own environment. If it compiles and it works, it can’t be affected by any other zipapp, even later or earlier compilations of the same system.
Spinning up a new server is as easy as getting a new VM, installing Python on it, and slapping the zipapps into place with their respective environment variable files. We intentionally keep it simple because this is a volunteer gig and there’s no reason to muddy the devops waters by adding something like Docker. But let’s say you wanted to add Docker or another container; would that be any more complex?
Honestly, not really. You’d basically just get the official Python container, drop your zipapp onto the root and set your launch script to call it. It does kind of defeat the purpose of easy updates, but it can be done!
Stuff to Watch For
As cool as using
shiv for easily-deployable apps is, there are a few things that you need to know if you want to use it in production.
shiv puts the virtual environment for that zipapp in
~/.shiv unless you override it, and since each zipapp (and each version of each zipapp) has its own virtual environment, that means that this directory can get reasonably busy. The easiest way to address this is to use the “preamble” section of
shiv’s setup, which allows you to run a single script before actually starting the zipapp.
There’s a complete preamble script that handles abandoned environments available in the
shiv documentation – adding and extending it is pretty straightforward, but it is something that you need to be aware of as environments can build up across deployments.
The only other major thing you need to watch out for is relative paths. Let’s edit our
If we run this directly with
python myapp/cli.py, here’s what I get:
But if we compile this into a zipapp, things will look a little different:
…which is not where we expected it to be. That means that if you want to load external files that are in the same directory as the compiled zipapp (and
os.getcwd() might not be accurate for your use case), you’ll need to ask shiv itself where you’re running. Thankfully, there’s an extra little bit that gets injected into the final zipapp during build time, and there are a couple of utilities there that can help us out.
Let’s rebuild our script a little bit – now we’ll do a check to see if
shiv is available in the current environment and then print out the directory that our zipapp is in. This would be useful anytime you want to load external files in the same directory, like a
.env file, a
db.sqlite3 file, or something else along those lines.
Here’s the new
Now, let’s try running it!
We use this method to create a
BASE_DIR variable that’s accessible throughout the application along the lines of Django’s BASE_DIR (which is very wrong when packaged with
shiv). That allows easy loading of external files and makes it work the way you expect it to. A little bit of boilerplate, but nothing too bad once you know to watch for it.
If you’re looking for a low-tech way to deploy a single python app (or a BUNCH of them!) or just want to pass scripts and things around to coworkers,
shiv is worth your time to check out. Coupled with building and releasing the zipapps in your CI pipeline, it’s a system that is really hard to beat. Usability is great, using it is a snap, and if you pair it with
click, you can get extremely usable CLI programs for essentially no cost.
The linked repositories in this post are a great place to look if you’re wanting to see what this looks like in practice, and you can also take a look at my self-updating utils script here if you’d like to see how
click can be used to add an easy
--update flag that checks for a new release and automatically installs it!