Docker in production -- running out of disk space

January 23, 2021 on Eric Bower's blog

tl;dr: If you are deploying docker containers to production, be sure to run docker system prune -f before pulling new images so you don’t run out of disk space.

When I’m building a new project, I generally learn towards using docker compose during development. When coupled with docker machine I have a quick and easy way to deploy my containers to the cloud provider of my choice. Overall, it works really well, but there’s one important thing to consider when using docker for production: running out of disk space.

Images, containers, networks, and volumes will continue to grow on a production VM which will inevitably lead to the hard drive running out of memory. This seems obvious, but it wasn’t obvious to me when I first started using docker-machine.

I recently deployed a new app listifi to google cloud compute. Here’s a brief overview of my deployment process:

This process works great. I don’t need to setup CI for a new project but it still provides me with the flexibility to deploy to my own VM and inspect its health with docker.

The problem

Things are working great, I’m iterating on feature development and deploying in between major changes. Only, the last deploy I tried to perform failed. The reason: hard drive is out of space. Hmm, my VM has 16GB of diskspace, why is it out of memory?

When I run docker system df the problem becomes clear: I have unused images soaking up all of my hard drive space. I have scoured docker deployment strategies and never came across documentation that references this issue. There are plenty of StackOverflow issues referencing the problem with a solution, but I never really made the connection to my production VM until I hit this problem.

The solution

Before I pull my new images in production, I run another command:

docker system prune -f

You can read the documentation about this command here. Once I added that to my deployment step, things are working much more smoothly.

Conclusion

I completely forgot that as I deployed new changes to production, there was lingering old docker images laying dormant on my production VM, increasing in size of time.

It never clicked for me until I ran into the problem, hopefully this short blog article will help others avoid this problem in the future as well.


Articles from blogs I read

Generated by openring

The Fediverse can be pretty toxic

Mastodon, inspired by GNU social, together with Pleroma, form the most popular components of what we know as the “Fediverse” today. All of them are, in essence, federated, free software Twitter clones, interoperable with each other via the ActivityPub protoc…

via Drew DeVault's blog July 9, 2022

Spearphishing: it can happen to you too

Image generated by MidJourney -- The Fool in a woodcut tarot card style For some reason, LinkedIn has become the de-facto social network for professionals. It is viewed as a powerful networking and marketing site that lets professionals communicate, find new…

via Xe's Blog July 9, 2022

SourceHut is committed to making IRC better

Internet Relay Chat (IRC) is a wonderful protocol with a 34-year history of helping free software, there at every step alongside the rise of the internet. Many real-time chat empires have risen and fallen during its tenure, some of them leaving behind lesson…

via Blogs on Sourcehut July 6, 2022