Category Archives: Programming

Tips for using a private Docker registry

This is a geeky post for those Googling for relevant phrases. Sometimes a Docker Registry is referred to as a ‘Docker repository’; technically, they’re different things, but the terms are often used interchangeably.

It can be useful to have a private Docker repository for sharing images within your organisation, and from which to deploy the definitive versions of your containers to production.

At Telemarq, we do this by running:

  • a standard registry:2 container on one of our DigitalOcean servers
  • an nginx container in front of it, with basic HTTP auth enabled
  • a letsencrypt system to provide HTTPS certificates for nginx, so the communications are secure.

The registry container can, itself, handle both authentication and the certificates, but it’s easier for us to deploy it this way as part of our standard infrastructure. It all works very nicely, and we’re just starting to incorporate it into some of our more serious workflows.

So how do you make sure that the right images end up in your repository?

One practice we adopt for any deployment system, with or without Docker, is to require that things pushed to the servers should come directly from the git repository, so that they aren’t influenced by what just happens to be in the directory on some arbitrary machine at some time. Typically we might have a script that creates a temporary directory, checks out a known version of the code, builds and deploys it to the server, and then tidies up after itself. (If you use a continuous delivery system, this may happen automatically on a regular basis.)

In the Docker world, you can take advantage of the fact that the docker command itself understands git repositories. So you can build a container from the current master branch of your github project using something like:

docker build -t myproject

and docker will do the necessary bits behind the scenes, assuming there’s a Dockerfile in the source. (More details here).

So, suppose you want to build version 1.6 of ‘myapp’ and upload it, appropriately tagged, to your Docker registry, you can do so with a couple of simple commands:

docker build -t \
docker push

I can run this on my Mac, a Windows machine, or any Linux box, and get a consistent result. Very nice. You could also tag it with the SHA1 fingerprint of the git commit, if wanted.

Listing the containers on your Docker registry

At present, there isn’t a convenient command-line interface for examining what’s actually stored on your registry. If I’m wondering whether one of my colleagues has already created and uploaded a container for one of our services, how would I know? There is, however, an HTTP API which will return the information as JSON, and you can then use the excellent jq utility to extract the bits you need:

curl -s -u user:password | jq .repositories[]

If you want to see the version tags available for mycontainer, you can use:

curl -s -u user:password | jq .tags[]

And you can of course wrap these in scripts or shell aliases if you use them often.

Hope that’s useful for someone!

What’s in a name?

Many years ago, I was helping a local church with a project which involved a database of the participating parishioners. This was stored in the columns of a spreadsheet, and occasionally printed out strange things in the lists of names – like ‘1/6’, or ‘1/5/05’. Most bizarre.

I eventually uncovered the problem: one of the members of the congregation was named ‘June’. Another was called ‘May’. And when they had been imported into the spreadsheet, it was being far too clever for its own good! I found out just before adding a nice lady whose name was ‘April’…

Even my simple double-barrelled surname causes some problems: a surprising number of systems can’t cope with hyphenated names. For a while I seemed to be undergoing a lot of security checks at airports, which one member of staff suggested might be because my passport had a hyphen in my name, but the airline systems invariably did not, so I never matched up as expected. The US Patent Office gets similarly confused, and in some search engines, a ‘minus’ indicates an exclusion, so if you search for ‘Stafford-Fraser’ you are guaranteed never to get me because I have a ‘Fraser’ in my name. Sigh. I don’t envy those who have names with more complicated punctuation…

Unusual initials, while general handy, also have their downsides. I could never get a good personalised license plate for my car, for example, because Qs are deemed to be too easily confused with Os or zeroes in the UK and are not allowed (except in a few very specific circumstances). My friend Brian Robinson told me about an occasion when his son Xavier was excluded from something at school, if I remember correctly, because they mistook his initial, ‘X’, as a cross indicating he was crossed off the list!

So it’s appropriate that it was Brian who forwarded a Wired article by someone with a problem I hadn’t previously considered: his surname causes much more confusion for many computer systems than mine, because his name is Christopher Null.

Optimising the size of Docker containers

Or ‘Optimizing the size of Docker containers’, in case people from America or from Oxford are Googling for it…

For Docker users, here are a couple of tricks when writing Dockerfiles which can help keep the resulting container images to a more manageable size.

Also available on Vimeo here.

Using nginx as a load-balancing proxy with the Docker service-scaling facilities

There’s a geeky title for you! But it might help anyone Googling for those keywords…

Recent versions of Docker have many nice new facilities. Here’s a demo of how you can use the service-scaling to run multiple instances of your app back-end, and Nginx as a front-end proxy, while keeping track of them using the round-robin DNS facility built in to the Docker engine.

All demonstrated in a few lines of code on my laptop, using the new Docker for Mac.

Also available on YouTube.

With thanks to Jeppe Toustrup for some helpful hints. Have a look at his page for more detailed information. Also see the Docker channel on YouTube for lots of talks from the recent DockerCon.

Replace wildcard imports in Python code

From our just-in-case-you’re-Googling-for-it department…

In Python code, wildcard import statements, such as:

from foo import *

can be very convenient, but are now usually considered bad practice.

I’ve written a really simple tool called dewildcard to help replace them with full expansions, which can then be trimmed down using a tool like pylint or pyflakes.

Just in case it’s useful…

Coming into alignment

The project I’m currently involved in at the University Computer Lab is investigating how we can make use of some of the lab’s computer vision and human interface expertise to improve the experience for car drivers. This is being done in conjunction with Jaguar LandRover, and we’ve been equipping a Discovery with several cameras, some looking at the driver and passengers, some at the surroundings.


We can also record various bits of other telemetry from the car’s internal CAN network, and some of the cameras are synchronised to this, but not all of them.

My knowledge of computer vision techniques is about a quarter of a century out of date, so as part of a learning experiment today I was thinking about ways of automatically synchronising the video to the data. The two graphs below show the first minute and a half of a recording, starting with the stationary car moving off and going through a couple of turns. This isn’t how we’ll eventually do it, but it was a fun programming experiment!

This first plot shows the angle of the steering wheel, as recorded by the car’s own sensors.


This second plot shows the average horizontal component of optical flow seen by one of the front-facing cameras: in essence, it’s working out whether more of the pixels in the view that the driver sees are moving left or right.


I’m quite pleased that these two completely independent sources of data show such a nice correlation. Optical flow is, however, pretty time-consuming to calculate, so I only got through the first 90 seconds or so of video before it was time for dinner.

Fun stuff, though…

Experimenting with Bower and Flask

This is another of those “just in case you’re Googling for it” posts…


  • Flask is a nice lightweight Python web framework – a handy way to create small web apps when you don’t need the power of something like Django.
  • Bower is a package manager for web components – a really easy way to get the latest versions of jQuery, bootstrap and anything else you might want.

Bower will put the packages into a folder called bower_components, with the idea that your deployment process will use something like Grunt to move the static bits from there into their final location. This is all well and good, but it seems like overkill if you’re just experimenting on your own machine.

Wouldn’t it be nice if Bower could just put the packages somewhere that they could be served directly as static files by Flask?

Well, it turns out that it can. If you create a file in the top level of your app called .bowerrc containing the following:

   "directory" : "static/bower_components"

then Bower will put its components in a subdirectory of the static folder and Flask then knows how to serve them up. You might do:

bower install jquery
bower install bootstrap

and then in your HTML you could use something like the following:

<link rel="stylesheet"
    href="/static/bower_components/bootstrap/dist/css/bootstrap.min.css" />

and for the Javascript:

<script src="/static/bower_components/jquery/dist/jquery.min.js"></script>
<script src="/static/bower_components/bootstrap/dist/js/bootstrap.min.js"></script>

You could leave out the bower_components bit at each stage and make your URLs even tidier, but I felt it was still useful to separate those bits that were being managed by Bower from those that weren’t.

Hope that helps someone!

The march of progress


One of the things that has always been a challenge for developers is the ‘progress bar’. It can be very difficult to predict in advance just how long something is going to take: you work it out for, say, a typical operating system update on the typical machine and then you find that some users have 100,000 things in their Trash, or are in the midst of a backup to a slow external drive, or whatever…

Installing Mac OS X Yosemite this morning, the progress bar sat at ‘About one minute remaining’ for well over half an hour, so I went and did some Googling and found that I was not alone – many people waited much longer than that, but it always completed in the end. Of course, like a watched pot, it will never get there if you’re sitting waiting for it, so you have to go and do something else. Usually, it is just a mild annoyance for impatient enthusiasts like me, but occasionally it can be a more serious problem if you’re told something will take 30 mins, for example, and you therefore assume you can do it before your appointment in an hour’s time.

Anyway, watching the slowly-progressing pixels gave me an idea…

There are tens of millions of people who will be going through this process over the next few months: surely you could improve the accuracy of their progress bars by uploading the timing information at the end of each installation, along with basic information about the system and then using that to give more accurate estimates to those with similar machines, similar disk usage, etc?

DevOps the Google way

DevOps is a trendy term in the software industry in the last few years.

If you haven’t come across it, it’s (roughly) about trying to break down the traditional barriers that can grow up between software developers (who want to create exciting new features and get them out there) and the operations people (who have to ensure the software runs reliably and deal with the problems if it doesn’t, so are a natural back-pressure on the speed of innovation and deployment).

At Telemarq, where we currently have about 2.5 people, this distinction is not large 🙂 We all do everything and we all deal with the bugs. But the bigger the organisation gets, the more danger there is that these relationships can be frustrating or uncomfortable.

By the time you get to be the size of Google, and particularly when you have a user base that you number in hundreds of millions, you need to find good ways to make this work. Google has a team called ‘SRE’, focused on Site Reliability Engineering. I can’t say that a talk on reliability engineering sounded particularly inspiring, but I learned a lot from this one by Ben Traynor, who runs it. Some very good management ideas here: recommended for anyone in the software industry, and particularly for those in larger companies.


Docker logoA geeky post. You have been warned.

I wanted to make a brief reference to my favourite new technology: Docker. It’s brief because this is far too big a topic to cover in a single post, but if you’re involved in any kind of Linux development activity, then trust me, you want to know about this.

What is Docker?

Docker is a new and friendly packaging of a collection of existing technologies in the Linux kernel. As a crude first approximation, a Docker ‘container’ is like a very lightweight virtual machine. Something between virtualenv and VirtualBox. Or, as somebody very aptly put it, “chroot on steroids”. It makes use of LXC (Linux Containers), cgroups, kernel namespaces and AUFS to give you most of the benefit of running several separate machines, but they are all in fact using the same kernel, and some of the operating system facilities, of the host. The separation is good enough that you can, for example, run a basic Ubuntu 12.04, and Ubuntu 14.04, and a Suse environment, all on a Centos server.

“Fine”, you may well say, “but I can do all this with VirtualBox, or VMWare, or Xen – why would I need Docker?”

Well, the difference is that Docker containers typically start up in milliseconds instead of seconds, and more importantly, they are so lightweight that you can run hundreds of them on a typical machine, where using full VMs you would probably grind to a halt after about half a dozen. (This is mostly because the separate worlds are, in fact, groups of processes within the same kernel: you don’t need to set aside a gigabyte of memory for each container, for example.)

Docker has been around for about a year and a half, but it’s getting a lot of attention at present partly because it has just hit version 1.0 and been declared ready for production use, and partly because, at the first DockerCon conference, held just a couple of weeks ago, several large players like Rackspace and Spotify revealed how they were already using and supporting the technology.

Add to this the boot2docker project which makes this all rather nice and efficient to use even if you’re on a Mac or Windows, and you can see why I’m taking time out of my Sunday afternoon to tell you about it!

I’ll write more about this in future, but I think this is going to be rather big. For more information, look at the Docker site, search YouTube for Docker-related videos, and take a look at the Docker blog.

The race is to the Swift?

swiftI love my Mac and iOS devices, but writing native apps for them has always been made somewhat less pleasurable by the programming languages available. Objective-C (which is behind the typical app on your iPhone or Mac) has its merits, or at least, had its merits when it was designed 30 years ago, but things have moved on quite a lot since then. And don’t get me started on the abomination that is AppleScript…

That’s why, amongst the panoply of geeky goodies that Apple announced at its developer conference this week, the thing that interested me most is their new programming language, Swift, which looks rather lovely. (You can find excellent introductory talks about it here.) It’s early days yet, but may be good enough that, henceforward, people will flock to Apple’s development environment because of, rather than despite, the language.

It’s not clear whether Swift will be available anywhere other than on Apple platforms, and there may be a certain degree of deliberate lock-in here. But that’s better than the old situation where Objective-C was available elsewhere, but nobody really cared.

All of which may help to explain why the book The Swift Programming Language had been downloaded by more than a third of a million people within the first 24 hours of anyone knowing the language even existed.

Wiki lacks?


The only example of a wiki that worked is Wikipedia, and that’s not really a wiki in the true sense, because it is heavily edited by such a small number of people.

That’s Russell Keith-Magee, speaking on a podcast (so I may not have quoted him exactly). He was talking particularly in the context of software documentation, but I think, on consideration, that he’s right.

Don’t get me wrong, I love wikis, use them quite a bit, and shook Ward Cunningham‘s hand very enthusiastically when I met him a few years ago. But on the other hand, if one has been spoiled by the quality of documentation in projects such as Django, it makes one’s heart sink when exploring a new piece of software to discover that the documentation is ‘only a wiki’.

Wikis are great for their free-form, collaborative nature – remember, they came into existence long before Google docs and SubEthaEdit – but they also suffer from a few key problems:

  • They can make you feel as if you’ve done something about documentation, so you don’t do the real work. “The crowd will fix it later”.
  • The fact that you don’t need to think about the structure early on means they often need to be reorganised – making it hard for others to find things even if they’ve visited before, and harder to link to things reliably.
  • It’s notoriously difficult to move content from a wiki into anything else, including another wiki package.
  • It can be hard, with a software package, to know to which version the documentation applies.
  • As with mind-maps, the free-form nature closely represents the mental model of the author. It’s good for personal notes and reference (Voodoopad is lovely, for example), but the difference between structured documentation and a wiki is the difference between a talk/lecture and a rambling conversation. The letter may be more fun, but does it achieve as much in the same amount of time?

None of this means, of course, that you can’t write good documentation with a wiki. It’s just very rarely done, and when it is done, it probably takes a lot of effort. Wikis are the whiteboard doodle, the back-of-the-envelope scribble: they can be useful within a small internal group of collaborators, but they’re not the thing you want to present to the world.

Actually, I think one of the best ways of doing documentation was the approach adopted by the PHP developers many, many years ago – you write proper docs but allow comments at the bottom of each page. Here’s an example. They did this long before the concept of comments-at-the-bottom-of-the-page was well established.

What do you think?

© Copyright Quentin Stafford-Fraser