Category Archives: Programming

Using nginx as a load-balancing proxy with the Docker service-scaling facilities

There’s a geeky title for you! But it might help anyone Googling for those keywords…

Recent versions of Docker have many nice new facilities. Here’s a demo of how you can use the service-scaling to run multiple instances of your app back-end, and Nginx as a front-end proxy, while keeping track of them using the round-robin DNS facility built in to the Docker engine.

All demonstrated in a few lines of code on my laptop, using the new Docker for Mac.

Also available on YouTube.

With thanks to Jeppe Toustrup for some helpful hints. Have a look at his page for more detailed information. Also see the Docker channel on YouTube for lots of talks from the recent DockerCon.

Update, spring 2017: Do note that if you’re using Docker Swarm, you may want to adopt a more complex approach, perhaps based on Interlock.

Possibly-related posts:

Replace wildcard imports in Python code

From our just-in-case-you’re-Googling-for-it department…

In Python code, wildcard import statements, such as:

from foo import *

can be very convenient, but are now usually considered bad practice.

I’ve written a really simple tool called dewildcard to help replace them with full expansions, which can then be trimmed down using a tool like pylint or pyflakes.

Just in case it’s useful…

Possibly-related posts:

Coming into alignment

The project I’m currently involved in at the University Computer Lab is investigating how we can make use of some of the lab’s computer vision and human interface expertise to improve the experience for car drivers. This is being done in conjunction with Jaguar LandRover, and we’ve been equipping a Discovery with several cameras, some looking at the driver and passengers, some at the surroundings.

2015-07-09_10-56-23-07

We can also record various bits of other telemetry from the car’s internal CAN network, and some of the cameras are synchronised to this, but not all of them.

My knowledge of computer vision techniques is about a quarter of a century out of date, so as part of a learning experiment today I was thinking about ways of automatically synchronising the video to the data. The two graphs below show the first minute and a half of a recording, starting with the stationary car moving off and going through a couple of turns. This isn’t how we’ll eventually do it, but it was a fun programming experiment!

This first plot shows the angle of the steering wheel, as recorded by the car’s own sensors.

steer

This second plot shows the average horizontal component of optical flow seen by one of the front-facing cameras: in essence, it’s working out whether more of the pixels in the view that the driver sees are moving left or right.

flow

I’m quite pleased that these two completely independent sources of data show such a nice correlation. Optical flow is, however, pretty time-consuming to calculate, so I only got through the first 90 seconds or so of video before it was time for dinner.

Fun stuff, though…

Possibly-related posts:

  • FFE Paul Bissex writes about his desire for a Focus Follows...
  • iChat and USB cameras Apple’s iChat AV allows you to make video calls. I...
  • Gears John’s musings about his old cars, and Sean and Nicci’s...
  • Ambient social networking Most of you will know by now that my company,...

Experimenting with Bower and Flask

This is another of those “just in case you’re Googling for it” posts…

Background:

  • Flask is a nice lightweight Python web framework – a handy way to create small web apps when you don’t need the power of something like Django.
  • Bower is a package manager for web components – a really easy way to get the latest versions of jQuery, bootstrap and anything else you might want.

Bower will put the packages into a folder called bower_components, with the idea that your deployment process will use something like Grunt to move the static bits from there into their final location. This is all well and good, but it seems like overkill if you’re just experimenting on your own machine.

Wouldn’t it be nice if Bower could just put the packages somewhere that they could be served directly as static files by Flask?

Well, it turns out that it can. If you create a file in the top level of your app called .bowerrc containing the following:

{
   "directory" : "static/bower_components"
}

then Bower will put its components in a subdirectory of the static folder and Flask then knows how to serve them up. You might do:

bower install jquery
bower install bootstrap

and then in your HTML you could use something like the following:

<link rel="stylesheet"
    href="/static/bower_components/bootstrap/dist/css/bootstrap.min.css" />

and for the Javascript:

<script src="/static/bower_components/jquery/dist/jquery.min.js"></script>
<script src="/static/bower_components/bootstrap/dist/js/bootstrap.min.js"></script>

You could leave out the bower_components bit at each stage and make your URLs even tidier, but I felt it was still useful to separate those bits that were being managed by Bower from those that weren’t.

Hope that helps someone!

Possibly-related posts:

The march of progress

progress

One of the things that has always been a challenge for developers is the ‘progress bar’. It can be very difficult to predict in advance just how long something is going to take: you work it out for, say, a typical operating system update on the typical machine and then you find that some users have 100,000 things in their Trash, or are in the midst of a backup to a slow external drive, or whatever…

Installing Mac OS X Yosemite this morning, the progress bar sat at ‘About one minute remaining’ for well over half an hour, so I went and did some Googling and found that I was not alone – many people waited much longer than that, but it always completed in the end. Of course, like a watched pot, it will never get there if you’re sitting waiting for it, so you have to go and do something else. Usually, it is just a mild annoyance for impatient enthusiasts like me, but occasionally it can be a more serious problem if you’re told something will take 30 mins, for example, and you therefore assume you can do it before your appointment in an hour’s time.

Anyway, watching the slowly-progressing pixels gave me an idea…

There are tens of millions of people who will be going through this process over the next few months: surely you could improve the accuracy of their progress bars by uploading the timing information at the end of each installation, along with basic information about the system and then using that to give more accurate estimates to those with similar machines, similar disk usage, etc?

Possibly-related posts:

DevOps the Google way

DevOps is a trendy term in the software industry in the last few years.

If you haven’t come across it, it’s (roughly) about trying to break down the traditional barriers that can grow up between software developers (who want to create exciting new features and get them out there) and the operations people (who have to ensure the software runs reliably and deal with the problems if it doesn’t, so are a natural back-pressure on the speed of innovation and deployment).

At Telemarq, where we currently have about 2.5 people, this distinction is not large 🙂 We all do everything and we all deal with the bugs. But the bigger the organisation gets, the more danger there is that these relationships can be frustrating or uncomfortable.

By the time you get to be the size of Google, and particularly when you have a user base that you number in hundreds of millions, you need to find good ways to make this work. Google has a team called ‘SRE’, focused on Site Reliability Engineering. I can’t say that a talk on reliability engineering sounded particularly inspiring, but I learned a lot from this one by Ben Traynor, who runs it. Some very good management ideas here: recommended for anyone in the software industry, and particularly for those in larger companies.

Possibly-related posts:

Docker

Docker logoA geeky post. You have been warned.

I wanted to make a brief reference to my favourite new technology: Docker. It’s brief because this is far too big a topic to cover in a single post, but if you’re involved in any kind of Linux development activity, then trust me, you want to know about this.

What is Docker?

Docker is a new and friendly packaging of a collection of existing technologies in the Linux kernel. As a crude first approximation, a Docker ‘container’ is like a very lightweight virtual machine. Something between virtualenv and VirtualBox. Or, as somebody very aptly put it, “chroot on steroids”. It makes use of LXC (Linux Containers), cgroups, kernel namespaces and AUFS to give you most of the benefit of running several separate machines, but they are all in fact using the same kernel, and some of the operating system facilities, of the host. The separation is good enough that you can, for example, run a basic Ubuntu 12.04, and Ubuntu 14.04, and a Suse environment, all on a Centos server.

“Fine”, you may well say, “but I can do all this with VirtualBox, or VMWare, or Xen – why would I need Docker?”

Well, the difference is that Docker containers typically start up in milliseconds instead of seconds, and more importantly, they are so lightweight that you can run hundreds of them on a typical machine, where using full VMs you would probably grind to a halt after about half a dozen. (This is mostly because the separate worlds are, in fact, groups of processes within the same kernel: you don’t need to set aside a gigabyte of memory for each container, for example.)

Docker has been around for about a year and a half, but it’s getting a lot of attention at present partly because it has just hit version 1.0 and been declared ready for production use, and partly because, at the first DockerCon conference, held just a couple of weeks ago, several large players like Rackspace and Spotify revealed how they were already using and supporting the technology.

Add to this the boot2docker project which makes this all rather nice and efficient to use even if you’re on a Mac or Windows, and you can see why I’m taking time out of my Sunday afternoon to tell you about it!

I’ll write more about this in future, but I think this is going to be rather big. For more information, look at the Docker site, search YouTube for Docker-related videos, and take a look at the Docker blog.

Possibly-related posts:

The race is to the Swift?

swiftI love my Mac and iOS devices, but writing native apps for them has always been made somewhat less pleasurable by the programming languages available. Objective-C (which is behind the typical app on your iPhone or Mac) has its merits, or at least, had its merits when it was designed 30 years ago, but things have moved on quite a lot since then. And don’t get me started on the abomination that is AppleScript…

That’s why, amongst the panoply of geeky goodies that Apple announced at its developer conference this week, the thing that interested me most is their new programming language, Swift, which looks rather lovely. (You can find excellent introductory talks about it here.) It’s early days yet, but may be good enough that, henceforward, people will flock to Apple’s development environment because of, rather than despite, the language.

It’s not clear whether Swift will be available anywhere other than on Apple platforms, and there may be a certain degree of deliberate lock-in here. But that’s better than the old situation where Objective-C was available elsewhere, but nobody really cared.

All of which may help to explain why the book The Swift Programming Language had been downloaded by more than a third of a million people within the first 24 hours of anyone knowing the language even existed.

Possibly-related posts:

Wiki lacks?

Quote:

The only example of a wiki that worked is Wikipedia, and that’s not really a wiki in the true sense, because it is heavily edited by such a small number of people.

That’s Russell Keith-Magee, speaking on a podcast (so I may not have quoted him exactly). He was talking particularly in the context of software documentation, but I think, on consideration, that he’s right.

Don’t get me wrong, I love wikis, use them quite a bit, and shook Ward Cunningham‘s hand very enthusiastically when I met him a few years ago. But on the other hand, if one has been spoiled by the quality of documentation in projects such as Django, it makes one’s heart sink when exploring a new piece of software to discover that the documentation is ‘only a wiki’.

Wikis are great for their free-form, collaborative nature – remember, they came into existence long before Google docs and SubEthaEdit – but they also suffer from a few key problems:

  • They can make you feel as if you’ve done something about documentation, so you don’t do the real work. “The crowd will fix it later”.
  • The fact that you don’t need to think about the structure early on means they often need to be reorganised – making it hard for others to find things even if they’ve visited before, and harder to link to things reliably.
  • It’s notoriously difficult to move content from a wiki into anything else, including another wiki package.
  • It can be hard, with a software package, to know to which version the documentation applies.
  • As with mind-maps, the free-form nature closely represents the mental model of the author. It’s good for personal notes and reference (Voodoopad is lovely, for example), but the difference between structured documentation and a wiki is the difference between a talk/lecture and a rambling conversation. The letter may be more fun, but does it achieve as much in the same amount of time?

None of this means, of course, that you can’t write good documentation with a wiki. It’s just very rarely done, and when it is done, it probably takes a lot of effort. Wikis are the whiteboard doodle, the back-of-the-envelope scribble: they can be useful within a small internal group of collaborators, but they’re not the thing you want to present to the world.

Actually, I think one of the best ways of doing documentation was the approach adopted by the PHP developers many, many years ago – you write proper docs but allow comments at the bottom of each page. Here’s an example. They did this long before the concept of comments-at-the-bottom-of-the-page was well established.

What do you think?

Possibly-related posts:

Time’s Wingèd Chariot

The best chat-up line ever invented has to be the poem To His Coy Mistress, by Andrew Marvell, where, in 46 lines of glorious verse, he explains to his girl that they really do need to get down to some hanky-panky, because they won’t always be able to do so, and time is running out. Extended courtship and foreplay is all very well, he argues,

But at my back I always hear
Time’s winged chariot hurrying near…

I was thinking of this as I created my first app for my Pebble watch today, inspired by a combination of a 17th-century poet and my friend Richard, a 21st-century technologist. He held a party a little while ago to celebrate being one gigasecond old, and has a nice web page where you can check your age, too.

So, since part of the idea of the Pebble is that you can personalise it to a great degree, I thought that a watch face that showed my age in seconds would be a good project. I named it ‘Tempus Fugit’.

But then it occurred to me that one could also do the opposite – a countdown. How long I’ve been around is an interesting topic, at least for me. But how long I’ve got – well, that’s even more compelling!

Of course, life expectancy is a highly personal thing (and one wishes to avoid the estimation errors made by the Rich Fool in the parable) but it’s interesting to speculate. Life expectancy, in statistical tables, tends to be quoted as ‘expectancy at birth’, based on an assumption that mortality rates will continue through your life at the same kind of level as they were at your birth. When I was born, in 1967, it was about 69 years in the UK. It’s now 78, according to Wikipedia, and will no doubt continue to rise. So on the one hand, one might expect that the inferior healthcare, nutrition advice etc in my youth would mean that I probably come somewhere between the two. On the other hand, today’s figure doesn’t take into account the improvements in healthcare that are inevitable in the future, which may mean that my allotted span may be more that the expectancy of those born today (as will theirs).

So, as a rough estimate, averaging these two factors, I decided to use the present life expectancy for the UK, currently around 78 for men and 82 for women. (It’s slightly more in Canada and Australia, and a couple of years less in the U.S.).

And so, with a minor variation in the software, I now have two new watchfaces for my Pebble. The first, Tempus Fugit shows how many seconds I’ve been alive. The other shows my best estimate of how many I’ve got left, a number which is already distressingly lower than the first, and, of course, counting down…

Some may think this morbid. But I prefer to think of it as an inspiration… a call to action.

Oh, and in case I should be short of ideas about what to do with the time that remains, I named this second one Time’s Wingèd Chariot. I think Marvell would approve.

Time's Winged Chariot

I may be celebrating a rather different kind of gigasecond soon!

Possibly-related posts:

Alfred 2 support for iTerm 2

iTerm 2 is a terminal program for the Mac with lots of great features beyond the standard OS X Terminal. Alfred is an excellent app launcher, which in its newly-released second version is taking the Mac world by storm.

If you don’t use either of these, I strongly recommend them.

If, on the other hand, you already use both of them, you might like my (very basic) plugin that lets you list your iTerm profiles using an Alfred keyword and fire up a new iTerm window using the selected one.

Alfred iTerm

Possibly-related posts:

Goldilocks for grey-haired geeks

When I was an undergraduate studying Computer Science, in the late 80s, the course was accredited by the British Computer Society on the condition that, as part of the syllabus, we were taught some COBOL. For those who have not encountered it, COBOL – the Common Business Oriented Language – is a very early programming language designed with the idea, amongst other things, that managers and other non-technical business types should be able to read and understand it.

In retrospect this wasn’t really a great idea (any more than it was when Apple tried something similar with AppleScript in 1993). Computer languages tend to get a kind of uncanny valley problem when you try to pretend that machines speak the same way as humans; the results are neither good English nor good programming languages. It’s a bit like Franglais, except that Franglais is enjoyable to use.

But COBOL can be forgiven; this was after all still 1959, and the world’s experience in programming languages was somewhat limited. It was designed by people as eminent as Grace Hopper (who was brilliant at handling David Letterman, as well as computers!). And COBOL was incredibly successful – for some decades it was the most-used computer language, and it’s still in use today, though it’s bizarre to discover there are people doing .NET programming with it.

Anyway, it was Dr Frank King who taught us most of our languages, and since he had to include a bit of COBOL, he did it mostly as a joke, starting with a properly-constructed “Hello World” program, which I seem to recall was 77 lines long.

At the end of the final lecture, he handed out photocopies of “The Common Business Oriented Goldilocks”, which had been published in Datamation magazine back in 1968, and which is written in something approaching valid COBOL. Maybe it is valid COBOL… I never wrote another line of COBOL after that lecture, so don’t remember. But I may have readers who do!

        THE COMMON BUSINESS ORIENTED GOLDILOCKS
            --- ------ -------- -------- ----------

IDENTIFICATION DIVISON.
PROGRAM ID.           A COBOL FABLE.
SECURITY.             INSECURE.
PROGRAMMER-ID.        ARTHUR SHAPIRO.
REMARKS.              SLIGHTLY MORE MANGLED VERSION OF ONE IN JAN., 1968
               DATAMATION.
DATE WRITTEN.         ONCE UPON A TIME.

ENVIRONMENT DIVISON.
CONFIGURATION SECTION.
OBJECT COMPUTER.      ANY MUSIC BOX, MEMORY SIZE 8X64 BYTES,
                      19 TAPE DRIVES, 11 DISK DRIVES, 1 GOLDILOCKS, 3 BEARS.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
    SELECT TAPE DRIVES, ASSIGN THEM TO CREDITOR.
    SELECT DISK DRIVES.
    SELECT GOLDILOCKS, SELECT BEARS, ASSIGN TO ONE COTTAGE.
I-O CONTROL.
    APPLY RED TAPE TO TAPE DRIVES, APPLY BRAHMS RECORD TO DISK DRIVE,
    APPLY GOLDI, BEARS TO COTTAGE.
    DATA DIVISON.
FD GOLDI.
    LABEL RECORDS ARE STANDARD
    VALUE OF IDENTIFACTION IS "GOLDILOCKS"
    DATA RECORD IS GOLDILOCKS.
01  GOLDILOCKS.
    02    HGT   SIZE IS 62 INS.
    02    WGT   SIZE IS 110 LBS.
    02    VITAL-STATS.
          03    B     38.
          03    W     24.
          03    H     36.
    02    RATING      100%.
FD  THREE-BEARS.
    LABEL RECORDS ARE STANDARD
    VALUE OF IDENTIFICATION IS "BEARS"
    DATA RECORDS ARE DADDY-BEAR, MUMMY-BEAR, BABY-BEAR.
01  DADDY-BEAR.
    02    HGT   70 INS.
    02    WGT   750 LBS.
    02    COLOR-OF-EYES   BLOODSHOT.
    02    DISPOSITION     UNBEARABLE.
01  MUMMY-BEAR.
    02    HGT   65 INS.
    02    WGT   700 LBS.
    02    COLOR-OF-EYES   BLUE.
    02    DISPOSITION     BEARABLE.
01  BABY-BEAR.
    02    HGT   40 INS.
    02    DISPOSITION     INFANTILE.
WORKING-STORAGE SECTION.
01 COTTAGE      PICTURE IS COZY.
    02    KITCHEN.
          03    TABLE     SIZE IS LARGE, VALUE IS 1.
          03    CHAIRS    SIZE IS MEDIUM, VALUE IS 3.
    02    PORRIDGE.
          03    KING-SIZE    OCCURS 1 TIME.
          03    QUEEN-SIZE   OCCURS 1 TIME.
          03    PRINCE-SIZE  OCCURS 1 TIME.
    02    DOOR  SIZE IS USUAL, VALUE IS OPEN.
    02    BEDROOM.
          03    BED.
                04 LARGE     OCCURS 1 TIME.
                04 MEDIUM    OCCURS 1 TIME.
                04 SMALL     OCCURS 1 TIME.
          03    WINDOW    SIZE IS SMALL, VALUE IS OPEN.
01  CORRECT-COTTAGE REDEFINES COTTAGE, VALUE IS SAME.
77 KING-SIZE-BED-SLEPT-IN    SIZE IS BIG, VALUE IS ROCK-BOTTOM.
77 QUEEN-SIZE-BED-SLEPT-IN   SIZE IS MEDIUM, VALUE IS DEPRESSED.
77 NO-PORRIDGE               SIZE IS SMALL, VALUE IS ZERO.
77 SIP                       SIZE IS LITTLE, VALUE IS "SSSLURP".
77 SLUMBERLAND               SIZE IS UNLIMITED, VALUE IS ZZZZZZZZZ.
CONSTANT SECTION.
01 COMMENT1     SIZE IS 36, VALUE IS "SOMEBODY HAS BEEN EATING MY PORRIDGE".
01 COMMENT2     SIZE IS 36, VALUE IS "SOMEBODY HAS BEEN SLEEPING IN MY BED".
PROCEDURE DIVISION.
FOREST SECTION.
START-OF-TALE.
    OPEN STORY. READ FOLLOWING.
FIRST-MOVE.
    MOVE GOLDILOCKS TO COTTAGE.
    IF DOOR IS CLOSED OR BEARS ARE GREATER THAN ZERO ALTER ENTER-GO3
          PROCEED TO HASTY-RETREAT.
ENTER-GOLDILOCKS.
    GO TO KITCHEN-SCENE.
KITCHEN-SCENE.
    IF PORRIDGE IS KING-SIZE, PERFORM TASTE-ROUTINE VARYING PORRIDGE-
          KING-SIZE BY 1 UNTIL PORRIDGE EQUALS PRINCE-SIZE
          OTHERWISE COMPUTE IF COTTAGE = CORRECT-COTTAGE GO TO BEDROOM-SCENE.
TASTE-ROUTINE.
    SUBTRACT SIP FROM PORRIDGE(KING-SIZE).
    SUBTRACT SIP FROM PORRIDGE(QUEEN-SIZE).
    SUBTRACT SIP FROM PORRIDGE(PRINCE-SIZE) GIVING NO-PORRIDGE.
BEDROOM-SCENE.
    MOVE GOLDILOCKS TO BEDROOM.
    ADD GOLDILOCKS TO BED(LARGE). DISPLAY "IT IS TOO HARD".
    SUBTRACT GOLDILOCKS FROM BED(LARGE) GIVING KING-SIZE-BED-SLEPT-IN.
    MOVE GOLDILOCKS TO BED(MEDIUM). DISPLAY "IT IS TOO SOFT".
    SUBTRACT GOLDILOCKS FROM BED(MEDIUM) GIVING OUEEN-SIZE-BED-SLEPT-IN.
    MOVE GOLDILOCKS TO BED(SMALL). DISPLAY "IT IS JUST RIGHT".
    ADD GOLDILOCKS TO SLUMBERLAND.
BEARS-RETURN.
    MOVE DADDY-BEAR, MUMMY-BEAR, BABY-BEAR TO KITCHEN.
    MOVE CORRESPONDING BEARS TO PORRIDGE.
    DISPLAY "DADDY BEAR ", COMMENT1.
    DISPLAY "MUMMY BEAR ", COMMENT1.
    DISPLAY "BABY BEAR ", COMMENT1, " AND EATEN IT ALL UP".
    MOVE BEARS TO BEDROOM.
BEARS-IN-BEDROOM.
    EXAMINE BEDS, REPLACING ALL GOLDILOCKS WITH BEARS.
    DISPLAY "DADDY BEAR ", COMMENT2.
    DISPLAY "MUMMY BEAR ", COMMENT2.
    DISPLAY "BABY BEAR ", COMMENT2, " AND HERE SHE IS".
HASTY-RETREAT.
    IF WINDOW IS OPEN EXIT GOLDILOCKS OTHERWISE MOVE GOLDILOCKS TO DOOR.
END-OF-TALE.
    CLOSE STORY, DISPLAY "WOULD YOU BELIEVE CINDERELLA IN PL/I?".
    STOP RUN.

Possibly-related posts:

Feeds

RSS  RSS feed

Categories

November 2024
M T W T F S S
 123
45678910
11121314151617
18192021222324
252627282930  

© Copyright Quentin Stafford-Fraser