ANDIY – the Art of Not Doing It Yourself

The best way to learn about car maintenance, I’ve always felt, is the way I did it in my youth: have cars cheap enough and elderly enough to need regular repairs… and not enough money to pay anyone else to fix them for you. My first three cars combined cost me less than my current phone, and, while it’s been a long time since I last stripped down a carburettor or replaced a leaf spring, it’s pleasing to know that, should the need arise again, I wouldn’t be completely unprepared. (Actually, now I come to think of it, the likelihood of either of those particular needs arising is becoming rather remote, but you get the idea.) These days, I tend to pay someone else to fix the car, but at least I can sometimes sound sufficiently knowledgable to dissuade them from ripping me off!

Often, that’s the pattern of growing up. You prove to yourself in your youth that yes, you can hang wallpaper or lay a patio, and you go to bed with a sense of achievement and the satisfaction of saving lots of money. As you get older, sometimes, the satisfaction comes from being able to afford a professional to do it, not, of course, because you are helpless or incompetent — you’ve proved twenty years ago that you aren’t — but just because you have more important things on your plate right now. And, actually, the professional did rather a good job, you have to admit…

When the job might be deemed to fall within your own profession, however, it’s more of a struggle. I have, for much of what might laughingly be called my career, earned my daily bread by writing software, or setting up companies in which other people can write software, and I greatly enjoy doing so.

So, while I have no desire even to attempt the creation of, say, another Photoshop or Excel — I have more important things on my plate, you understand — I have sometimes been reluctant to pay for simpler utilities because I feel that I really ought create the same functionality myself, just as soon as I get around to it.

Well, a couple of recent purchases for my Mac have convinced me that this is often a foolish attitude. If someone has gone to the trouble of creating a nicely polished utility that does just what you need, it may be worth a few bob to tide you over until you get around to building your own string-and-sealing-wax version. Who knows, you might find the (other) professionals did rather a good job…

Hazel

The first of these is Hazel, a utility that monitors folders in the background and performs actions on the files in them, based on certain rules. It’s the sort of thing any decent programmer might dismiss as being for those poor sods who can’t throw together a cron job, folder action and a bit of Perl when they need it. And actually, the idea of monitoring a filesystem as a way of initiating tasks is not, perhaps, a natural one for computer scientists, who would generally think of more efficient ways of kicking off a process. But it can be incredibly handy, starting with simple things like making sure your desktop or your downloads folder doesn’t become too cluttered: Hazel can file stuff away in neat subfolders (or the Trash) if it’s been hanging around there too long.

Then you listen to podcasts like this one and this one, and you realise that when you scan a document into whichever folder your software cares to dump it, Hazel rules can pick it up, send it to your OCR program if needed, rename it to include today’s date, and then, if the OCRed content includes the words ‘HSBC’ and ‘statement’, file it away in the appropriate folder where you keep your financial records. It’s like putting a bit of paper in your out-tray and having a secretary pick it up, classify it appropriately and file it away in your filing cabinet for you. Nice.

Hazel becomes especially powerful when combined with Dropbox, where things you drop in a folder on your laptop while away can be handled by Hazel on your desktop back at home. You need to get into the mindset, so go and find out what other people are doing with it, and you’ll start to realise what it can do for you.

Arq

Arq is an off-site backup program. There are plenty of those around – Carbonite, Crashplan, Backblaze, etc. But this has a couple of key differences. Firstly, it’s not a service, it’s an app purchase. You buy it once, and you use your own Amazon S3 account for the storage. The format in which it’s stored is fully documented. You can specify your backup budget — “I want my home directory on this machine not to cost me more than $8/month” — and Arq adjusts the amount of history it stores to keep within that figure. And if you have several machines, or if you provide IT support for your family and they’re all happy to backup to the same Amazon S3 account, you can check up from one machine on the backup status of the others. Finally, it’s just been updated to allow use of the new Amazon ‘Glacier’ storage. If you’re willing to put your data here (where it may take a few hours to get it back), it’s an incredibly cheap backup. If you have on-site backup as well and this is just for real emergencies, Glacier may be a good option. Either way, Arq handles it nicely.

Day One

For the last six years or so, I’ve been keeping a ‘journal’. (For younger readers, a journal, or diary, is like a Facebook or Twitter account but without any followers. Huh?) Mine’s not really a diary, so much as a cross between a lab notebook and a set of Post-It notes stuck on the screen: odd notes jotted down, URLs, booking references, phone numbers, snippets of code… but it’s very valuable and I search it regularly to jog my memory. I used to do this in a big folder full of text files, named according to the day concerned, and a set of keyboard macros and editor configurations to make it quick, easy and relatively nice looking. Then I discovered Day One, a journalling app for the Mac, iPhone and iPad (it keeps them all in sync), which makes it even quicker, easier and much nicer looking than my system. Its format is also reasonably straightforward, so I was able to import my six years of past notes, and can be pretty confident of getting everything out again if needed. It has a feature which I thought would be irritating but am surprised to find I rather like: it can pop up a little box in which you type your current activities or thoughts and then just hit Cmd-Enter to save. You can set it to do this a given number of times per day – three, in my case – and it’s a very low-overhead way to keep a track of what you were doing and when.

All of the above come highly recommended – and, for the record, I have no connection with any of them other than being a happy customer. Feel free to comment with thoughts on any of these, or indeed tales of any other ways in which you are becoming a skilled ANDIY practitioner…

GRUB, Ubuntu, and failed boots

Another geeky technical post. Ignore if it’s not your thing….

I wrote recently about the GRUB bootloader and how it can sometimes cause a remote or headless server not to come back online after, e.g. a kernel update.

This can happen in other situations: The configuration used on recent versions of Ubuntu is such that, if the system thinks that the last boot attempt failed, it stops it automatically booting into the same configuration again by cancelling the countdown timer, setting the number of seconds to -1, causing it to wait indefinitely at the boot menu until you decide on the best action to take. This is a sensible default, because a machine that goes into an infinite loop of reboots is doing nobody any good and puts a fair bit of strain on its own hardware.

Unfortunately, though, other things can trigger this behaviour. If you have a power fluctuation, for example, such that the machine restarts, gets part-way and then power-cycles again, you may find yourself with a machine that doesn’t come back online of its own accord.

On the most recent Ubuntu versions (12.04 with updates, and later) you can add a setting to /etc/default/grub:

GRUB_RECORDFAIL_TIMEOUT = 30

for a 30 second timeout after it has recorded a failure condition. You can use 0 if you don’t want it to pause and show the boot menu at all, but remember that it could then go into fairly fast repeated reboots if something really does go wrong.

On earlier versions, you’ll need to edit /etc/grub.d/00_header and find the line near the bottom, in the maketimeout()_ function, which sets

set timeout=-1

and change that to your preferred value.

In either case you’ll then need to run:

update-grub

to make your changes take effect.

Old men forget; yet all shall be forgot, but he’ll remember, with advantages…

Richard and I have been playing with flash cards as a way of learning things.

The great thing about an electronic implementation of the old ‘question on one side, answer on the other’ idea, is that it can make smart decisions about when and how frequently you should be presented with a particular card. Things you find easy to remember need only occasional repetition, while those which are new or more challenging need more regular viewing until they stick in your memory. When you see the answer, you just say whether or not you got it right, and how hard you found it.

Richard wrote a little while back about using this model to learn a reading he had been asked to give at a wedding. I’ve always liked learning poetry or bits of Shakespeare, but often find that large chunks will flow easily while there are one or two lines I always forget. Could this be the solution?

One of the popular flashcard systems out there is an Open Source one called Anki, created by Damien Elmes. It has Windows, Mac, Linux and Web clients, plus Android and iOS (though these don’t yet work on the latest version). And there are various ways you can get decks of cards in and out. The user interfaces are rather quirky, I find, and even the web sites can be confusing to navigate, but the underlying system works fine.

It’s easy to find plain-text versions online of most things I want to learn, so I wrote a little script called poem2anki which will take a text file containing lines of poetry (or prose!) and convert it into a file suitable for importing into Anki.

A question:

and the answer:

It will create these for all the lines in the poem, but you’ll quickly find you’re only tested on the ones you find difficult to answer.

You can find poem2anki here if wanted.

Configuring GRUB to boot the right kernel after an upgrade

This is based on a very useful post by a chap named Arie. The credit for much of the following goes to him: I’m just reposting a summary to make it more widely discoverable.

If you upgrade your Linux installation, which on Ubuntu/Debian you might do with something like this:

sudo aptitude update
sudo aptitude safe-upgrade

then you may find that your kernel (linux-image package) and GRUB bootloader configuration have been updated. In particular, it may not now, by default, automatically select the right boot menu option when you next reboot… which you’re probably just about to do because you’ve just upgraded everything. This makes some sense because, if the upgrade failed for some reason, you may want to select a different option from the boot menu and go back to the previous kernel.

Well, it makes sense, unless, of course, you aren’t sitting in front of the machine with a screen and keyboard. If you’re upgrading a remote server, you may find that it doesn’t come back up after your reboot. In my case, the server was very remote, and I was lucky to have someone on site to press the right keys.

So, before you reboot after a new kernel installation, you may want to configure GRUB to try the new kernel automatically, and if it fails, go back to the old one.

GRUB is configured by the file /boot/grub/grub.cfg, but on a modern Ubuntu, rather than editing this big and complex file directly, you edit a few settings in /etc/default/grub, and then run update-grub to rebuild it out of lots of separate bits.

So, look at the variables in /etc/default/grub and make sure the following are set:

GRUB_DEFAULT=saved
GRUB_TIMEOUT=2
GRUB_CMDLINE_LINUX_DEFAULT=”panic=5″

This tells GRUB to use the last-saved selection, to boot it automatically after 2 seconds, and tells all kernels that they should reboot after 5 seconds if they die completely. Then you need to configure which kernel is initially the one that’s ‘saved’. Set it to be the one you know works. e.g.:

sudo grub-set-default "Ubuntu, with Linux 3.2.0-29-generic"

(You can look at the kernel you’re currently running with uname -a and find the label used to select it in /boot/grub/grub.cfg. Try grep menuentry /boot/grub/grub.cfg, for example.)

Then tell GRUB to try the new kernel on the next reboot, e.g.:

sudo grub-reboot ”Ubuntu, with Linux 3.2.0-32-new”

(This doesn’t actually do the reboot)

Then save all of these things using:

sudo update-grub

And try rebooting.

If it works OK and you come up in the new kernel, set that to be the saved default for the future:

sudo grub-set-default "Ubuntu, with Linux 3.2.0-32-new”

DNG (Does Not Go)

I shoot almost all my photos in RAW format, which means that my important originals are in a variety of different formats: some Nikon, some Canon, some Panasonic. All of these are proprietary formats (though they’ve generally been reverse-engineered), and so not really ideal for long-term archiving.

It was for this reason that Adobe, some time ago, came up with the DNG — Digital Negative — format: an open standard intended for things like archiving. There are tools for converting most things into DNG, and the idea is that you’ll always be able to get your images out. Some cameras even save DNG as their native format. It’s a very good idea.

In theory.

The problem is that almost none of the tools I use support it. Adobe Lightroom does, of course, and makes it nice and easy to convert images automatically as you import them from your camera. But, once I have my image as a DNG, I find I can’t open it in Aperture, Preview, Acorn or even Photoshop CS3. I don’t get thumbnails in the Finder. I tried reverting to earlier, less efficient versions of the DNG format with fewer fancy options but it still didn’t help, unless you go back to really early variants, which can multiply the file size by three.

I could, of course, view my DNG files in Photoshop if I adopted the standard Adobe solution: pay hundreds of pounds to upgrade a product I paid hundreds of pounds for a little while ago. But I have the latest versions of other software products and none of them can open recent DNGs. Some of it may boil down to insufficient support in Apple’s underlying libraries. Whatever the reason, everything can open the closed, proprietary formats, whereas even Adobe’s DNG Converter can only convert to other forms of DNG. A few special tools like RPP can convert them to TIFFs, as long as you’re not using the latest DNG variants.

I’ve written about this before, but I repeat the experiment periodically to see if things have improved, because I really like the idea of storing things in an open raw format. But, sadly, at present, putting your files into DNG seems to imply locking yourself into expensive Adobe upgrades.

Meet Tomahawk

It’s not often I’d voluntarily share an advertisement here… but this caught my eye despite my normal banner-ad blindness. I think it’s very nicely done – and hard to ignore!

HBT

This morning, as I was getting ready for breakfast, I was contemplating what must, surely, be one of the finest phrases in the English language.

Not for its own intrinsic poetry, which I will grant you is minimal, nor indeed for the very pleasant experience it actually describes, but for all the additional images that it conjures up of comfort, safety, return from exciting adventures, and a sense that all is well with the world. Is there any other phrase that so encapsulates these ideals as the simple three words, hot buttered toast?

I think not.

The grass is always greener

The tactics of queueing, beautifully illustrated in a simple animated GIF:

Seemed appropriate for Status-Q!

Many thanks to Martin Kleppmann for the link.

Bayeswatch

OK — here's my deep thought for the day. Or it may not be deep, but I haven't finished my first coffee yet…

Is Hume's Maxim simply a restatement of Bayesian Inference?

I'm sure this is not a new idea, but I hadn't made the connection before. Hume's Maxim, which I've always liked, basically states that:

no testimony is sufficient to establish the existence of a miracle, unless it is more likely that the miracle occurred than that the testimony was false.

(I paraphrase slightly. More info here.)

Bayes' Rule is a little more complex, so hang while I just make some more coffee… Ok. Brace yourself. This won't hurt much.

It's the following equation:

where P ( X | Y ) means 'the probability of X given Y'. It’s often written using H and E for hypothesis and evidence.

It says that you can calculate the probability of a hypothesis (say, that a miracle occurred) given some piece of evidence (Mrs Jones reports having seen it).

The probability will depend on three things in the right-hand side of the equation:

  • P(H) – The probability of the miracle itself independent of any reports. (e.g. did the laws of physics change on this particular day for Mrs Jones?)
  • P(E) – The probability that the evidence would present itself independent of anything actually occurring. The combination of possibilities that Mrs Jones was either mistaken, deceived, deluded, fibbing or had some other motivation — possibly a perfectly good one — for coming up with such a report in any case.
  • P(E|H) – The likelihood that Mrs Jones would have reported a miracle, given that it actually occurred. Well, we’re not interested in miracles that happen quietly in the middle of a wood somewhere. We’re talking here about miracles for which there is testimony, so this term is probably 1 or close to it* and so can be removed from the equation in this case. David Hume didn’t include it. It’s worth noting, though, that if people are often abducted by aliens and neglect to mention it, you need to take that into account when they do.

If the likelihood of the laws of physics changing are greater than that Mrs Jones’s report is mistaken – i.e. that P ( H ) is greater than P ( E ) – then the probability that the miracle occurred, given her testimony, is greater than one – i.e. her testimony has established the veracity of the miracle.

So the statements are saying very similar things. Interestingly, they date from about the same period, too – late 18th century – and show an unusual overlap of two magesteria – philosphy and mathematics.

For such a simple equation, Bayes' Rule is incredibly powerful, though its revelations can sometimes be hard to grasp intuitively. You benefit from it every day, though, because it turns out to be phenomenally good, for example, at working out the likelihood that a given piece of email is spam. And a deep understanding of it, combined with a good marketing team, was the foundation of Autonomy, the Cambridge company sold to Hewlett-Packard last year for $10bn.

Sadly, it isn't used often enough for assessing the reliability of conspiracy theories and Daily Mail articles, perhaps because it doesn't tend to stick in one's mind.

So for normal, day-to-day understanding of the world around you, I recommend David Hume's version.

* Update: The probability that I will get statistics correct without help is about the same as the probability that Status-Q’s author is more intelligent than its readers. Thomas points out, quite rightly, that a probability can’t be greater than 1 – something that had been bothering me a bit too! P(E|H) is always less than P(E), so all this is really saying is that the likelihood of a miracle having occurred given Mrs Jones’s report is a bit less than the likelihood of a miracle having occurred. I may be pushing a few things too far: not least my understanding of stats. Hume is talking about the relative values of P(x), not the relationship between them. Can we draw anything more from this? A topic for discussion…!

Radio Days

Tonight I gave a talk about the Raspberry Pi to the Cambridge and District Amateur Radio Club. I pointed out that I didn't really belong in the group, unless you stretch the word 'Amateur' almost to breaking point, for I know almost nothing about radio.

But I understand the appeal. I made little crystal sets as a child and discovered that the aluminium double-glazing frame in my bedroom was an astonishingly good aerial, provided you wanted to listen to BBC Radio 2. Any other station was likely to be disappointing, but this it received so well that I could almost just connect high-impedance headphones to the frame without any other components. I read, with great jealousy, American novels where the kids had adventures involving walkie-talkies. Many years later, the opening up of CB radio and other bands here in the UK made such things a possibility, but for me, radio was something you received, not transmitted.

On a couple of occasions, visiting a friend or distant family member, I was taken out to a 'shack' in the back garden and allowed to watch and even participate in the strange ritual of starting up shelves of valve–powered equipment, which existed to connect a little speaker and microphone, via an enormous roof-mounted antenna, to people in far-flung parts of the country or even, if atmospheric conditions were right, of the globe.

What I realise now, of course, is what these guys were doing. These were the ones who weren't the passive consumers of radio technology like the rest of us. They had gone to the trouble to amass the equipment and expertise to be allowed to transmit as well as receive. And what were they doing with it?

Late at night, decades before Mr Zuckerberg was even born, they were disappearing into their strange dens to engage in global social networking.

 

Listening for the word

Since I turned on my Mac’s firewall a few months ago, I get occasional messages which cause me to wonder:

Now, I can understand why Word might want to make outgoing connections, to check for version updates, to print, etc. Anyone know why it would want to be listening on my network? It works just fine if you say no, but I am curious…

The other revealing thing, for me, was that I only saw this message today, which means I haven’t used Microsoft Word for more than four months. This time, as is usually the case, I only need to run it when I’m sent something by a lawyer.

You’ll believe a frogman can fly…

Oh, boy – I’d love to try one of these. What a great idea! Half-way between jet-ski and jet-pack…

More info here. Many thanks to Simon for the link…

© Copyright Quentin Stafford-Fraser