Cloud Control

While computing in the ‘cloud’ brings us a lot of good things, there’s one area in which it is often not very strong: longevity.

For me this is most apparent as I peruse the archives of my blog — in which, for example, none of the Google Video clips can now be played — or look back at tweets from a couple of years ago which often linked to things using the cli.gs URL-shortening service, many of whose links already no longer work.

And my last company, Camvine, used Google Apps for Business. The company has now gone, and so, with its Google account, have all the associated documents and emails. Thousands of them.

It’s easy, at the time, to think “I can link to this safely, or store my documents safely here, because Google isn’t about to vanish overnight”. Well, all of the service providers I’ve been tripping over in looking through my archives are still around. But for one reason or another, the links no longer work.

It’s obvious, but it’s worth repeating: Using, or linking to, someone else’s service, may be a good strategy for today, but don’t rely on it for anything you might want to access tomorrow. The only data I can be sure of is on servers I run (and backup) myself.

Customise app defaults with AppleScript

Nobody likes AppleScript. Well, almost nobody. It’s an attempt to make a programming language look like a natural language, which means that knowing what constitutes valid syntax in any given situation is almost impossible. I once suggested that what the world really needs is a Perl-to-Applescript translator, because Perl is a language that’s pretty easy to write but impossible to read, and AppleScript is easy to read but impossible to write. The syntax is a bit dependent on the apps with which you’re trying to interact, too, and the debugging options are exceedingly limited.

But the most annoying thing is that, on occasion, it’s exceedingly useful, and there aren’t really good alternatives for the kind of things it can do.

So just in case anyone out there is googling for this kind of thing, here’s how I made it change the default options on a dialog box that I use every day.

I’ve been inspired by David Sparks’s e-book Paperless, and my new-found fondness for the kind of things you can do with the Hazel utility, to get a better, more automated workflow for scanning in documents.

A key component, of course, is that you want them to be OCRed so that you can search for them, or search within them, later. I want something that does this automatically, or can be made to do it automatically, when a scan ends up in a folder on my disk, with minimal manual intervention. Good OCR programs are fairly costly on the Mac – Abbyy FineReader, at £79, is generally agreed to do the best OCR job, but the Mac version is not very scriptable. PDFpen, at £47, does a reasonable job and has better scriptability, and if I were starting now I’d probably use that.

But a while ago I splashed out on NeatWorks, which has good OCR, plays nicely with my wonderful ScanSnap scanner, and provides a complete filing system for my documents, with flexible metadata options. It’s a nice package. But the problem is that I no longer want a complete filing system for my documents – I want to do that myself.

So for the moment I’m using NeatWorks to capture my scans, OCR them, enter some metadata and then export them as PDFs to the folder where Hazel and other things take over. They typically stay in NeatWorks for about a minute.

OK – that was a long run-up to explain why I regularly – often several times a day, do File > Export… and get this dialog:

At this point I can almost just hit [Return], except for one problem: the default is to export all the items in the currently selected folders and I just want to export the thing I last scanned. So every time I do this, I have to switch from keyboard to mouse, click the little radio button by ‘Selected items only’ and then carry on.

AppleScript to the rescue! I used Automator to create a service that just applies to NeatWorks and runs the following AppleScript:

This runs ‘File > Export…’, clicks the appropriate radio button, and then clicks the Export… button.

Finally I used the Keyboard section of System Preferences to assign a keyboard shortcut to this service.

Now, I drop some paper into the scanner and press the button on the front. NeatWorks pops up and OCRs it. I type in a title, document date and any other keywords I fancy, then just hit my magic keystroke and check the name and folder before hitting return to save.

At that point, Hazel takes over and does something like “if this file was created by NeatWorks, and has a name containing the word ‘Telemarq’ and the word ‘receipt’, then file it away in the appropriate folder of my receipts directory with a suitably reformatted filename”.

Organisations don’t think, people do

“Can we arrange a time for a conference call with you?”, said the enthusiastic email that landed in my inbox last year from some company’s marketing department. “We're very excited to tell you about our new viral videos!”

To which my response, of course, was that if they were really viral, they wouldn't need to tell me about them!

 

I thought of this while watching Euan Semple's keynote from the State of The Net conference in June, which, in contrast, has a gentle, understated style yet includes some nice ideas that come from years of careful thinking about corporate communications, both internal and external.

Euan Semple is the author of Organizations Don’t Tweet, People Do.

Curiosity has Landed

On Aug 5th, the Curiosity rover landed on Mars. I hadn’t really absorbed, at the time, just what a technical achievement this was: not so much getting the thing to Mars, but landing it safely and ready to roll shortly after it hit the Martian atmosphere at 13,000 miles per hour. It weighs nearly a tonne. The Jet Propulsion Lab had, of course, made CGI simulations showing how the process would work, in advance of the landing, but in this brilliant piece of video editing they intercut it with footage of the control team on the ground celebrating its arrival. I suggest you turn the volume up and watch it full-screen.

Rose’s aunt used to work at JPL, and, when I visited many years ago, one of the directors made an offhand comment which basically amounted to a job offer. At the time, it was all very quiet, and though I was interested in the work, the elderly rows of SPARCstations tracking satellites didn’t grab me as particularly thrilling. Now, however, there can’t be many organisations that could put together a recruitment video like this!

Cut and paste comes to the Mac

Well, OK, the title’s a little misleading, but here’s a very handy hint for anyone running Lion or later. It’s to do with moving files.

Though the Finder has, for a very long time, supported the copying and pasting of files from one folder to another, it has never allowed cut and paste. I could never fathom why something so simple wasn’t in there until I realised that, actually, there is a problem with implementing the concept cleanly: normally, when you cut things, they disappear. What happens if you cut a file but never paste it? Especially accidentally? (What happens on Windows? I forget…)

Still, this was an annoying omission, particularly if you’re used to Windows, or if you have a small display without much space for dragging things about.

Well, now you can do it. Instead of the normal cut & paste keystrokes (Cmd-X, Cmd-V) you do a copy and a kind of alternate paste (Cmd-C, Alt-Cmd-V). Intuitive? No. But I guess it makes a kind of sense.

Thanks to the excellent David Sparks for the hint.

Obligatory Mac Utilities

There are a handful of utilities on my Mac that I use all day, every day. I’ve written about most of them before over the years — try the search box on the right — but since people liked some of my past posts about favourite iOS apps, I thought I’d gather these into a quick list here.

  • Dropbox
    But then, you could have guessed that one. It’s on everybody’s list. ’Nuff said.
  • LaunchBar
    Unless your needs are very minimal, you need something to launch applications and utilities without having to burrow through folders in the Finder. Some people just use Spotlight, but there are quicker and more powerful options. In the past, I tried most of them but settled on LaunchBar and, despite occasional experiments with others, and though I use a tiny fraction of its facilities, I’m still using it 10 years after first installing it.
  • A clipboard history utility
    These give you the ability to store more than one thing in your clipboard. Cut, Copy and Paste will work as expected, but you have the option to go back and paste the thing-before-last that you copied. Or things from yesterday. This is one of those habits that you may never get into; it took me a long time to get used to the fact that I could copy a quote from a web page, and the author’s name, and the URL, and then go to my blog and compose a post with all the information at hand. But it’s very liberating: you can stick stuff in the clipboard without worrying about what it’s overwriting, and the history becomes a sort of temporary scratch space, like the back of an old envelope on your desk, in which you can put all sorts of short-term stuff. There are several utilities about, but LaunchBar comes with a good one built-in, so I just use that. Trust me, train yourself to do this and make sure the keyboard shortcut is the same on all your machines. I use alt-cmd-L, which is quick to type, and has become as instinctive as the cut, copy & paste keystrokes.
  • TextExpander
    This lets you type just a few characters, in almost any app, and have them converted instantly into a much larger chunk of text, optionally with lots of clever extra features and options. If you’re me, for example, it doesn’t take long to discover that ‘Quentin Stafford-Fraser’, what with its punctuation and capitals, can be a tedious thing to type many times a day, and to type ‘qqsf’ and have it automatically converted is a much better option. My own email addresses, which these days are often my usernames on various services, each have their own three-letter abbreviation, and so forth.
    I first used this back when it was called Textpander, and was free, and when that changed, for a long time I resisted paying $35 for something that just typed a few keystrokes for me. Until, that is, I realised just how many keystrokes it had typed for me — over 38,000 on this machine, for example (it keeps a record). I’m not sure over what time period that is, but that’s just on one of my machines, and I’m not a heavy user.
  • SuperDuper
    It clones your disk. Efficiently, flexibly and reliably, and if you’re cloning your main system disk, it will make the result bootable. I don’t use it often, but I do it without fail every single time I’m about to do a major operating system upgrade, so it’s had a lot of use recently!
  • Scrivener
    Not a utility, this one, but a substantial application. They used to say of Emacs, that it was not so much a text editor, more a way of life. Well, Scrivener inspires almost religious devotion amongst its enthusiasts. It is a word processor designed for professional writers — a category into which I certainly do not fall — but I’ve become rather fond of it for gathering together information associated with any project, and if I were to write anything substantial, it’s certainly what I’d use. I could write a great deal about Scrivener, but others have done so in numerous places, so I’ll just mark it as a favourite, and encourage you to investigate for yourself. There are plenty of screencasts, podcasts, books and many other resources to tell you about it and why it’s so yummy.
  • 1Password
    This is on all my devices, and in the toolbar of all my browsers on my Macs. How do people live without this (or something like it)? If you live without it by using the same password on many websites, then I hope you’re spending the money on medication instead to help you sleep at night.

Serendipity as a plugin widget

It would be terribly presumptuous to think that my readers, not satisfied with whatever I might burble about today, might want to go on to explore the Status-Q archives…

However, the fact remains that there are over 2200 posts here now, and I certainly can’t remember everything I’ve written, so it’s fun for me, at least, to browse a bit. The ‘related posts’ at the bottom of each entry’s page often pop up things I’d completely forgotten, but now I’ve added a ‘From the archive’ box on the right: a completely random selection of five posts from the last decade, updated every few minutes.

Go on – have a browse. Whatever you find is bound to be more interesting than what you’re reading now!

The Business of Bond

John Gruber points out this very nice visualisation comparing the budgets and box-office returns of the different Bond movies.

Make sure you check the ‘adjust for inflation’ switch.

We’ve always rather liked Timothy Dalton, who is a Real Actor, but it’s clear that the public in general didn’t share our enthusiasm, perhaps for him or perhaps for the plots of those movies, which only gave a four- or five-fold return on investment. Still pretty good when compared with your average savings account, but Diamonds are Forever paid back its investors sixteen times over. Ah well, watch his excellent Mr Rochester in the best production of Jane Eyre, instead…

This is also a fine example of the kind of web design that would have been inconceivable without Flash not long ago…

Olympian detachment?

Well, my tweets last night were mostly either bemused or rather negative, so I should emphasise that there were bits of the Olympic opening ceremony I thought were rather good.

I liked the levers of the industrial revolution hoisting enormous smoking chimneys into the sky, though one gets the impression that nothing good came from this Saruman-style destruction except the forging of five giant gold rings to rule us all. A pleasing effect, but some other industrial achievements might have been nice: Stephenson’s Rocket, perhaps? The spectre of Voldemort hovering over children in hospital beds, until chased away by Mary Poppins, was quirky but amusing, though perhaps he was really hovering over the NHS? It, like so much of the ceremony, must have been completely bewildering to hundreds of millions of viewers.

I cringed at some of the inevitable political correctness, was proud of the music we used to produce until about 20 years ago, was pleased by Her Majesty’s involvement in the Bond escapade and felt sorry for her obvious boredom at having to sit through the rest of it. It’s good that Tim Berners-Lee finally gets appropriate global recognition, tweeting ‘This is for everyone’ from the middle of the arena; given his natural humility it must have been a challenge to get him to agree. And the cauldron was, indeed, very pretty.

To give a true history of modern Britain, I thought, we should have had a huge influx of Polish people at the end! And then I realised that they had probably been there all along, behind the scenes, making everything work. And work it did; it was certainly an impressive technical achievement, and it looks very good in the BBC’s six-minute edited highlights.

Then the athletes came in, and many of them looked like rowdy drunken yobs coming out of… well… a sporting event. Still, I suppose that’s something else that the world knows us for.

Now, I know I’m not the target audience for this stuff; I’ve made my feelings clear about the financial outrage that is the Olympics. For the same money we could have given a shiny new MacBook Pro to every schoolchild in the country. Or employed 1000 teachers for 400 years. Or… well… take your pick of better investments. So I tried to divorce my feelings about the ceremony from its association with the bigger picture. And as my friend Jeff Jarvis put it, to set the context for his tweets last night: “I cannot abide opening ceremonies or folk dances”.

And I’m also very out of touch with popular culture – I recognised about half a dozen faces last night: the Queen, Rowan Atkinson, Sir Tim B-L, Daniel Craig and Kenneth Branagh. OK, five. But it would have been six if I’d stayed awake long enough to see Paul McCartney. So I imagine there were probably lots of sports personalities, soap-opera stars, rap ‘musicians’ and winners of X-Factor that might have been recognisable to others.

So it’s probably better to rely on others’ commentary than mine. I liked:

Allesandra Stanley in the New York Times:

It’s hard to imagine any other nation willing to make so much fun of itself on a global stage, in front of as many as a billion viewers. It takes nerve to look silly; the cheesy, kaleidoscopic history lesson that took Britain through its past, from pasture through the workhouses and smoke stacks of the Industrial Revolution to World War I and, of course, “Sgt. Pepper’s Lonely Hearts Club Band,” was like a Bollywood version of a sixth-grade play.

But bad taste is also a part of the British heritage. The imagery mixed the glory of a royal Jubilee with the grottiness of a Manchester pub-crawl. Britain offered a display of humor and humbleness that can only stem from a deep-rooted sense of superiority.

The NBC anchors Matt Lauer and Meredith Vieira did their best to get in the spirit of British nuttiness, but at times their energy flagged, and their bewilderment became obvious. After a hospital sketch that morphed into a children’s nightmare — and a giant fake baby floating on a bed — Lauer said, “I don’t know whether that’s cute or creepy.”

The whole show veered from cute to creepy and from familiar to baffling, including a pop music tribute to Tim Berners-Lee, the inventor of the World Wide Web. Most of all, it showed a love of movies that celebrate British eccentricity. “Isles of Wonder” seemed most inspired by a scene from the movie “Love Actually,” in which Hugh Grant, playing the prime minister, explains that Britain is still a great nation because it is “the country of Shakespeare, Churchill, the Beatles, Sean Connery, Harry Potter, David Beckham’s right foot.”

Andrew Gilligan in The Telegraph:

Some of the rest was bitty and disjointed; the sub-mobile-phone advert style of the digital section was particularly weak. It was more political than I expected. Voldemort loomed over the NHS. Tonight marked perhaps its final transformation from a healthcare system into a religion. Dancers made up the CND symbol. The Royal Family looked bored, but the new Right-On Royal Family – Doreen Lawrence and Shami Chakrabarti – got to carry the Olympic flag.
The NHS segment in particular underlined how surprisingly parochial this ceremony was. The idea of the Health Service as a beacon for the world is, bluntly, a national self-delusion. Most other Western European countries have better state healthcare systems – and healthier people – than we do. Does the average Chinese person even know what the letters stand for?

But I suppose the whole Olympics is in a broader sense parochial. Three weeks ago, I was in Libya witnessing that country’s first free election in sixty years: an end, or at least a beginning of the end, to decades of madness and tyranny which killed tens of thousands and blighted the lives of millions. To borrow the words of tonight’s over-excited TV commentators, that really was an inspirational and historic moment. Tonight, by contrast, was just a show.

Light humour

One of my favourite podcasts at present is The Skeptics Guide to the Universe. Highly recommended, if you don’t know it. I liked this quote from a recent episode where they were discussing the Higgs particle:

“I read this great book about antigravity. I couldn’t put it down..”

How to fix the opening ceremony…

The Telegraph reckons that it has inside information on what’s going to be included in the top-secret opening ceremony of the Olympics.

A stage backdrop of hills, streams, meadows and a thatched cottage will evoke Britain’s rural past. The landscape will be dotted with live animals, including 12 horses, three cows, 70 sheep, three sheepdogs and a horse-drawn plough, along with milkmaids, picnicking families, an Edwardian village cricket team in flannels, caps and braces, and people dancing around maypoles.

At one end of the arena will be a recreation of Glastonbury Tor, with an oak tree on top and a festival “mosh pit” at its foot. At the other end will be a space for crowds recreating the Last Night of the Proms.

It gets better…

…a third “act” of the ceremony will look at the post-war transformation of Britain, with models of Big Ben and other London landmarks, and a parade of dancing nurses and ancillary staff pushing hospital beds to represent the NHS and the Welfare State.

Oh good. Can’t wait for the dancing nurses.

…alongside 12,000 dancers, drummers, skateboarders, acrobats, and actors dressed as British historical figures, such as Emmeline Pankhurst, the suffragette, and the Caribbean migrants who arrived on the Empire Windrush in 1948…

I hope Emmeline Pankhurst will actually be on a skateboard.

I was just thinking that this could be a national embarrassment for which even the BBC’s reporting of the Jubilee River Pageant was insufficient preparation, when I suddenly realised that they must have a secret plan, because there is one way in which this could all be saved, could be put in the right context, and could turn the whole thing into a most enjoyable evening’s entertainment:

Get Terry Wogan to do the commentary.

Protecting your IPv6 world

This is the third and (probably) final post in my series about enabling IPv6 on your home network. The first, Banish Mavis and Connect to the Future, explains the basics and why I think this stuff is really important. The second, Tunnelling your way to the future, tells you how, if you have a Linux box on your network, you can give full IPv6 connectivity to all your machines, even if your ISP doesn’t support it yet.

This is all great fun, but at the end of the last article I pointed out that there were also security considerations associated with making all of your machines accessible to the ravages of the outside world, and it was only something you should do if they had a modern operating system, up-to-date security patches, and were running a firewall. This may, understandably, make you feel a little nervous, so in this final section I’ll show you how to implement a simple firewall on the machine that’s acting as your IPv6 gateway, so you can choose what traffic you want to let in, from where, and to which machines.

This is still, by the way, very much better than the old NAT-based world of IPv4, because you are separating policy from capability: you can choose to allow SSH and HTTPS traffic to two different web servers inside your network, FTP access to one of them, and plain HTTP to a webcam, and give them all proper DNS addresses, while still blocking incoming connections to anything else.

As a simple example, I’m going to explain how to set up a firewall that allows incoming HTTP, HTTPS and SSH connections over IPv6 but blocks everything else, so you’re not at risk when, say, enabling file sharing between your home computers.

Turning the IPtables with Shorewall

If you set up your system as per my description in the previous article, your linux machine is connecting to an IPv6 tunnel provider, receiving the IPv6 packets for your network on a virtual interface we called he-ipv6 and routing them out to your network over the normal ethernet interface, eth0.

Now, the way in which the Linux kernel routes data between interfaces is controlled by an internal table of rules, which is used to examine each packet and work out whether and how to send it on its way. These tables can be manipulated from the command line through a utility called iptables, or ip6tables for the IPv6 variant, and you could use these to set up the rules you need to let some stuff in and keep other stuff out.

It’s all very powerful, but unless you are part-robot yourself, trying to do much of this will make your brain hurt. This is the assembly-language equivalent of network configuration; you can make it do anything, and a very large number of the options you could choose will be the wrong ones. Eventually, you will get everything nicely set up, and all will be well until 18 months down the line when you have to change something and you have to remember what all those arcane lines of configuration were about.

So we’re going to use a system called Shorewall, which has a nicer way to describe what you want in a few simple configuration files, and then arranges all that iptables stuff for you behind the scenes.

Installing and configuring Shorewall6

We want the IPv6 version of Shorewall, so, assuming you’re on a Debian/Ubuntu-type system, you can install it with:

 sudo apt-get install shorewall6

This will install both shorewall (the IPv4 version) and shorewall6, and will create configuration directories /etc/shorewall and /etc/shorewall6.

It will also put files to control the basic startup of each in /etc/defaults/shorewall and /etc/defaults/shorewall6. For our purposes, we can ignore the IPv4 version. There’s a line in each of these files that looks like:

startup=0

The firewall will not start up unless you change the 0 to 1, so do this in /etc/defaults/shorewall6 and leave the other one alone. The only other change you need to make to the defaults is to look for a line in /etc/shorewall6/shorewall6.conf that says:

IP_FORWARDING=Off

and change it to

IP_FORWARDING=On

otherwise the machine will stop operating as a router.

OK. Now we’re going to create four configuration files. Four?, I hear you squeak. Yes, four, but trust me, they’re wonderfully brief and straightforward. We’re just going to create them from scratch because they’re so simple.

Here’s how it works:

  • We’re going to define three ‘zones’: net, which represents the outside world, fw, which represents the firewall itself, and lan, which represents your local network.
  • We’re going to say which network interfaces connect to each zone.
  • We’re going to define the basic high-level policy of how data can flow between zones.
  • Finally, we define some rules which specify in more detail the connections that are and aren’t allowed.

These are done in simple text files called:

/etc/shorewall6/zones
/etc/shorewall6/interfaces
/etc/shorewall6/policy
/etc/shorewall6/rules

See? At least the naming is nice and logical. There are man pages describing these in detail if you want to know more – just run man shorewall6-zones, man shorewall6-interfaces, etc. I’m certainly not an expert, having tried this for the first time today, but the following files work for me.

/etc/shorewall6/zones

This is just a list of zone names and types:

fw     firewall
net  ipv6
lan  ipv6

/etc/shorewall6/interfaces

Specify where these zones live:

#ZONE  INTERFACE  ANYCAST  OPTIONS
lan     eth0       -        tcpflags
net     he-ipv6    -        tcpflags

/etc/shorewall6/policy

By default, data from the firewall or LAN destined for the outside world is accepted. Data from the outside world is dropped. Everything else is rejected.

#SOURCE DEST    POLICY

fw      net     ACCEPT
lan     net     ACCEPT
net     all     DROP
all     all     REJECT

At this point, you can start up the firewall to test it:

  /etc/init.d/shorewall6 start

And you should then find that you can make outgoing connections, for example to ipv6.statusq.org, but not incoming ones. If you don’t have easy access to an external IPv6-capable machine, try using a service like this one on mebsd.com to ping one of your internal IPv6 addresses. It should fail.

/etc/shorewall6/rules

Finally, in this file, let’s define some rules to let in ping, http, https and ssh.

# These rules apply to all new connections
SECTION NEW    

#ACTION     SOURCE      DEST      PROTO      DEST PORT    

# Allow ping6 from outside world to firewall or LAN
ACCEPT      net         fw        ipv6-icmp
ACCEPT      net         lan       ipv6-icmp    

# Allow http, https and ssh from outside world to firewall or LAN machines
ACCEPT      net         fw        tcp        80,443,22
ACCEPT      net         lan       tcp        80,443,22

That’s it! Restart your firewall:

/etc/init.d/shorewall6 restart

And you should now find you can ping, ssh or get web pages from machines inside your network, but all other incoming connections will be blocked. If you want to allow access to a particular machine instead of all of them, change the rule line to include both the zone and the IP address:

ACCEPT  net    lan:2001:470:1f39:1824:fa1e:dfef::d5dc   tcp    80,443,22

Well done! There’s an amazing amount more you can do with Shorewall – have a look at the man pages – but that at least should give you the basics and let you sleep at night!

Feedback and suggestions welcome, as always: I’m a novice at this too.

© Copyright Quentin Stafford-Fraser