Tag Archives: backups

Let’s ask Quentin (RIP)!

I’ve been contemplating how to achieve immortality.  I’m sure you often do the same thing over breakfast on a sunny morning.

It has often occurred to me that, since so much of my output is in digital form, it may vanish without a trace once I’m gone, since nobody will be paying the hosting fees. Had I written more things that had made it into print, they might at least have lingered in the dark recesses of a library somewhere for a rather longer period.  Perhaps even gather dust on one or two people’s bookshelves.  Probably nobody would ever read them, but it would be comforting to know that they were there!

In reality, of course, digital data should last a lot longer, as long as it’s maintained.  If I were really wealthy and cared enough about this vanity project, I would leave behind an invested sum big enough to pay for web hosting in perpetuity plus one day per year of an IT consultant’s time to update the formats, check the backups, etc.

Fortunately, though, I have some hope that my 20+ years of blog posts won’t just vanish into the ether when Rose forgets to pay the web hosting bill after I’m gone, partly because there are periodic snapshots on the wonderful Internet Archive. (Here’s what Status-Q looked like in early 2001.)  

Brewster Kahle, the man behind the Archive, was good enough back in 2005 to give me a tour of their headquarters, which was then located in the Presidio of San Francisco. Brewster’s an inspiring guy doing important work, and a much better use for my hypothetical legacy would be to leave it to them.    I wonder if they would guarantee, in exchange, to keep my memory alive, in much the same way that donors to religious organisations used to get prayers said in perpetuity for their departed souls….

But then I started wondering about the next stage.

If you were to train an AI system on all of my blog posts, YouTube videos, academic papers, podcast & media interviews, etc… how convincingly could you get it to respond to questions in the way that I would have done?  Perhaps a deepfake video character could even give future interviews on my behalf?  I can’t quite decide whether that’s exciting, or thoroughly creepy.

But I tell you what… I do think it’s inevitable.  

Perhaps not for me: I imagine that when I’m gone a few friends will shed a quiet tear and everyone else will breathe a huge sigh of relief and switch the servers off.   But for others; those more prolific, more wise, more entertaining, I think this is bound to happen.  You will be able to ask questions of Mother Theresa, or Christopher Hitchens, or the Dalai Lama, or Warren Buffet.  You’ll be able to get Handel to compose your wedding march, and Peter Ustinov to speak at the reception afterwards.  And for a bit of spiritual advice, you could always ask God. Or a ChatGPT engine trained exclusively on his revelations to mankind from whichever source you prefer them.

Today’s systems would, of course, do a very fallible job, but what will the AI systems be like in 100 years’ time?   That will only help you, of course, if they still have access to your data, in non-proprietary, open, standard formats.  In the past, if you had sufficient wealth, you might have chosen to spend it on Cryonics.   I can’t help feeling that to achieve immortality now, a better bet would be to spend it on good, globally-accessible backups of your data.



The Fragile Free Cloud

Here’s a timely reminder, if one were needed, that you should never assume anything you store online is going to be there for very long, unless it’s on a system (a) that you are paying for and ideally (b) that you run or manage.

Flickr has announced that it’s going to start removing photos from its free accounts: everyone can still have 1,000 images, but that’s much less storage than they offered for free in the past. If you have more than that, they’ll start deleting the older ones first. I starting uploading things to Flickr about 13 or 14 years ago, so 90% of my 10,000 Flickr images will vanish over the next few months.

Most of the Snapchat/Instagram generation are probably not interested in anything that happened more than 1000 images ago! But people who have used Flickr for archiving the first pictures of their children or grandchildren may be in for a surprise. The name ‘Flickr’ might have a certain irony to it…

Now, this is a perfectly reasonable thing for the company to do, and there are several ways you can deal with it: you can start paying for your account, you can download your images if you don’t have local copies, or you can migrate them over to Smugmug (who now own Flickr). But only the first of those options will keep your photos nicely arranged in their albums, and, more importantly, will preserve your image URLs, so I imagine there will be a very large number of pages around the world with Flickr-shaped holes in them where an image used to be. Whichever option you choose, do it before the end of the year.

Now, I’ve been a fan of Flickr for a long time, and paid for an account for about a decade — it’s a good service and reasonably priced — but I switched to Smugmug a few years back because it was a better fit for my occasional bits of professional work. I don’t mind paying for one photo storage service, but I’d rather not pay for two, especially from the same company! So my photo archive has been copied to SmugMug, and I’ll probably need to write a bit of code to go through my blog and fix Flickr URLs. The album arrangements, though, will vanish if I take this approach.

Anyway, the moral of the story is this: You need to look after your own data. Don’t assume that anyone else will do it for you, on a long-term basis, and especially if you’re not paying for the service! In particular, don’t assume that any URL is going to continue to work in the future unless it’s on a domain that you control and manage.

And lastly…

Remember that this will almost certainly also happen at some point to the pages you have on Facebook, the images you have on Instagram and the videos you have on YouTube. Don’t assume that a service will continue indefinitely because the company is large or because it has a model based on advertising revenue. I had stuff on Google Video too…

Update: Thanks to John for pointing me at Thomas Hawk’s post explaining that Flickr’s action is a good thing; yes, I agree overall!

Disk Risk

Mmm. I seem to have had a lot of hard drive failures recently – Seagate drives, mostly, though, to be fair, the majority of my drives are Seagate just because my favourite supplier happens to like them, so I would expect see more failures there. The last one, though, is just 18 months old and has started making ominous clicking noises. They don’t make ’em like they used to. Stuff I’ve read online tends to suggest that it’s hard to assign blame to particular drive manufacturers, but particular models do tend to have rather different failure rates.

I do, I realise, have rather a lot of hard disks. I have three 4-bay Drobo enclosures, for a start, so that’s 12 drives even before I start adding on the miscellaneous backup disks, TV-recording disks, etc. Not to mention the internal ones in all our various machines. There must be 20-25 hard disks around here, and even though manufacturers’ specs talk about a <1% annual failure rate, studies tend to suggest that real-world figures are rather higher. One of the biggest studies, done by Google a few years ago, showed failure rates of 1.7% in the first year, rising to over 8% in the third year.

Yes, many of my drives are about that age, so if I really have 25 of them, I guess I should expect one to die every six months or so. Bother.

This suggests to me that money spent on things like my Drobo enclosures is worthwhile, because, though they are pricey, especially once you’ve filled them up with drives, any single drive failure is unlikely to be catastrophic – as disks die, you just replace them with whatever size is currently in vogue. My main Drobo currently has two 2TB drives, one 1.5TB, and a 1TB. There are those, I know, who have had less positive experiences with some Drobo kit – I found a DroboShare networking add-on to be decidedly wobbly at a past company – but in the simple use case of a Drobo plugged into a computer, I’ve been very happy and have replaced several drives without ever losing data.

The other thing that the Google study found was a strong correlation between when disks start reporting errors (which they can do using the S.M.A.R.T technology built into modern drives) and a failure soon afterwards. It’s worth, therefore, having something that checks the S.M.A.R.T status and lets you know about issues as soon as they are reported, even if the drive is still apparently working OK. On the Mac, Disk Utility can tell you about issues, but only when you go and look, so I use SMARTreporter to give more regular checks.

OK, things are getting better. There is another issue, though.

On the Mac, at least, most external drives are connected by USB or Firewire, and in general S.M.A.R.T information is not read through those interfaces – if you look in Disk Utility, you’ll see it’s ‘Unavailable’. More sophisticated enclosures like the Drobo will check the S.M.A.R.T status themselves and warn you when things look dubious, but your average USB-connected backup drive may give you no such warnings.

So I was interested to discover this kernel driver project which enhances the standard OSX USB and FireWire drivers to make S.M.A.R.T available for a lot more interfaces. (Download v0.5 here). I’ll try it on my Media Mac Mini, which has three external drives, and see how it goes…

© Copyright Quentin Stafford-Fraser