Status-Q

To the London LinuxWorld today, which seemed miniscule after the San Francisco one only a few weeks ago. I was surprised at just how small it was, given the importance of Linux to the UK, and the importance of the UK to Linux, but it was still interesting.

On the train I was listening to podcasts, as I often do now while travelling. That has been the biggest impact of the iPod for me personally: the fact that I no longer consider waiting time and travelling time as wasted time, even if I’m driving or walking and couldn’t easily read a book. I spend much more time on my iPod listening to speech than I do to music…

Anyway, one of the interviews I listened to was with Jeremy Allison, a key developer of Samba. For those who don’t know, Samba is free software which shares files and printers over a network using Microsoft’s protocols, meaning that Windows machines can communicate with Macs, Linux & Unix boxes and a variety of other devices that use Samba under the hood. It’s a very important piece of software, and I’ve been using it for about 11 years.

Of course, Microsoft don’t publish the details of their protocols. If they had their way, then Windows machines would only be able to talk to other Windows machines. So Andrew Tridgell, the creator of Samba, has to work out what they’re doing through a variety of cunning techniques. He wrote a nice article about how he does it. I particularly liked his description of what he calls ‘The French Cafe Technique’:

Imagine you wanted to learn French, and there were no books, courses etc
available to teach you. You might decide to learn by flying to France and
sitting in a French Cafe and just listening to the conversations around you.
You take copious notes on what the customers say to the waiter and what food
arrives. That way you eventually learn the words for “bread”, “coffee”
etc.

We use the same technique to learn about protocol additions that Microsoft
makes. We use a network sniffer to listen in on conversations between
Microsoft clients and servers and over time we learn the “words” for “file
size”, “datestamp” as we observe what is sent for each query.

Now one problem with the “French Cafe” technique is that you can only learn
words that the customers use. What if you want to learn other words? Say for
example you want to learn to swear in French? You would try ordering something
at the cafe, then stepping on the waiters toe or poking him in the eye when he
gives you your order. As you are being kicked out you take copious notes on
the words he uses.

The equivalent of “swear words” in a network protocol are “error packets”.
When implementing Samba we need to know how to respond to error conditions. To
work this out we write a program that deliberately accesses a file that
doesn’t exist, or uses a buffer that is too small or accesses a file we don’t
own. Then we watch what error code is returned for each condition, and take
notes.

Warning – geeky post ahead… I’ve been doing some coding this week…

Anyone who’s done any programming, at least if it’s outside the limited confines of an integrated development environment, will have come across the make utility, which was developed nearly 30 years ago at Bell Labs.

Make lets you list which bits of a program depend on which other bits, so that when you make a change, say, to one of your source code files, you can just type ‘make’ and the bits which need to be updated as a result all get rebuilt automatically.

People use it for other things, too; many years ago I had to produce Rose’s PhD thesis through a rather complicated process which started with floppies from a dedicated Panasonic wordprocessor, ran through a C program I wrote to decode the disk format, a Perl script to convert the files to LaTeX, and then Latex itself and finally dvi2ps to get the Postscript output for printing! Each stage generated different files, updated tables of contents etc, and when she fixed a typo on her Panasonic, I needed to ensure that it was propagated through the entire pipeline and made it into print. Make was good for that.

But anyone who’s built a project of any size will also know that make is far from perfect. It really hasn’t evolved much in its thirty years and the syntax, while elegant for very small projects, becomes unintelligible for large ones.

Here’s a small segment of a Makefile I was writing last week:

define PROGRAM_template
$(1): $$($(patsubst %,%_OBJS, $(notdir $(1)))) \\
        $$($(patsubst %,%_LIBS, $(notdir $(1))))
     g++ -o  $$@ $$^ $$($(patsubst %,%_LDFLAGS, $(notdir $(1)))) $(LDFLAGS)
all:: $(1)
$(foreach obj, $($(notdir $(1)_OBJS)), \\
     $(eval $(call OBJECT_template, $(obj),$(notdir $(1)) )) )
endef

# Build every program defined in PROGS
$(foreach prog,$(PROGS),$(eval $(call PROGRAM_template, $(prog))))

Don’t bother trying to understand this. The point is that it’s pretty arcane stuff and I wasn’t trying to do anything too sophisticated here.

If you’ve written Makefiles before you probably know roughly what’s going on, but do you know exactly what’s going on? Would you have got the right number of $ signs in the right places? Could you say why $$^ is in here and not $$< ? Why I have to call things and then eval them? Then try and imagine what it’s like for somebody seeing a Makefile for the first time!

And here’s the worst bit: for this to work at all, the gap at the beginning of the ‘g++’ line must be a tab, not spaces. So simple code that looks perfectly correct may not actually work when you try to run it. It’s a nightmare.

So last week I decided that using make in the 21st century was probably ridiculous, and it was time to search for alternatives.

Continue Reading