Dispatches from Andyland

There was a post that used to be here about boston.com hyperlinking on its web pages. I pulled it down after I noticed it getting linked to from a few pages. Seeing people link to it made me think twice about what I said. It made some speculations on other peoples thoughts and actions that I really can’t attest to.

Yes, Boston.com’s CMS has a process that looks for text strings that look URL-ish or email-ish and makes them clicky. It runs over many of the feeds that boston.com has, not just the Boston Globe feed. The default is on, to make the URL clicky, but it easily be turned off if there is a URL that they don’t want linked. There probably isn’t a convenient way differentiating a good link from a bad one. (Well, I did think of one. We could purchase one of those net-nanny-style content filtering firewalls, and the CMS could try to fetch the URL before making it into a link. The problem is that the cost of the software and the man-hours of integrating it with the CMS would probably defeat any cost savings over checking the links by hand.)

Many people discuss trends in terms of “the tipping point.” The point where a smaller movement becomes a larger, irreversable trend. It sort of irks me when I hear it because most of the people who use it are absolutely unaware of where it came from and its original meaning.

The term was first coined in the early 1960s by Morton Grodzins, when studying the racial integration of neighborhoods. White families stayed in their homes as long as the ratio between white and black households remained relatively high. At a certian point though, as the ratio changed, the white families would leave en-masse in a white flight.

It seems to me that the current meaning of tipping point has strayed from the original. The original seemed to imply when irrational xenophobia took over, the current seems to be a simple marketing analysis. In usupuring the word, I’m afraid it diminishes the fact that the racial issues that were originally being discussed still exist.

In a recent article titled The Long Snout, Tim O’Reilly uses the term  Tipping Point for what seems to be a simple financial equation. Book publishing has a high fixed cost for a print run of books, and high variable cost for warehousing and shipping of small quantities over time. But when a book is a runaway hit, they aren’t in the warehouses, so the warehousing costs fall. More copies are ordered at once, so shipping costs fall, and the chance of returns shrink. Books that are hits are real hits. Books that don’t quite meet projects probably start to pile up losses very quickly. To me it doesn’t seem as much of a little things can make a big difference tipping point, but rather a somewhat interesting multivariable equation.

I know some people who will throw out terms like “grammar nazi” relatively easily. Probably because “grammar fascist” doesn’t quite have the same ring. I know others that want to maintain the association of the word Nazi with a particular historical evil and do their best to not dillute the word by mixing it up with someone with an in-depth knowledge of English and a case of OCD. I’m afraid that tipping point has long since been disassociated from its original meaning, and from now on will be used to describe any sharp upward curve  on a graph.

Sometimes I feel sorry for Ken Olsen, the founder of Digital Equipment Corporation. Then I get over it. He gets teased about some of the things that he has said, but there are times that those comments are either misquoted or taken out of their original context. Unfortunately, even after correcting the quote or the context, you still wind up seeing someone who was missing a certain larger perspective of the changes were occuring in the computer industry.
He’s often said to have called Unix “snake oil”, but the if you look at even the rest of the sentence, and preferably the sentances around it, you find that it wasn’t quite that simple. A more complete version of Ken Olsen’s snake oil quote is:

Asked to comment on the recent uproar over the AT&T and Sun Microsystems Inc. Unix development alliance, Olsen without mentioning particular companies, likened some vendors of Unix products to “snake oil” salesmen and said the claim that Unix will resolve incompatibility problems within multi-vendor networks is “a naive idea.” “It still won’t resolve the problem of interchangeability”, he said, adding that the operating system is just one of the several components needed to achieve compatibility. He cited windowing ability and communications protocols as two other major components. Olsen went on to call Unix “one of the most proprietary operating systems”. But he expressed suport for standards and development of the POSIX interface, saying that will resolve the problem of making disparate operating systems compatible. “But that’s the unimportant part of making things interchangeable”, he said. Compatibility “doesn’t come by stamping Unix on the label. It doesn’t solve everything; there is no magic. It’s snake oil, absolute snake oil,” he said

Its not Unix itself that he was railing against, but rather some vendors claiming that just being Unix would solve every sort of compatibility issue. A “This is a Unix system. I know this” moment, I guess. And for anyone who might argue that connecting two Unix system two of anything else, consider how much of this has been ironed out over the since the quote was uttered. NFS vs RFS, job control vs. shell layers, and all of the Unix homonginization that was done by POSIX and FIPS.

Another quote that Olsen gets slammed for is:

“There is no reason for any individual to have a computer in his home.”
–Ken Olson, president and founder of DEC,
at a 1977 meeting of the World Future Society in Boston

Later, he had backtracked on that saying that he meant mainframe style computers. This might be argued as showing a bit of tunnel vision though. Some might view the quote, even with the correction, as a bit of tunnel vision. Saying “computers” and meaning “mainframe style computers” to me implies that he hadn’t considered what other forms computers for homes might take.

The next quote is the one Ken Olsen quote that I’ve always found the most intreging.

One of the questions that comes up all the time is: How enthusiastic is our support for UNIX?

Unix was written on our machines and for our machines many years ago. Today, much of UNIX being done is done on our machines. Ten percent of our VAXs are going for UNIX use. UNIX is a simple language, easy to understand, easy to get started with. It’s great for students, great for somewhat casual users, and it’s great for interchanging programs between different machines. And so, because of its popularity in these markets, we support it. We have good UNIX on VAX and good UNIX on PDP-11s.

It is our belief, however, that serious professional users will run out of things they can do with UNIX. They’ll want a real system and will end up doing VMS when they get to be serious about programming.

With UNIX, if you’re looking for something, you can easily and quickly check that small manual and find out that it’s not there. With VMS, no matter what you look for — it’s literally a five-foot shelf of documentation — if you look long enough it’s there. That’s the difference — the beauty of UNIX is it’s simple; and the beauty of VMS is that it’s all there.

– Ken Olsen, president of DEC, DECWORLD Vol. 8 No. 5, 1984

One way of taking that quote is simple Unix bashing. But I often consider the fact that Unix, at the time the quote was uttered was a much simpler system. It was simpler by design. Its often been said that the name Unix was a pun on Multics, and that it was designed to avoid much of the needless complexity of Multics. But what was considered needless? Nearest I can tell, that complexity included things like memory mapped I/O, shared libraries, ACLs, SMP, and a
fully instrumented kernel (allowing things like Solaris’ dtrace.) Most of these things have found their way back into Unix and Unix-like systems now, so to some extent we are right back where we started.

Modern Unix isn’t a simple system anymore. and it is a credit to the initial designers that it has been able to be extended so cleanly for so long. I think the major place where Ken Olsen got it wrong was that when people ran into the limitations of Unix, instead of abandoning it for more complicated system, they extended Unix as they needed.

If you read the Zope mailing lists, you frequently find suggestions for speeding up a Zope site by using Squid’s ESI support.

This is what happens when someone actually tries it. [Zope] Squid & ESI

The Director of Technical Operations just gave me a new desktop computer. An Apple Macintosh G5.

At least somebody there loves me.

I think what I got was the single 2GHz machine. Dan said to check if the second CPU can be user installed, and if I think I need it to let him know. He also told me that he bought it with the minimum RAM because he didn’t want to pay Apple’s build-to-order prices, but to keep bugging him until he remembers to buy the RAM upgrade.

He probably gave me the machine at about 4:00PM or so, was mostly set up and ready to go by the time I left at 6:30. Macintosh OS X has a neat feature that called “Migration Assistant.app” that is launched during initial setup. If you answer “yes” to “did you already own a mac before now?” it will instruct you on how to hook the two machines together with a firewire cable and Migration Assistant will copy all the old user accounts, Applications, and setting from the old machine. Or almost all the settings, I ran into a few problems.

Migration Assistant knows nothing about NIS or the automapper, so those had to be reconfigured on the new machine. (I have NIS done, I just need to take care of the automapper.)

The G5 has two gigabit ethernet jacks, the older G4 that I was using had a single 100BaseT jack. It transferred the network IP address to port 1, but reaching around the back to plug it into the new machine, I put it in port 2. That was easily fixed, when the networking wasn’t, um, working, and it said that port 1 was disconnected but port 2 didn’t have an address, I just switched the cable to port 1. (Migration Assistant also warned me that since it copied over a static IP address configuration, not to plug the old machine into the network. Of course I knew that but I found it interesting that it took care that scenario.)

Update:
Although Dan said that I had a single processor system, as I look at it now, I find that it is a dual 2GHz system. 512MB RAM, not Bluetooth, no 802.11 (the older G4 had an airport card and was rarely if ever used. I wouldn’t have minded bluetooth, but am really happy with the dual processors. (Maybe I should just ask Dan for a USB bluetooth adapter. They are cheap enough. I used to have one that I carried back and forth to work in my pocket, but I think the pocket lint static killed it.

I was in a near panic this morning as I was heading to work, because I realized that when I was decommissioning the older Mac, I didn’t “de-authorize” it with iTunes, which meant that it still had the one of the five DRM keys. When I got into the office, the old G4 was still sitting there in my cube, so I was able to boot it one last time to run itunes and turn in the key.

I just came across a web page about a release of New iTunes Usage Stats, Demographics. Its an odd mishmash of stuff. 2.2 times more likely to own a Volkswagen? I assume that they mostly newer Volkswagen models, as the only person I know with a vintage Bug doesn’t use iTunes. (Although for that matter the only current co-worker I know who owns a Volkswagen owns a Jetta and is strongly anti-iTunes and anti-iPod. He has strong opinions about lots of things though.) 12 – 17 year-olds are twice as likely to be iTunes Music Store users? Watches Cartoon Network at 1.4 times the average rate?

I just read the following article, Details on Intel’s beta Mac development tools. Intel’s compilers have a gcc compatibility mode, so they can be drop in replacements for the gcc tool chain and integrate with things like the Xcode IDE that expect gcc.

Unfortunately, according to the article “Intel has no plans to support Objective C”, so it will be hard to build standard Macintosh applications with Intel’s compilers.

I just came across a ComputerWorld article Hands On: Learning Monad, the scripting language for Windows Vista, adapted from the O’Reilly book Monad: Introducing the MSH Command Shell and Language

There are a couple of things that I found interesting in it. First of all, instead of connecting applications together with pipes and filters, it is designed to integrate COM components and .NET assemblies. Essentially, the way that for Unix the “filter” and “tool” are the components of reuse, for Monad it is the COM object. (To some extent, for Microsoft it has been for a while. To me, Visual Basic’s real use has been the ability to build UIs around a collection of COM objects. Monad is essentially the same for CLIs.

The next thing that I found interesting is that Monad can provide it its functionality as a service to other applications, so an application that needed to have similar facilities just needs to hook into Monad. For example, a GUI admin tool could add a Monad command line.

From the nearest that I can tell about pipelines, what passes between one cmdlet (monad component) and into another isn’t a text string like a Unix tool or filter, but rather and object or component, which seems like it would be a richer interface. Or a more confusing interface. I guess we’ll see.

From a somewhat snarky point of view though, its funny is that the output of the “get-help” command essentially looks like a man page. Beyond snarkiness into disdain, I find it frustrating that they’ve built things like registry access into “pseudo-filesystems” into the shell. (I have no problem with pseudo-filesystems, things like /proc are great, recent additions to Unix and Unix-like operating systems. I just have a problem that the shell’s view of the world is skewed by these illusions.)

I’ve been doing some investigation on how OS schedulers interact with Python’s I/O, and from what I’ve seen so far, performance is very dependent on how the OS schedules threads and processes. Many of python’s fileobject operations look something like this:

Py_BEGIN_ALLOW_THREADS
result = fwrite(data, 1, n, filepointer);
Py_END_ALLOW_THREADS

Where the intent is to give up the GIL, write the data, while this thread is blocked on I/O run another thread, and then regain the GIL. What seems to happen with some schedulers is: hand the GIL off to another thread waiting for it. Since it is one of two runnable threads, the *other* thread gets the CPU. When the *other* thread is done, this thread wakes up and immediately executes the I/O operation (blocking again) and then tries to regain the GIL (perhaps blocking again.) So depending on the scheduler, Py_BEGIN_ALLOW_THREADS can either allow overlapped threading and I/O, or defer all I/O operations until CPU bound threads are done, adding a timeslice quantum’s latency to I/O ops.

In the general case, there may be some very good reasons for the CPU scheduler to choose the other thread. Compared to the thread that just gave up the GIL, the new thread seems CPU starved (the current thread has run n ticks of the current timeslice, while the newly unblocked thread has run none.) the scheduler may see the GIL as a contentious resource, and wants the thread that has it to finish quickly and return it. (in the common case, locks tend to be around important shared resources, so giving up the lock means that you don’t have anything interesting to do.) But in Python’s case, often giving up the GIL is the thing that occurs just before you have something important to do.

If I can prove to myself that this is indeed what is happening, what I want to try to do is alter thread priorities within Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. Perhaps raising the priority of the thread will allow it to remain on the CPU after giving up the GIL and all the way to the blocking I/O call (where it will give up the CPU anyway.)

If I can’t prove it to myself, then I’ve probably learned a lot about Solaris’ much hyped dtrace utility and OS X’s Shark.app profiler.

In the comments to a previous blog entry, someone brought up the naughty word “Bonzai”. It reminded me of a movie that everybody who lived through that software development project should see, at least to commiserate with the subjects of the movie.

Everyone should find a copy of the movie Lost in La Mancha. It was originally intended to be one of those “The making of: The Man Who Killed Don Quixote” style documentaries. The only problem was that the movie they were supposed to document started going horribly wrong. It started as a series of small slips and mishaps, and then it grew, and grew. By the end of the movie, when they finally cancel production, sell off the assets and everyone goes home; it has an incredible feeling of relief. A feeling of a nightmare that is finally passing.

I think Shane and I went to see that movie in the fall of 2003, at the Kendall Square Cinema, around the time that our Bonzai project was being deployed. it was eerie how similar the film production felt to the software development project.