Dispatches from Andyland

At my work, we use a product that has a purely object oriented persistent storage. (as opposed to an object/relational model where relational data is mapped onto objects.) I’m not sure which object oriented persistent storage is important, but if you gathered them all up and alphabetized them, you would get far through the list before you found it.

Some of my earliest introductions to object oriented software development described the flexibility and nimbleness of developing software in this model over previous methodologies. Encapsulation allows the implementation to change without changing the interface to an objects clients. Object derivation allows you to extend object behavior of subclasses without modifying existing code in the base classes. The common features that were touted seemed to be along the lines of allowing change to be easier so that development would move faster.

I’m suspecting that at least in our case, an object oriented persistent store negates this advantage by tying object classes to specific implementations at a specific point in time. Our application has two objects that act as repositories or brokers to a heterogeneous collection of objects. One is a manager of “content” objects like stories and the other is the manager of “resource” objects like CSS files, and image and text pieces that make up the look and feel of the visual presentation. Both derive from the same class, and are composed of things like “catalog” classes for efficient searching and storage classes for ease of access by key. The software has tools to import a directory of resources off the filesystem and into the persistent store, or export the persistent store into a tar archive for use storage elsewhere (like the CVS revision control system.)

In our last software release, someone created a resource with a ridiculously obscure name “2nd_nav_topstories_off” or something like that. The attempt to import the resources failed. What There was already an item with that name in the repository, but any attempts to read that item to modified it would fail. I wound up finding out was that that the item was initially put there, but then after it was stored, someone made a modification to the repository code so that it would call .makeObsolete on the objects that were removed. Our image types were all modified to have that method, but this existing object was one of the basic image types (ones that come with the app server.) and knew nothing about it.

Describing in terms of a statically typed object oriented language, someone made some internal changes so that the signature of the method needed to change from being “public Object getResource(Key k)” to “public Resource addResource(Key k)” (by internally within the method of treating the object as a Resource), but not all of the data saved by the original version were our application’s Resource types. Also looking at this from the statically typed Java point of view, it solves things problem by hashing the fields and the type signatures of objects as it serializes objects and checks them as it unserializes them. It will throw an exception if trying to deserialize an object from a different version than you serialize it as. (with an escape hatch called the serialVersionUID field.)
Its sort of odd, using a less dynamic language than python wouldn’t allow this to happen. All of the objects in the repository object would likely need to adhere to a specific interface or abstract superclass. Some people will say that the only solution is a Dump and reload of the data, and that is what projects built on RDBMs do. I’m not sure if I buy that either. First of all, a “dump and reload” of an arbitrarily connect object graph is much more difficult than a tabular structure. Secondly, we had most of the dump and reload functionality built into the application, and it didn’t help us. Thirdly, it isn’t the data attributes of the objects that changed, or the interface of the repository objects, so I’m not sure what a dump and reload could have helped here.What would have prevented this? I guess a bunch of things. If the “import from the filesystem” code cleared the repository, older objects would have been erased. That is almost what the writers intended on doing, and I guess the only argument I have is that if a system makes it easy for errors like this to happen, the errors will occur. A fixup script to run before deploying new code? I guess that could have worked too, although somewhat annoying to have to keep track of. When mulling this over with a co-worker, where older versions of the code get saved along with the current version, and small adaptors are written from one version to the next.

One final thing, I guess I should mention is that this incident broke the Open/Closed Principle, and I guess some people might say that the result was inevatable (or at least reasonably possible.) I guess my concern isn’t so much the fact that the bug occured in a persistant object store, but that the repurcussions were so much larger than in systems that don’t have one.

Yesterday I was out in the pool with my two daughters and they were developing some sort of game/story for themselves. “Let’s pretend that we’re both 12 1/2. I’ll be ‘Ashley’. I’m going to be a gymnast and a professional swimmer. You can be ‘Jessica’, you can be a ballerina. Dad, you can play our dad, a lifeguard, and also be the director of the show.”

To that I replied, “OK, as long as I can be a super-hero too.”

I had a conversation with my four year old daughter which went like this:

Abbie: Remember that book that I made you for this Father’s Day?

(I hadn’t heard about the book until this point, but that’s OK. She writes a lot of books so it seems likely she’d make one for me)

Me: Not yet, but I’m sure it will be great. I can’t wait to get it this Sunday

Abbie: I have another present for you too. Try to guess what it is!

Me: I don’t know. (scanning quickly across the room) Is it a balloon with a face drawn on it and feet made from a sheet of paper?

Abbie: No! Guess again! I’ll give you a hint. It begins with the letter “B”

Me: Hmm, “B”, and you said it wasn’t the balloon, and you wouldn’t make a “bottle”, hmm…

Abbie: I’ll give you another hint. The next letter is “X”. (pause) Actually, I don’t know how to spell yet.

At work today, I was working on building a new ZEO storage sever on an new Solaris machine for our test cluster. Soon I’ll be mimicing the setup for a new production ZEO server. Although a ZEO server is predomently I/O bound, I wanted to see what I could do about CPU performance for the python interpreter. A co-worker name Bill pointed out some articles like SPARC Optimizations with GCC that show how much of an improvement compiling with UltraSPARC specific opcodes can do. Since the earliest UltraSPARC chips are 10 years old, I can’t imagine needing pre-UltraSPARC compatibility.

Besides that, I tried some of the profile-directed optimizations. Basically what you do is you initially compile and link the file with one of the profile generating options like -fprofile-arcs or -fprofile-generate, and then you run the resulting binary which will write out profiling files with information on which branches are taken or skipped and how varibles are used. Once these profiling files are created, a new binary compiled with -fbranch-probibilities or -fprofile-use will read those files and adjust the compiled code based on that profiling data.

Until I ran into one little problem, I was hoping to be able to just do something like:

env LDFLAGS=-lgcov CFLAGS=-fprofile-use ./configure
make  "`echo \`grep ^OPT= Makefile\` -fprofile-generate`"
make test
make clean
make
make install

The problem I ran into was a GCC 3.4 bug that would create unusable profiling data files under certian specific circumstances. Because of that, instead of running the test site, I substituted ./python -c”from test.pystone import main;main(). This would run fine, and I would get a python that ran about 20% faster.

Once I got home, I tried this on the various Intel boxes I have around (from 450MHz Pentium IIs to 1.5GHz Celerons). I haven’t tried the Macs yet. The results so far seem similar, but not quite as impressive. The Celerons results were:

$ for dir in Python-2.3.5-singlecompile Python-2.3.5-profile_recompile; 
do 

     echo $dir;
     $dir/python  -OO  -c 'import test.pystone;test.pystone.main()'; 
     $dir/python  -OO  -c 'import test.pystone;test.pystone.main()'; 
    $dir/python  -OO  -c 'import test.pystone;test.pystone.main()'; 
done
Python-2.3.5-singlecompile
Pystone(1.1) time for 50000 passes = 1.484
This machine benchmarks at 33692.7 pystones/second
Pystone(1.1) time for 50000 passes = 1.5
This machine benchmarks at 33333.3 pystones/second
Pystone(1.1) time for 50000 passes = 1.484
This machine benchmarks at 33692.7 pystones/second
Python-2.3.5-profile_recompile
Pystone(1.1) time for 50000 passes = 1.328
This machine benchmarks at 37650.6 pystones/second
Pystone(1.1) time for 50000 passes = 1.298
This machine benchmarks at 38520.8 pystones/second
Pystone(1.1) time for 50000 passes = 1.328
This machine benchmarks at 37650.6 pystones/second

Update:
A couple of things I just realized that I should have mentioned mentioned last night when I wrote this. One thing I've been warned about for profile directed optimizations is to make sure you run your instrumented build in a similar way the production one will be. Sometimes your regression tests or unit tests might not be the best choice. For example, most Python C extension functions or extension object methods start with something like:

if (!PyArg_ParseTuple(args, "iiO", &i, &j, &value))
   return NULL;

and when running a normal program, the profiler will learn that nearly overwhelmingly that PyArg_ParseTuple will never return false. A regression test, on the other hand, may spend a lot of time testing functions with incorrect numbers of arguments, and skew the results. In the particular case of the Python interpreter though, I think a lot of the python level tests would still count as representative. The main interpreter loop is still executing opcodes the same way. The normal loop, arithmetic, and function call opcodes are going to be doing their normal thing.

Adam over at Universal Hub points to a piece, Inside a Metro editorial meeting imagining a staff meeting at the Boston Metro trying to figure out how they can be as useless as possible. What I find funny about this is that if you look at Adam’s Blog Log column in the Boston Globe, a weekly column about interesting things found in blogs, it doesn’t have URLs pointing back to the original blog. Some of the time, there is a single URL at the bottom “You can see the complete posts cited above at …” and a from there a link to the sources. The rest of the time (when the Globe editors need the extra space?) there is no links at all within the article.