The incredibly readable Joel Spolsky writesa very readable talk at Yale. It meanders a little but that just makes it all the more crunchy. I had one or two lecturers like that at University and I’d say we learned more and listened more when they made the story interesting:
The Windows Vista team at Microsoft is a case in point. Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.
The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.
None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug†by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible. It turns out that nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.
One of the things I’ve been doing recently is developing a test plan for Rickshaw. We’ve been nailing bugs here and there mostly due to weird and wonderful undocumented things in Mac OS X. I think, next time round, I’d like to build something simple and solid, like a text editor or maybe a few HUD-enabled Hello World applications. The whole uncertainty of doing something really freaking cool with undocumented APIs just continues to put the willies up me. Things are progressing and there’s not much left to do.
Our test plan consists of using the software and making sure we run through the fifty or so tests we’ve written down. Some of them are just re-iterations of previous tests and some of them are designed for testing in the various versions of Mac OS X. As long as Apple doesn’t release 10.5.2 or 10.4.12 in the next few weeks, we’ll be all set.
Testing in itself is boring. Using, on the other hand, is great. I used an early build and for a time forgot I had it installed. I was sending out URLs left, right and centre to everyone who I sent an attachment to and not only did I get no complaints but people kept asking me how I was doing it and what “Rickshaw” was. I was using the unregistered version which has some additional signature text, sosumi!
Anyway.
The last paragraph in Joel’s talk is absolutely chilling and not copied here so, go read. I’ll be waiting here for when you get back.
When I was teaching at the ‘Tec’ in Lisburn, it was like this. There were some courses which were relatively hardcore like Computer Architecture (which I taught) and then there was the Web Design course. In my third year teaching, they introduced the Website Admin course which was the next step in the Computing, Business and IT curriculum and involved use of a scripting language and some sort of database for dynamically-generated web pages. At the time I wasn’t a programmer but I understood the principles and could hack together some simple scripts in PHP to create an address book or something. The students had the pre-requisite of the Web Design course where they learned about HTML, hyperlinks and the like.
Or so I thought.
I was introduced to the Website Admin course two weeks into the course. The previous instructor, who was the logical choice as he was he same guy teaching Web Design, had given up and walked out of the class leaving 23 confused adults. The course head came to me as I was finishing up a Computer Architecture class and asked me to take it. I agreed to consider it if she could get me the syllabus now and I’d let her know in the morning when I’d had time to read it.
No, she replied quietly. She needed me to take the course NOW.
A little shell-shocked I grabbed the syllabus from her and wandered into the class and had a sit down talk with the 23 adults and established that though they had completed the Web Design course with merits and distinctions all, not one of them had even seen HTML. I told them what we’d be learning and there was a horrified hush in the audience. They explained the Web Design course consisted of creating pages in Word or Powerpoint and exporting to HTML. I was nearly sick. I had to start from the first principles with them.
And then the class was over and I had six days to prepare 12 weeks of a course. Some of them struggled, one or two fell in love with the more technical aspects of it and by the end of the 12 weeks were doing some amazing things with PHP – amazing to me anyway considering what I’d had to teach them. Lack of resources and short sighted administrators meant that our “toolkit” consisted of Notepad, the MS-DOS command line, my laptop running an early version of Mac OS X, mysql and Apache and my personal server (an old PowerMac 8100 running Mac OS X Server). We had no support from the faculty office, no resources, no server and, frankly, no time. I gave them logins on my server, I hosted all of my lecture notes online so they could work on the assignments from home or work and I gave them my personal email address.
Not long after I stopped teaching. They moved more into providing “pre-packaged” certification courses and began to pump out a hundred MCPs and 40 CCNAs a year. I was bored of trying to teach things the right way and not getting any resources, any consideration and working for hours and hours outside the classroom for free just to create a decent practice environment. Maybe I would have been better just giving them Powerpoint?
But they coped and as I said, some of them thrived. Everyone but one over three years of teaching that class passed, there were distinctions and merits too. The one failure was for many reasons which I’ll not go into here. And about three years later I was up at the University of Ulster Jordanstown campus (giving a talk to some undergraduates about IT and entrepreneurship) and bumped into one of my students. He said he was poor as a churchmouse but loving very minute of his Computer Science course. His aspirations when he started the Business IT course was to help his bricklayer boss with the accounts. The Website Admin course showed him there was more to the web and he changed his life. (He now works for a web consultancy firm in South Belfast).
Joel’s article helped me remember that today and cheered me up no end.