Podcast #39

This is the 39th episode of the StackOverflow podcast, where Joel and Jeff discuss database design and the shell game of performance, the value of short, focused presentations, and the importance (or not) of a prestigious degree for software engineers.

• Joel insisted that we run the stackoverflow.com books on QuickBooks. I was more open to using an online accounting solution, but we were worried about someone holding our business finances hostage. Joel also maintains that "every accountant in the world knows QuickBooks". I'm not happy about it, but small business reality.

• Now that we can (theoretically) afford to pay ourselves to work on the site, we deployed two new features: reputation bounties on questions and new reply notifications.

• Joel expresses some concern about the value of panel discussions at conferences. There are lots of variables: who is on the panel, what are the topics, and how skilled is the moderator? It's almost better to give each person on the panel a 10 minute window to talk about some topic they think will have value for the audience.

• I often think of small, focused presentations as "Grok Talks". One presentation format which hews closely to this model is Pecha Kucha. It's generally a good idea to err on the side of keeping it small. The larger the talk, the more material you should have whittled away and deleted to get it down to size.

• In building Stack Overflow, we copied a key part of the Wikipedia database design. This turned out to be a mistake. We are due for a massive database refactoring, which is going to be very painful, but it is necessary to avoid excessive joins in a lot of our key queries.

• We also begin to appreciate why the giant multi-terabyte table schemas (like Google's BigTable) are completely join-free, basically simple hashtables or tuples spread across a server farm.

• My benchmarking shows that CPU speed is surprisingly important on our database server. Going from 1.86 GHz, to 2.5 GHz, to 3.5 GHz CPUs I see an almost linear improvement in typical query times! The exception is queries which don't fit in memory, but right now with 4GB RAM most of our DB fits in ram; with the new 24GB RAM server I expect the database to be entirely in memory for quite a while.

• Joel talks a bit about queueing theory, and its relation to OS scheduling. I propose that performance is a shell game with bottlenecks, where you're constantly trying to decide whether the bottleneck should be memory, CPU, or I/O.

• More discussion about unit testing: when it's appropriate, when it isn't, and what it is for. Remember, 100% code coverage (or even 90% code coverage) is not free. You're playing another shell game, so decide what axes of that resource balancing equation are important to you. Perhaps the greatest risk is being dogmatic about it.

• If you enjoy this podcast, you might also enjoy Hanselminutes -- Joel gives it his seal of approval!

• Joel revisits the SOLID principles, and compares them to designing replaceable batteries, or a headphone jack, into your product. Appropriate in some narrow cases, but not all the time. Imagine a consumer product where every single part of it could be plugged in and replaced with another compatible part. Is that realistic?

• Perhaps new language features will advance programming a lot faster than any given set of programming principles, no matter how good they are. I've often thought that design patterns are how languages evolve.

• Joel says you don't have to go to a prestigious name brand school, necessarily, but you need to choose schools that have a rigorous selection process. "These people are a tiny little bit more likely to succeed at our rigorous selection process." I concur -- if you aren't seeking out challenges, whether they are large, small, or somewhere in-between -- what are you doing?

Our favorite Stack Overflow questions this week are:

• Jeff: Is mathematics necessary for programming? It can't hurt, and often helps, but it is not strictly necessary in our opinions.

• Joel: Does it matter where you get your CS degree? In brief, yes, but not for the reasons you might think.

We answered one listener question on this podcast:

1. Michael Coney: "Joel mentioned the SOLID principles on the previous podcast. Our software architect was also fond of over-isolation but was talked out of it. In the real world, if you need to change something, you just rewrite it."

If you'd like to submit a question to be answered in our next episode, record an audio file (90 seconds or less) and mail it to podcast@stackoverflow.com. You can record a question using nothing but a telephone and a web browser. We also have a dedicated phone number you can call to leave audio questions at 646-826-3879.

The transcript wiki for this episode is available for public editing.

Episode source