Friday, May 11, 2007

Brain-teasers make bad interview questions

The other day BitTorrent posted a job posting to the BayPiggies mailing list with the following brain-teaser to solve in it:
What is the exponent of the largest power of two whose base seven representation doesn't contain three zeros in a row?

Normally, I hate "brain-teaser" type interview questions. I make it a point to never ask brain-teasers in interviews and don't put too much weight on candidates' answers when my fellow hiring managers ask them. The reason is simple: the typical brain-teaser is too easy to game.

Take BitTorrent's question, for example, the answer is 8833.

BitTorrent asked for python code to solve the problem, but that is a red-herring. You cannot solve the problem in python. Sure, you could write a program like the following that prints all of the powers of 2 that do not have three consecutive zeros in the base-7 representation:
    def base7(x):
digits = []
while x != 0:
x, r = divmod(x, 7)
return ''.join(reversed(digits)) or '0'

i, x = 0, 1
while True:
if '000' not in base7(x):
print i,
i += 1
x += x

But BitTorrent asked for the largest power of two that meets the requirements. How do you know when a number printed by the above program is the largest? You don't! There are infinitely many numbers and this program has to test them all. That'll take a while.

The full solution to BitTorrent's problem requires you to find the upper-bound of the search -- which can only be done without a single line of python code (despite their requirement to use python): you have to solve it using math. Which implies that BitTorrent actually wants their ideal candidate to be a mathematician first and foremost. That isn't necessarily a bad thing; I don't know anything about BitTorrent's business, maybe they need people with particularly strong math backgrounds (much stronger than mine, that is).

But there is actually another way to obtain the solution: use Google. So knowing how to work a search engine will get you in the door just as effectively as having an advanced math degree. Go figure.

Neither skill says anything about the candidate's programming ability or knowledge of python in particular. In fact, I would argue that attempting to "solve" the problem by writing a python program without including the proof supporting your solution should disqualify the candidate as demonstrating that they don't understand the problem. Writing code without understanding the problem is well into Code Cowboy territory.

Anyway, the point I wanted to make was that it has been my experience with "brain-teasers" that at best they give the interviewer some vague idea of the candidate's problem-solving ability. Far more often, however, they randomly filter out people who may be perfectly good programmers but who didn't happen to have memorized (or found online) the answer to the particular brain-teaser you throw at them.

When I do interviews, I figure I only have a limited time to interview a potential employee. I would rather spend that time asking questions specific to software engineering, computer programming, or the problem domain we are hiring for. I always prepare a progression of real-world questions ranging from "easy" to "guru", with no expectation to ever ask the "guru" questions -- my intention is to see what the candidate's level of ability is and, more importantly, to see how they handle a question or two slightly beyond their ability. Maybe it's because I'm not too bright myself, but that tells me much more about the candidate's suitableness for a position than an abstract brain-teaser ever does.


jjinux said...

I answered the question using Python before interviewing there several weeks ago. I specifically said, "This is the highest number that I could find, but I don't have a mathematical proof that it's the highest number." It's nice to see that you and I think in the same ways ;)

I interviewed with Bram, the creator of Bittorent. He's spent the whole interview giving me brain teasers. He asked me which was more likely, a straight or a flush. I'm not a poker player, so it took me 20 seconds to even figure out what he was talking about!

Bram is wicked smart, but the organization is what I would call chaotic. He's a fast thinker, but I'm a slow, methodical programmer. I felt like I was a turtle, and Bram was a hare doing laps around me. It was the worst interview I have ever attended. I left the interview visibly shaken.

This reinforces my previous belief that being wicked smart doesn't necessarily make you a great software engineer. I'm sure Bram's a great coder, but the company has no style guide, doesn't embrace testing, and works between 35-80 hours a week. That's not what I call smart software engineering.

Kelly Yancey said...

And so top-notch engineering talent goes elsewhere. Their loss.

Anonymous said...

You make a throw away comment I think is important. You 'have a range of questions, from easy to guru'.

Without a range of questions you can't find out where someone is on the scale. I see too many programming tests which are all hard questions, or all C++ language corner cases. If someone fails to answer hard questions then have you just thrown out a very good, but not super exceptional coder, or someone who knew nothing. You can't tell, and a very good coder with other skills might be just what you need.

(If someone can't answer questions about language corner cases - hire them. They'll write much more maintainable code! ;-) )

Michael Hunter said...

Some brain teasers are goofy but I'm not entirely against the one that gets the programmer to not jump straight to the keyboard. Knowing if the person aims before he or she fires is an important part of hiring them.