TopTal, Codility, and Automated Skill Testing

Just to get this out of the way, I failed the TopTal[^] application.

How did I even find TopTal? In a Code Project newsletter!  I’d never even heard of them before.

TopTal’s primary screening process is to use Codility[^] to see how good your skills are.

Now, the 90 minute timed test at Codility asked me to solve three problems:

1. the point in which in an array, the count of X from the left != count of X from the right.
2. given some bit encoding scheme, convert N to -N with the least number of bits.
3. the optimum number of moves a chess knight must make to get from (0, 0) to (m, n)

Regarding the last, someone on Code Project had posted about the idiocy of that last question a few months back.

In fact, they all are extremely poor tests of skill. Each (particularly #2 and #3) probably involve a simple trick to figure it out, and if you don’t know the trick (like me) you’re spending a lot of time just thinking about the problem. At least I was.

How exactly does solving an arbitrary algorithm test coding skills? How does it demonstrate good OOP practices, or DB architecture skills, or an understanding of things like Linq, or really much else other than “getting the trick?”

After working hard at #1, writing assertions, commenting the code, testing edge cases, optimizing the algorithm for O(1) performance, I realized I had spent an hour on one stupid problem. That left 30 minutes for the remaining two. Riiiight.

Now, Codility says something like “don’t worry if you don’t complete all three tests, just show your best work, even if you only complete one test.”

And then the test results are amusing. The requirements don’t state what to do with incorrect inputs into the “solution” method, and they clearly can’t handle exceptions being thrown — I noticed my score in their example test dropped dramatically when I used exceptions.

Now to TopTal. I got an automated email rejecting me because my score was too low. It was pretty obvious from the email that no one had even bothered to look at my code!  Relying on automated test results, not even looking at the quality of code, those are not ways to gain points with me.

Sadly, this sort of testing methods is probably going to be used more and more.

Now, oddly, I don’t particularly have a problem with the concept behind Codility (except of course that it doesn’t really test skills) but I definitely have a problem with how TopTal handled it – I would have expected, at a minimum, a follow up discussion with a technical person.