A few months ago I wrote a post on calibrating data science interview questions to see where data scientists scored in comparison to their peers. We eventually had over 600+ data scientists take our quiz that was eight questions of varying topics in data science interviews.
P.S. If you haven't done the quiz yet and don't want to see the results to bias your score, take it here and stop reading!
A quick summary. We had over 600 responses on the quiz with six multiple choice questions and two short response. I was curious to see what kind of short responses we would get, but generally they weren't very descriptive and sometimes completely random. So I threw them out of the ending quiz scoring.
The easiest question on the quiz was the machine learning question regarding regularization in which over 56% of respondents got right.
The hardest question on the quiz was the algorithms one. The correct answer was actually the second most answered. Generally this isn't too surprising given this required the most deep thinking and complexity to understand.
When we plotted the total scores, we found it interesting how close to a normal distribution it was. Most people got just two answers correctly with a mean of 2.35 and a standard deviation of 1.37. And if you answered four out of six questions correctly, you were already in the top 20%!
The last interesting insight was that at the end, we asked our users if they would be open to offers from companies if they scored well. We plotted both distributions to see if there would be a difference in the results based on their job seeking behavior.
A potentially slight increase but nothing that really stands out. It's pretty difficult with a low question sample size to see a significant difference here.
Ultimately this quiz was an interesting test to understand how calibrated our questions were. But I think it's obvious that maybe the biggest variance in the results is based on how hard each user decided to work on the quiz. While it's meant to be equivalent, each user has different expectations on how much time it should take and their overall investment into the quiz.
Also, do users that score well on the quiz actually perform well on their data science interviews? The data science field has such a wide array of roles and responsibilities that probably can't be measured by how well you answer technical questions. But more commonly, they do act as a flag, an initial attempt to weed out false positives for many hiring managers. For example, I really liked the SQL question in the quiz because it tested two concepts; if you understood how a LEFT JOIN worked under the hood and what the distribution of the query results looked like.
Lastly if we were to increase the number of questions on the quiz, would that give us a better distribution and measurement of proficiency for each data scientist? At one end of the spectrum, we don't want to lower the conversion of quiz and have users drop out halfway. But at the same time more questions means more data points to trust in the results and to use to understand individually where a user could improve in their data science interviewing skillset.
Ultimately while data science interviewing will continue to be a black box, we'll continue to try to shine a light in there every so often to see if we can't just figure some more things out!