How to interview people that are more competent than you
I'm an ML engineer by trade. But recently I had to conduct technical interviews for a data analyst position. The company needs someone at the factory, looking for ways to help people with data and providing data-driven solutions. We expect the person to be good at both autonomously communicating with people and doing the technical work of analysis using Python, SQL, and BI tools.
Here's the catch: I'm not an analyst myself and don't know how to interview analysts. I have worked as a product analyst in the past, so I have some insight, but not much more than that. I don't know which tricky questions to ask or which technical skills to test. Plus, I only have one technical interview to form an opinion about the candidate.
I've come up with an idea: it's better that I don't interrogate candidates about what they know, but instead have them explain it to me. A good analyst will be able to explain the specifics to me in a way that I will understand. For example, I don't know the intricacies of running A/B tests. But I can definitely tell a good explanation from a bad one. Additionally, it will test communication skills, because at work the candidate will have to explain complex things to people who understand much less than I do. In short, I can ask open-ended questions instead of closed questions.
There is a small chance that the person will turn out to be a great storyteller and salesman, but can't write "SELECT * FROM table". That's why I added a simple SQL problem to the interview process. All you have to do to solve it is to join the table with itself.
The final structure of the interview is as follows:
1. Introduction, explaining how the interview will be conducted, talking about the company and the team.
2. Let the candidate talk about their experience. I then choose one of his projects and dig into the details. At this stage, I form an opinion about their experience and depth of knowledge.
3. Case question: “Imagine you work in a company and you need to find a way to evaluate the efficiency of a factory so that you can compare multiple factories”. I explicitly tell the candidate that they can ask any questions.
What I expect:
The person asks a lot of questions
Articulates the need to create a metric
Comes up with an appropriate metric. Understands that the metric needs to be normalized so that factories of different sizes and configurations can be compared.
One of the correct answers is ROI (return on investment), but it is not the only one.
At this stage I form an opinion about their problem-solving skills.
4. SQL task. I form an opinion about the level of programming and SQL.
5. If there is time left, I ask an additional question to check the knowledge of statistics.
I consider it to be an easy interview. However, we have already screened seven candidates (not counting those eliminated at the screening stage). Only three of them formulated a metric, and in my opinion, none of them asked enough questions. Only two solved the SQL problem. So far no one has demonstrated knowledge of statistics at the level of "to determine if there is a statistically significant relationship using linear regression, an F-test is applied".
Fortunately, two candidates out of seven passed the interview with a B. One of them quickly got an offer from another company, so we lost him. In any case, the conversion is not bad.