Everyone believes in data-driven decision making—until they start hiring

Best practice—think Six Sigma in manufacturing or disciplined pipeline management in sales—is typically something to strive for in a business context. After all, if you’re looking for the most reliable way to achieve a desired outcome, your best bet is to lean on proven processes and scientific methods that are rooted in empirical data and analytics.

But when it comes to hiring, too often people revert to instinct, emotions, and an “I’ll know it when I see it” approach. They let their own experiences and preferences cloud their judgment and assume they’ve understood someone faster and more fully than they actually have. In short, they commit interview malpractice.

We’re all familiar with the scar tissue that can result from bad hiring decisions. Our research shows that a single mis-hire can add up to as much as 15x a person’s annual salary in “hard” costs (e.g., compensation, severance, recruiter fees) and “soft” costs, ranging from the opportunity cost of failing to make the most of business opportunities to the negative impact of one bad apple on overall talent quality and engagement.

When hiring decisions fail, interview malpractice is often the root cause—and it can take many forms…

We’re guessing that at least a couple of these interview “don’ts” have a familiar ring to them. You may have used one or more of them yourself, or even been on the receiving end. (A participant in one of our trainings encountered a Trickster who borrowed a page from The Wolf of Wall Street, handed the candidate his pen, and asked her to sell it back to him—even though the job had nothing to do with sales.)

The good news is that there is a cure for interview malpractice…

Do: get curious instead

People are complicated. Understanding how someone operates across different contexts and roles takes work—there is no magic bullet (or magic question). The secret is to make a choice at the start of the interview: get curious, and stay curious longer than feels comfortable. Don’t rush to a verdict. You’ll have an instinct early on. That’s normal. Just don’t let it make the hiring decision for you.

In practice, getting curious is straightforward. You ask open-ended questions and follow up with more open-ended questions until you know specifically what the candidate accomplished, how they did it, and the results. The follow-ups are basic: “What?” “How?” and if all else fails, “Tell me more.”

The hard part is not moving on too quickly. You want to understand what someone has done, not what they want you to think they’ve done. That means digging beneath the polished, canned answers and finding evidence of how candidates behaved in real situations. Ask more open-ended questions and keep double-clicking and triple-clicking until you know exactly what happened.

Consider a candidate we assessed for a CEO role. The recruiter described him as a “turnaround guru,” and our client was in desperate need of a turnaround. The phrase “led successful turnaround” appeared multiple times on the candidate’s résumé. And when we asked him what he was most proud of from his current role, he said, “I singlehandedly turned the business around. It was on life support, and now it’s thriving.”

That might very well have been true. But how could we know for sure? We only had his word for it—in a courtroom, the opposing attorney would move to dismiss his testimony as hearsay rather than admissible evidence.

So we got curious about what the candidate did and how he did it. We asked, “What did you do first (and second and third)?” “How did you decide where to start?” “What was your step-by-step approach?” “What would I have seen you doing if I were a fly on the wall?” “Where were your fingerprints felt the most?” As we drilled down, a picture began to emerge of what actually happened. It was nowhere near as neatly packaged and clear-cut as the candidate’s top-line answer. And that’s fine—in fact, the messy parts can offer the biggest insights.

In this case, the candidate’s first move was to call in a firm of turnaround consultants he’d worked with in several previous roles. He leaned on them to figure out which levers to pull, put together an implementation plan, and establish a PMO to guide the process. Where the candidate added the most value was in getting his executive team bought in to the plan and leading the change management program to rally the broader organization.

His ability to get everyone rowing in the same direction turned out to be exactly what our client needed—they already had a well-vetted, cost-effective turnaround plan ready for the right leader to put in motion. But it would have been a different story if the CEO was going to be plunged into a situation where neither the problems nor the path forward were obvious and the company didn’t have the time or resources to call in an army of consultants and shell out millions in professional fees. Getting curious about the “what” and “how” of the candidate’s behaviors was the key to understanding whether he would be a fit.

You’ll want to get equally curious about answers that seem like objective statements of fact. If a CFO candidate says she increased EBITDA margins at her current company, you can probably stipulate that the margins increased. But it’s still more hearsay than evidence.

Start by getting the bookends of performance: “What were the margins when you started? And what were they at the end of this role?” Then you’ll want to use a similar approach to the one we used with the CEO to understand how the candidate most directly contributed to the margin improvement.

But don’t stop there. You’ll also want to calibrate what “increased” really means. The “Three P’s”—Past, Plan, and Peers—can come in handy here. Ask, “How did the margins compare to where they were historically? How did they compare to the budget or forecast? How did they compare to industry benchmarks or key competitors?” If earnings had been flat for the last five years and beat plan by 25% while competitors’ earnings shrank, that’s something to get excited about. If they beat plan by 25% but had previously been growing 50% year-over-year and the broader industry was still growing at 50%, “increased” becomes a lot less exciting.

Leaders’ claims about their style are worth digging into as well. If a COO candidate says, “I’m very hands-on,” everyone nods. But “hands-on” means different things to different people. He could be a player-coach who tackles certain workstreams directly while pitching in on his team members’ workstreams when they need more support. Or he could keep a close eye on progress through regular check-ins and reviews and be quick to offer direction when obstacles arise. Either approach might work well – if that’s what is needed in the role. But when we stayed with it, we realized that “hands-on” for this candidate meant stepping into nearly complete projects and upending them, which occasionally led to better outcomes but more frequently created churn and undercut his team’s sense of ownership.

In addition to getting curious about candidates’ achievements, we encourage interviewers to spend as much or more time getting curious about what didn’t go as well. Candidates will naturally steer you toward their strengths and gloss over their weaknesses or development areas, but we find that the fumbles and missteps that rarely show up on a résumé can be even more illuminating than their biggest wins.

For example, a commercial leader might walk you through a successful product launch, and because you got curious, you know exactly how they shepherded the product from concept to pilot to going live in the market. But asking questions like “What was the hardest part of getting that done?” and “What didn’t go as planned?” and double-clicking on those answers will unlock data about the candidate’s mistakes and setbacks, and what they learned from them (or didn’t, depending on how self-reflective they are). The bumps along the way that only became obvious in hindsight can be a rich source of insight.

We also find that some of the most useful information comes from comments that may seem obvious or don’t require further exploration. Consider the candidate who says that their least favorite part of a job was “the corporate politics.” Nobody likes corporate politics, and you might be tempted to say, “I hear you – I hate that stuff, too” and move on. But they’ve just given you a golden opportunity to understand how effectively they partner with colleagues across organizational lines. Ask for examples, and keep asking until you have the information you need.

Don’t: commit debrief malpractice (top don’ts to avoid)

Congratulations! You nailed getting curious and came out of the interview with a well-rounded picture of the candidate. The next step is to debrief with the other interviewers and make a decision.

You arrive armed with a hire/do-not-hire recommendation that is based on all the calibrated detail you gathered in your interview. However, smart people, some data, and time pressure can be an unhealthy mix – everyone wants to make a decision and get on with it. This is when interview malpractice can morph into its less infamous but equally perilous cousin, debrief malpractice.

Debrief malpractice can also take many forms. Some common pitfalls include:

• One loud signal drowns out the others. An especially vocal and/or senior-most interviewer declares, “The candidate is a strategic genius.” The room latches onto his conclusion, and no one pokes at it to check whether there’s sufficient substance behind it. Get curious in debriefs, too, and ask for the data.
• The group votes before everyone’s had a voice. This one is a close relative of the loud signal pitfall. A couple of the first people to speak are positive, and their comments create momentum, with fewer people offering a different take. The dissenters may even start to doubt their own opinions. But unless you surface those views, you’ll be making a decision without getting all the information on the table.
• Familiarity starts standing in for evidence. Someone says something like, “I used to work at [candidate’s current company], and you can’t survive there without being good at [critical competency].” That might be a safe assumption. But it can tilt the room. We’ve seen two candidates tell very similar stories, but one lands better because their background, communication style, or emotional response parallels that of the interviewer. Focus on the specific examples that point to the candidate’s capabilities.
• The last candidate feels like the strongest. It’s not that the candidate is stronger, but people are tired of interviewing and the most recent guy is the freshest. You hear people say, “The last one really stood out.” Earlier candidates may have been just as strong or stronger; you just remember them less clearly. Take notes so you’re not relying on memory, and then compare candidates side by side against the same criteria.
• Vague language and verbal shorthand sneak in. One candidate has “great executive presence” and another “would be a good fit.” Abstract descriptions defy data-driven assessment. Ask people to back up their conclusions with examples rather than feelings or impressions.

The common thread across these pitfalls is allowing perceptions and emotions to color your decisions. Many of them mirror the cognitive biases that have been documented by Daniel Kahneman and his colleagues[1]: the loud, senior voice parallels anchoring and authority bias, and the early-vote momentum is akin to groupthink. Meanwhile, leaning on familiarity translates into affinity bias, and “the last one stood out” is a perfect example of recency bias.

Rather than allowing these mental shortcuts to distort high-stakes judgment…

Do: take a structured approach to debriefing and decision making

When it comes to debriefs, a little consistency and process discipline go a long way.

Provide interviewers with copies of the scorecard[2] you’ve created for the role. Ask them to rate the candidate against each of the outcomes and competencies on the scorecard immediately after the interview. Being comprehensive at this point will reduce the likelihood that interviewers come back with blurry overall impressions. Also ask them to capture the data that backs up their ratings, so that they’re not trying to remember the specifics after the fact.

Then when it comes time to debrief, designate a “debrief captain” who will be responsible for making sure every voice is heard. It might make sense to call on those who tend to be less vocal first and save the louder voices for last. The captain should be sure to check for the examples underlying each interviewer’s perspectives and pressure-test that they’re relying on data rather than hearsay.

The captain can also ask one interviewer to present the bull case and another to present the bear case for each candidate, to facilitate a constructive debate (this approach can be particularly effective if you ask people to argue the opposite of their own position). Additionally, following the same steps in discussing each candidate and comparing across candidates can help guard against the bias traps discussed above.

Conclusion

Good interviewing isn’t about asking the perfect “gotcha” question or having a great “feel” for who people really are. It’s about getting curious and insisting on evidence over instinct—both in the interview and in the debrief. The best hiring decisions come from understanding not just what a candidate says they did, but how they actually operated when it mattered most.

Everyone is an A player in their own way—the last three decades have taught us that this is as true of CEOs as it is of entry-level interns. Your mission is to discover whether a candidate is an A player when matched to your specific context and needs.

—    —    — 

[1] Kahneman, Daniel, Olivier Sibony, and Cass R. Sunstein. Noise: A Flaw in Human Judgment. Little, Brown Spark, 2021.

[2] For more information about scorecards, check out Scorecards – They’re Not Just For Hiring Anymore.