Structured Interviews: The Single Highest-Validity Hiring Practice

In 2016, Schmidt, Oh, and Shaffer updated one of the most-cited meta-analyses in industrial-organizational psychology: a century of research on what actually predicts job performance. They ranked every selection method — IQ tests, work samples, structured interviews, unstructured interviews, reference checks, personality tests, experience, education — by validity coefficient.

Two findings stood out. First, structured interviews ranked near the top, alongside work samples and cognitive ability tests. Second, unstructured interviews — the default at most companies — ranked near the bottom, barely above coin-flipping in many studies.

"Structured" doesn't mean rigid. It means consistent across candidates and tied to a rubric. Specifically:

• Same questions, same order for every candidate interviewing for the same role
• Behavioral and situational questions tied to job-relevant competencies, not "fit" feelings
• Pre-defined scoring rubric with anchored examples for each rating level (1–5 or 1–7 scales work best)
• Independent ratings by each interviewer before the group debrief, to prevent anchoring
• Trained interviewers who have practiced the rubric and calibrated against shared cases

What it's not: a checklist of yes/no questions. Structured interviews are still conversational, probing, and revealing — they're just consistent enough that two candidates can be fairly compared.

The reasons unstructured interviews score so poorly on predictive validity are well-documented:

• First-impression bias. Decisions are typically made in the first 10 seconds and rationalized for the next 45 minutes.
• Similarity bias. Interviewers rate candidates similar to themselves higher — same school, same hobbies, same way of speaking.
• Confirmation bias. Questions shift based on early impressions; positive candidates get easier questions, negative candidates harder ones.
• Halo effect. One strong (or weak) trait spills over into the rating of unrelated traits.
• Memory degradation. By debrief time, interviewers remember vibes, not specific evidence.

The Google re:Work team published research showing that adding a fifth interviewer in an unstructured loop adds almost zero new signal — because by interviewer five, the variance is just noise.

Step 1: Build a competency model

Identify 4–6 competencies the role actually requires (not 15). For an engineer, these might be: technical problem-solving, system design, code quality, collaboration, communication. Be specific.

Step 2: Write behavioral and situational questions

For each competency, write 2–3 questions. Behavioral: "Tell me about a time you…" Situational: "How would you approach…" Tie questions directly to evidence the candidate must produce.

Step 3: Build a scoring rubric

For each competency, define what a 1, 3, and 5 looks like. Use anchored examples drawn from the role. Without rubric anchors, ratings drift and inter-rater agreement collapses.

Step 4: Assign questions to interviewers

Split competencies across the loop so each is covered by 1–2 interviewers, not all 5. This forces specialization and reduces redundant signal.

Step 5: Require independent ratings before debrief

Each interviewer submits ratings and written evidence before joining the group debrief. This is the single most important step. Group discussion before independent rating produces anchoring within 90 seconds.

Step 6: Train and calibrate interviewers

Run mock interviews against video or transcript samples. Compare ratings across interviewers. Anything where ratings disagree by more than 1 point is a calibration issue.

Validity coefficients from Schmidt & Hunter (1998) and updates by Schmidt, Oh, and Shaffer (2016). A coefficient of 1.0 would be perfect prediction; 0.0 is chance.