How to Match Resumes to Job Descriptions with LLMs
A trustworthy resume-matching system doesn’t ask an AI model “how good is this candidate?”. It asks narrow questions, “where does this resume show this skill?”, and adds up the answers with rules your team can read. That makes results explainable, consistent and testable for bias.
The short answer: parse the job description into a weighted skill list and each resume into structured sections. Mask protected details. Use hybrid search to shortlist candidates cheaply, then ask a language model to extract evidence for each skill with a quoted line. Score with a fixed formula, not a free-form model opinion. Test the whole pipeline on a recruiter-labelled golden set and on matched resumes for bias. Trigger any follow-up through a tool layer that waits for a recruiter’s approval.
This is the architecture behind our business guide, AI Resume Screening. It is written for HR-tech CTOs and teams building matching into an applicant tracking system (ATS), and it reflects how we design agents in our Agent Engineering work.
The architecture at a glance
Step 1: Parse the job description
Turn free text into a schema the rest of the pipeline can use:
| Field | Example |
|---|---|
| Skill | Financial reporting |
| Level | Must-have or nice-to-have |
| Weight | 1 to 3, agreed with the hiring manager |
| Minimum experience | 3 years, if stated |
| Synonyms | Management reporting, MIS reporting |
| Source quote | The line in the job description it came from |
Normalise skills against a skill taxonomy, your own or a public one such as the EU’s ESCO list, so “JS”, “JavaScript” and “ECMAScript” mean the same thing. Show the parsed list to the hiring manager before matching. A wrong skill list makes every downstream score wrong.
Step 2: Parse the resumes
Resumes arrive as PDF and Word files in every layout. Parse them into sections (summary, roles, projects, education, certifications) and build a timeline of roles with start and end dates, so experience can be computed rather than guessed. Keep line and page positions for every piece of text: the evidence step depends on them.
Then mask the fields that should not affect scoring: name, photo, contact details, date of birth, graduation years, gender markers and similar personal details. Store the mapping securely so the recruiter can unmask after shortlisting.
Step 3: Pre-filter with hybrid search
Running a strong language model over every resume for every role is slow and expensive. Instead, index resumes for both keyword search and meaning (vector) search, as described in our hybrid RAG guide, and pull a candidate pool for each role. Set the pool wide enough that good candidates aren’t cut here; this step only saves cost, it doesn’t decide.
Step 4: Extract evidence, skill by skill
This is where the language model earns its place. For each candidate and each skill, ask a narrow question and require a structured answer:
- evidence level: none, mentioned, applied, or led;
- quoted span: the exact resume text, with its line reference;
- duration: how long, from the role timeline;
- confidence and a one-line reason.
Validate the output: the quoted span must actually exist in the resume, or the evidence is discarded. This one check removes most invented evidence. Use a smaller, cheaper model for parsing and a stronger one for evidence extraction.
Step 5: Score with rules, not opinions
Compute the score with a fixed, readable formula: evidence level times skill weight, plus experience where it matters, with must-have gaps shown clearly rather than averaged away. Because the formula is fixed, the same evidence always gives the same score, and recruiters can see exactly why one candidate ranks above another.
Step 6: Test for bias, on every change
Bias testing belongs in the release pipeline, not in an annual report.
- Matched pairs. Create resumes that are identical except for name, gender, age or background markers. Scores should not move.
- Selection rates. On a realistic test set, compare shortlist rates across groups. The ratio of one group’s rate to the highest group’s rate, often called the impact ratio, is the measure bias audits such as New York City’s use.
- Recruiter overrides. Log every time a recruiter moves a candidate up or down, and look for patterns.
Our post How to Test AI Agents covers how to wire these into an automatic gate.
Step 7: Act through a tool layer, with approval
After shortlisting, the agent can draft interview invitations, scheduling messages and rejections, and update ATS records. Route every action through a tool layer that:
- gives the agent only the permissions it needs (read candidates, draft messages, update status);
- holds every outgoing message for recruiter approval;
- logs who approved what and when.
Pin the model version and version prompts, taxonomy and formula together, so you can roll back the whole bundle if a change hurts quality. Our post DevOps for AI Agents explains the release bundle.
How to measure it
| Metric | Plain meaning |
|---|---|
| Shortlist precision | Share of the top candidates that recruiters agree belong there |
| Recall of good candidates | Share of recruiter-picked candidates the system also shortlisted |
| Evidence accuracy | Share of quoted evidence that truly supports the skill |
| Consistency | Same resume, same role, several runs: same score |
| Bias measures | Score shift on matched pairs; impact ratio by group |
| Cost per resume | Tokens and time per candidate per role |
Common mistakes
- Asking the model for a single overall score.
- Not checking that quoted evidence exists in the resume.
- Masking after scoring instead of before.
- Testing bias once at launch and never again.
- Letting the agent message candidates without approval.
How SHVX can help
Building matching into your ATS? We’ll review your pipeline against this checklist: parsing, evidence, scoring, bias tests and approvals.
Learn more about Agent Engineering.
FAQ
Why not just use embeddings to match resumes?
Embedding similarity is good for finding a candidate pool, but it can’t explain itself or check must-have skills. Use it to pre-filter, then extract evidence.
Should the LLM produce the final score?
No. Let it extract evidence; let a fixed formula score it. That keeps results consistent and explainable.
How do we test for bias?
Run matched resumes that differ only in protected details, and compare selection rates across groups on a realistic test set, on every change.
Which model should we use?
Test two or three on your own labelled data. A small model often handles parsing; evidence extraction usually needs a stronger one.