# How to evaluate AI mortgage QC software

Updated September 26, 2026

Test it blind on loan files your team has already reviewed, and judge it against what the agencies still hold you responsible for. A tool is worth buying only if your reviewers can check every finding it makes and its accuracy holds up on your own files.

## Key numbers

- **10%** of a vendor's post-closing sample the lender must re-review itself each month (D1-1-02)
- **3 months** of defect trending, at least, in each monthly management report (D1-1-03)
- **3 years** minimum retention for QC records

## Start from what stays yours

The QC topics of the Selling Guide (Chapter D1) don't mention AI or software. The outsourcing rules apply whenever a vendor performs any part of the QC file review, and they are blunt. Fannie Mae holds the lender "fully accountable" for its QC program whether the work is done in-house or by a vendor, and a vendor contract "is not a substitute" for the lender's own QC plan (D1-1-02).

So the vendor reviews files under your plan, and its results feed your reporting, your severity levels and your corrective action. Each month you must re-review at least 10% of the vendor's post-closing sample yourself, loans with and without defects. That review "may not be contracted out," and its report must show concurrence rates and discrepancies. QC staff must be independent of production, underwriting and closing, or the plan must explain the offsetting controls.

If your own staff run the software, the output is still your QC review and has to meet the same D1 standards. Every question below follows from that.

## Run a blind pilot

The Guide says nothing about how to test a tool. Common practice is a blind side-by-side pilot on files your team has already reviewed: the vendor sees the files but not your results. Include clean files, known highest-severity defects, and your hardest products. Then score four things:

- agreement: your findings the tool also raised, divided by all your findings
- missed findings (false negatives): your findings the tool didn't raise, where a missed highest-severity defect weighs far more than a missed minor one
- false positives: tool findings your reviewers reject after checking the evidence
- new valid findings: tool findings your team missed but confirms on review, counted separately rather than as errors

Have someone who wasn't part of the original review adjudicate disputes.

Here is what a scorecard looks like, with illustrative numbers. A lender runs the pilot on 20 already-reviewed files. Its QC team had recorded 30 findings, including 4 files rated not eligible as delivered.

| Measure | Count | Calculation | Result |
|---|---|---|---|
| Your team's findings | 30 | | |
| Tool findings | 34 | | |
| Matched findings | 26 | | |
| Missed findings | 4 | 30 − 26 | |
| New valid findings (confirmed on review) | 3 | | |
| False positives | 5 | 34 − 26 − 3 | |
| Agreement with your findings | | 26 ÷ 30 | 86.7% |
| Tool findings that held up | | (26 + 3) ÷ 34 | 85.3% |
| False positives per file | | 5 ÷ 20 | 0.25 |
| Highest-severity files caught | 3 of 4 | 3 ÷ 4 | 75% |

Agreement looks strong, but one of the four ineligible loans slipped through. Find out why before you decide: an uncovered program, an overlay, or a misread document. The 3 new valid findings are worth checking against your own review process.

## Look hard at a single finding

The Guide doesn't define the format of a finding either. In practice, a finding your reviewer can defend in a rebuttal, and hand to a business unit for correction, shows five things: the document and page that support it; the guideline section or overlay, with the edition in force for that loan; the inputs and arithmetic where a number is involved; the effect on the loan, such as whether it makes the loan ineligible as delivered; and the document or correction that would clear it.

Every missing piece costs reviewer time. It also matters when you self-report, because the lender must send Fannie Mae a summary of its findings with copies of the supporting documents (D1-1-01). Ask the vendor to run a file you supply and show you the findings.

## Questions to put to every vendor

1. Which programs and overlays does it test? Your QC sample must cover all channels, all products (including special or niche programs) and all underwriting methods (D1-1-01). Ask for a rule inventory by program with edition dates, covering what you originate: Fannie Mae, Freddie Mac, FHA, VA, USDA and non-QM. Ask how your investor and internal overlays are added and tested, and pilot every program in your mix, not only conventional loans.
2. How fast are guideline changes applied? A finding should cite the rule in force for that loan, which is common practice. Ask for the change log with dates.
3. Does it support reverification? Random full-file reviews require reverifying income and employment, tax returns (IRS transcripts), assets, credit history, property eligibility and value, and occupancy. Differences from the underwriting data require re-underwriting, and failed attempts must be documented and tracked (D1-3-03). Ask for a demo that reconciles reverification documents with the origination data and records unsuccessful attempts.
4. Can it produce the monthly report? Post-closing reports go to management in writing at least monthly, with sample descriptions, at least three months of defect trending, the highest-severity rate against target at least quarterly, a split between compliance-with-law and underwriting or eligibility defects, and the 10% vendor review results where applicable (D1-1-03). Categories must stay consistent month to month. Ask for a sample report, and for an export that supports loan-level re-review of the 10% sample.
5. Is the output reproducible, and who signs off? Run the same file twice. Ask how model and rule changes are versioned and see the audit log and sign-off workflow. A reviewer should confirm each finding before it enters your QC record, and the required QC audit checks that conclusions are recorded and applied consistently (D1-1-01). These model-risk questions are common practice, not agency rules.

## Security and data terms

The Guide requires an information security program and written procedures for approving and managing vendors (A3-4-01, A4-1-01). Borrower information can go to contractors only on a need-to-know basis and under confidentiality terms at least as strict as the Guide's. QC records must be kept at least three years.

Beyond that, it's common practice to request an independent SOC 2 Type II report, the data processing terms and the subprocessor list, and to ask whether customer data trains models (and whether you can opt out), how each customer's data is isolated from other tenants, how long data is retained and how it's deleted, and which subprocessors, including model providers, receive loan data.

Hold every vendor to the same method, Flightline included: run it blind on files you have already reviewed and score it as above.

## FAQ

**What is the best way to compare AI mortgage QC tools?**
Run each one blind on the same set of files your team has already reviewed, and compare agreement, missed findings and false positives. This is common practice, not an agency rule.

**Does using software change the lender's QC responsibility?**
No. Fannie Mae holds the lender fully accountable for its QC program whether the work is done by the lender or a vendor (D1-1-02).

**How many files should a pilot include?**
The Guide sets no number. Common practice is enough files to cover each program in your mix, with both clean files and files with known highest-severity defects.

**Can software replace the lender's 10% vendor re-review?**
No. When a vendor performs post-closing QC reviews, the lender must itself re-review at least 10% of that sample each month, and that work may not be contracted out (D1-1-02).

**Does Fannie Mae require a SOC 2 report from QC vendors?**
The Guide doesn't name SOC 2. It requires an information security program and written vendor management procedures (A3-4-01, A4-1-01). Requesting a SOC 2 Type II report is common diligence practice.

## Sources

- Fannie Mae Selling Guide, published September 2, 2026: https://selling-guide.fanniemae.com/
- D1-1-01, Lender Quality Control Programs, Plans, and Processes (04/01/2026): https://selling-guide.fanniemae.com/sel/d1-1-01/lender-quality-control-programs-plans-and-processes
- D1-1-02, Lender Quality Control Staffing and Outsourcing of the Quality Control Process (04/01/2026): https://selling-guide.fanniemae.com/sel/d1-1-02/lender-quality-control-staffing-and-outsourcing-quality-control-process
- D1-1-03, Lender Quality Control Reporting (04/01/2026): https://selling-guide.fanniemae.com/sel/d1-1-03/lender-quality-control-reporting
- D1-3-03, Lender Post-Closing Quality Control Reverifications (04/01/2026): https://selling-guide.fanniemae.com/sel/d1-3-03/lender-post-closing-quality-control-review-data-integrity
- A3-4-01, Confidentiality of Information (02/05/2025): https://selling-guide.fanniemae.com/sel/a3-4-01/confidentiality-information
- A4-1-01, Maintaining Seller/Servicer Eligibility (08/05/2026): https://selling-guide.fanniemae.com/sel/a4-1-01/maintaining-sellerservicer-eligibility

This guide summarizes agency requirements as of the date above. It is not legal advice.