Flagship 01 · Commercial Risk Systems · Rigorous Decisioning

Loan Underwriting Copilot

A commercial underwriting copilot that scores credit applications with calibrated probabilities, decomposes scores into exact additive SHAP factors, retrieves historical precedent cases with known outcomes, and generates natural language explanations verified against strict numeric grounding guards.

Source on GitHub → UCI Statlog German Credit · 1,000 Applications 233 Tests Passing (228 Unit/Int, 5 Skipped) ChromaDB Vector Precedents

Held-out ROC-AUC

0.786

95% CI 0.72–0.85

Decision Threshold

21.2%

5:1 Asymmetric Cost Matrix

Cost Per Applicant

0.584

vs 0.700 decline-all baseline

Brier Score

0.167

Isotonic Calibrated (isotonic)

Statistical Integrity over Marketing Hyperbole. An ROC-AUC of 0.786 is not 0.85; the 95% bootstrap interval is [0.72, 0.85] on 250 held-out applications, and a baseline logistic regression scores 0.778 on that same holdout. The core contribution of this system is the engineering layer above raw scoring: cost-optimal threshold calibration, additive attribution, adverse action compliance, and automated numerical grounding that rejects hallucinated LLM rationale.

1. The Threshold is the Decision Engine

Under the official German Credit asymmetric loss matrix, approving a bad loan costs 5 cost units, whereas declining a good loan costs 1 cost unit. A standard 0.50 cutoff costs more per applicant than declining every single borrower.

Bayes-Optimal Decision Rule

For probability \(p = P(Y=1 \mid x)\), the expected cost of approving is \(5p\), while the expected cost of declining is \(1(1-p)\). Equating the expected costs yields the theoretical minimum:

p^* = \frac{C_{\text{FP}}}{C_{\text{FP}} + C_{\text{FN}}} = \frac{1}{1 + 5} = \frac{1}{6} \approx 16.7\%

Empirically calibrated over out-of-fold training probabilities, the loss-minimizing operational cutoff shifts to 21.2%, achieving expected cost \(0.584\) vs \(0.700\) for a naive policy.

Live Cost-Matrix Threshold Explorer

Recomputes confusion matrix and expected loss live over all 250 held-out applicants

CUTOFF: 0.212

Decline Threshold (\(p \ge \text{threshold} \implies \text{Decline}\)) 0.212

0.01 (Approve Almost Everyone) 0.99 (Decline Almost Everyone)

Cost / Applicant

0.584

0.116 better than baseline

Approval Rate

61.2%

153 of 250 applicants

Bad Approved (FN)

×5 penalty units each

Good Declined (FP)

×1 opportunity unit each

Expected Cost per Applicant Baseline Decline-All (0.70) Current Threshold Location

2. Individual Applicant Deep-Dive & SHAP Attribution

Inspect any of the 250 held-out applicants. The model score is decomposed into exact additive SHAP log-odds contributions, paired with regulatory adverse-action reasons and ChromaDB vector precedent cases.

Applicant Case File & Explanation Inspector

Additive TreeSHAP decomposition with regulatory reason codes

Select Applicant

Filter Portfolio

P(Default)

—

Decision

—

Actual Outcome

—

Model Verdict

—

Application Attributes

Additive SHAP Feature Attribution (Log-Odds Impact)

SHAP values sum to the difference between the applicant's predicted log-odds and the expected baseline: Show protected attributes

Adverse-Action Notice (FCRA / ECOA Compliance)

Historical Precedents Retrieved from ChromaDB

3. Rigorous Permutation Fairness Testing

A standard naive bootstrap over group approval rate ratios creates a biased estimator that falsely flags noise as disparity. We replace it with an exact label permutation test.

Protected Attribute Disparity Inspector

Protected Demographic Feature

Demographic Group	n	Approval Rate	Bad Rate	Approved Among Non-Defaulters	ROC-AUC

Protected Attribute Ablation Experiment

What happens if protected attributes are eliminated completely? We refit the model without them, recalibrate isotonically, and re-optimize the threshold out-of-fold:

4. Multi-Tiered Architecture & Grounding Verification

Four layered endpoints ensure that lending decisions never depend on LLM uptime, hallucinations, or API latency:

Endpoint	Payload Output	LLM Invoked	Failure Mode
`GET /health`	Model readiness and vector store status	No	Never (Readiness probe)
`POST /score`	Calibrated probability & cost-optimal decision	No	Artifacts missing
`POST /explain`	The above + SHAP factors & adverse action codes	No	Artifacts missing
`POST /ask`	Verified conversational answer with numeric grounding	Yes	Provider down (graceful fallback to verified template)

Three Verification Checks on Every Answer

1. Numeric Grounding: Every single number in the LLM answer is regex-extracted and asserted against the mathematical context.

2. Citation Verification: Every case ID cited must match a record actually retrieved from ChromaDB.

3. Protected Rationale Guard: Protected attributes cannot be framed as causal rationale for declines.

Failing Answers are Replaced, Not Annotated

Showing a hallucinated answer with a disclaimer badge is how bad decisions happen. If an answer fails grounding, the service substitutes a deterministic rule-based template built from verified data and sets grounded: false.

Over 50% of the guard test suite validates that correct answers are not falsely rejected.