Risk Stratification in Primary Care: Value-Based Contributions of Provider Adjudication

Brian C. Ricci, Jonathan Sachs, Konrad Dobbertin, Faiza Khan, David A. Dorr

Research output: Contribution to journalArticlepeer-review


Background: In primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes. Objective: We assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm. Design: (1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses. Participants: Primary care patients aged 18 years and older with an adjudicated risk score. Approach and Main Measures: (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models. Key Results: 47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15–1.34), age > 65 (OR 2.55, CI 2.24–2.91), Black race (1.26, CI 1.02–1.55), polypharmacy >10 medications (OR 4.87, CI 4.27–5.56), a positive depression screen (OR 1.57, CI 1.43–1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52–2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model. Conclusions: Provider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.

Original languageEnglish (US)
Pages (from-to)601-607
Number of pages7
JournalJournal of general internal medicine
Issue number3
StatePublished - Feb 2022


  • healthcare utilization
  • mortality
  • patient care management
  • population health
  • primary health care
  • racism
  • risk assessment
  • value-based care

ASJC Scopus subject areas

  • Internal Medicine


Dive into the research topics of 'Risk Stratification in Primary Care: Value-Based Contributions of Provider Adjudication'. Together they form a unique fingerprint.

Cite this