Overview
This report looks at the effectiveness of AI-based prediction models depending on the quality of underlying data, and how observed insurance data may not measure intended risk cleanly. The research looks at applications for various insurance sectors; indirectly associated variables, measurement bias and group disparities; predictive models and mitigation approach; and evaluation design and group-audit metrics.
More research and analysis are needed to understand this emerging area. Actuaries are helping identify model usage to help inform the risk assessment process.
Key Findings
The guiding question informed in this report is whether when observed insurance data contain group-correlated measurement distortion or indirect association, can model design improve the alignment between model outputs and underlying risk while preserving useful predictive performance. The research looks at applications in various areas including life insurance, health insurance, credit insurance, mortgage insurance and auto insurance.
Observed variables such as territory, ZIP code, credit-related scores, recorded violations, health-care utilization, or morbidity scores may contain useful predictive information, but they may also reflect measurement differences, administrative processes, access barriers, or social and geographic patterns. If a model treats these observed variables as pure risk signals, it may learn distorted information rather than the underlying risk the insurer intends to measure.
- The report treats biased measurement and indirect association as model-quality issues as well as fairness concerns. If a model relies on an observed variable that is systematically distorted relative to underlying risk, the resulting prediction may be less credible as an estimate of the risk being priced.
- The simulations in the report are mechanism-based case studies rather than empirical estimates or legal findings. When observed insurance data are systematically distorted or indirectly associated with group membership, it is typically beneficial to evaluate model-validity concerns and group-level audit outcomes jointly.
Podcast
Evaluating Bias Mitigation Strategies in AI Models for Insurance Pricing and Risk Assessment
Acknowledgements
The researchers’ deepest gratitude goes to those without whose efforts this project could not have come to fruition: the Project Oversight Group and others who generously shared their wisdom, insights, advice, and guidance, and for their diligent work overseeing, reviewing and editing this report for accuracy and relevance.
Project Oversight Group members:
- Dorothy L Andrews, ASA, MAAA, PhD
- Carlos Arocha, FSA
- Joan Barrett, FSA, MAAA
- Adrian Clark, FSA< MAAA
- Andrew Clark, PhD
- Jean-Marc Fix, FSA, MAAA
- Robert Gomez, FSA, CERA, MAAA
- Sven Haunerland, FSA, CERA, ACIA
- Anton Kobelev, ASA, CERA
- Sin Rong (Bernice) Lim, FSA, CERA, FCIA
- Honora Lo, ASA, MAAA
- Jennifer Loftus, FIA, FSIA
- Chris Lombardi, FSA, CERA, MAAA
- Marilyn McGaffin, ASA, MAAA
- Tyson Mohr, FSA, ACAS, MAAA
- Norman Niami, FCAS, MAAA
- Nicole Persson, ASA, MAAA
- Ronald PoonAffat, FSA, MAAA, FIA
- Shisheng (Rose) Qian, FSA, CERA
- Francis Rahil, FSA, CERA
- Bruce B Rosner, FSA, MAAA
- Jason Sears, FSA, CERA
- Igor Shamshtein, FSA
- Ren Zhong, ASA, MAAA
At the Society of Actuaries Research Institute:
- Lisa A. Schilling, FSA, EA, FCA, MAAA, Director, Practice Research
- Barbara Scott, Sr. Research Administrator
FAQs
An adversarial neural network (ADV-NN) is a mitigation model. It is trained to predict the insurance outcome while making a designated observed indirectly associated variable more difficult to recover from the learned representation.
There are many considerations as insurance models do not determine an action for every applicant or policyholder. It can help identify a small subset for manual review. For example, a small distortion in the upper tail of predicted risk can therefore have material consequences for who receives additional scrutiny or less favorable treatment.