Say you’ve piloted a retention programme in a handful of teams. Voluntary attrition was lower than in the comparison group, but the estimate was noisy and the result wasn’t statistically significant. The budget meeting is next week.
I liked how Terence Chau and colleagues tackled a version of this problem in a new NBER working paper on ShotSpotter, a gunshot detection system Chicago used.
They used a geographic regression discontinuity design, comparing shooting fatality rates just inside and just outside the coverage boundary. Their best estimate was a 2.5 percentage point drop in the share of shooting victims who died (p = .286). If that estimate held across the covered districts, it would translate to about 49 fewer deaths at 2024 shooting levels. By their calculation, you’d need roughly 87 years of Chicago data to have 80% power to detect an effect that size with the same design.
The authors used a Bayesian model to turn that noisy estimate into a range of possible net benefits. The estimated chance of coming out ahead ranged from 70.1% to 99.6% across the four scenarios in Figure 3. The 99.6% scenario uses the higher dollar value for a statistical life and assumes up front that ShotSpotter can’t increase the risk of dying after a shooting.
Back to the retention pilot. Say a rollout costs $100k a year and each avoided departure saves about $20k. Five fewer departures a year would cover the cost. I’d want to see how likely we are to beat that number and how much we might lose if we fall short. I’d also rerun the numbers with a lower estimate of turnover costs and a prior that allows the programme to backfire.
P.S. If you’d like to try this with people data, Andrew Marritt’s free online book Bayesian Thinking for People Analytics explains the approach in plain English and provides the code as well. I wrote about it recently here.