Decision theory and statistics
Synopsis
Statistical analysis requires choices about targets, losses, available information, and the consequences of error. Those choices shape both the method and the meaning of its output. Cross-validation uses a fixed sample for both fitting and evaluation, creating a tradeoff between predictive loss and uncertainty in its assessment. Generalized Bayes uses loss-based updating, requiring a distinction between conditional belief and a distribution chosen as a decision rule. Decision theory provides a common framework for making these specifications explicit and for evaluating prediction, model comparison, and policy choice.
Specific topics
- Validation design
- The validation proportion and fold count determine how a fixed sample is used for fitting and evaluation.
- Prediction and evaluation uncertainty
- Validation decisions balance predictive loss against uncertainty in the estimated performance of the learning procedure.
- Generalized Bayes
- Loss-based updating requires a stated interpretation of the baseline distribution, the output, and any model-comparison quantity.
- Statistical decisions
- Policy choice and predictive regret depend on the actions being compared and the consequences used to evaluate them.
Selected papers
When Is Generalized Bayes Bayesian?
A Comment on "Adaptive Treatment Assignment in Experiments for Policy Choice" (2021)
An Entropy–Energy Identity for Predictive Kullback–Leibler Regret in Infinitely Divisible Location Models