Train and check probabilities

A model that says "90% sure" and is right 60% of the time is dangerous to route by. See how training can reward useful probabilities, how one fitted number changes confidence, and why the result still has to be checked.

What you’ll explore

• Supervised cross-entropy is the negative log score, and a strictly proper scoring rule (log, Brier, and for ordered answers the ranked probability score) is maximized in expectation by reporting the true distribution, while accuracy and linear scores are not. Training on a proper score encourages useful probabilities without guaranteeing them, a fitted temperature changes confidence but not ranking, and calibration must be checked on held-out data, including the cases the policy will automate.