We define calibration of ranking models in calibration, the benefit it can bring to prioritize calibration and how to achieve it without affecting normalized cross entropy / AUC metrics.
Thanks for sharing your insight in calibration. I enjoyed reading it. One thing I don’t quite get is why we can calibrate within the model (i.e. adding layers or change loss), since the training data might be up/down sampled or weighted to account for imbalance, or in general any other filters. If we calibrate within the model, then it calibrates to the distribution of training data which is still biased?
Thanks Devin for your comment. This is a great point. I think in the case you want to do calibration and sample the data then you will need to adapt your calibration to account for it or make sure your sampling strategy keep the original distribution. Please note that sampling data for recsys is not the most common as far as I am aware. You only do that if the added data do not provide value.
Thanks for sharing your insight in calibration. I enjoyed reading it. One thing I don’t quite get is why we can calibrate within the model (i.e. adding layers or change loss), since the training data might be up/down sampled or weighted to account for imbalance, or in general any other filters. If we calibrate within the model, then it calibrates to the distribution of training data which is still biased?
Thanks Devin for your comment. This is a great point. I think in the case you want to do calibration and sample the data then you will need to adapt your calibration to account for it or make sure your sampling strategy keep the original distribution. Please note that sampling data for recsys is not the most common as far as I am aware. You only do that if the added data do not provide value.