3 / 10
Why is RMSE on ratings a poor metric for a recommendation feed?
RMSE measures how precisely a rating is predicted, while a feed shows the top K items. An error on a 4 predicted as 3.5 counts the same as an error that puts a wrong item into the top — yet the user only ever sees the latter. RMSE is also computed over rated items, i.e. items the user already found and rated; the catalogue they never saw is absent from the metric. The famous example: the Netflix Prize winner improved RMSE by 10% and never shipped, because the gain did not convert into behaviour.