(2021), still very interesting. Especially the "post-overfitting" training strategy is unexpected.
By xg15 11 hours ago
I remember vaguely that this was observed when training GPT-3 (probably?) as well. Just trained on and on, and the error went up and then down again. Like a phase transition in the model.
By luckystarr 8 hours ago
The low sample efficiency of RL is well explained.
By xg15 11 hours ago
By luckystarr 8 hours ago
By esafak 7 hours ago
By 7 hours ago