Commit

9ba65ae

I tried distilling a width-256 LM from a width-768 one, but when I measured it with a ruler calibrated for the changed width, it only came out to the equivalent of 1.22x the compute. There isn't much room left to draw out of a small model through training tricks, and the capacity seems to have already plateaued.
Translated from Japanese

commit 9ba65ae