Peer Review Isn’t the Point — Scientific Inquiry Is

A lot of the blame should fall on misguided narratives (common across academia) about what makes science work, which have been entrenched by the best-funded scientific institutions. A core scientific norm is that disputes should be resolved with reference to concrete empirical tests or rigorous proofs, judged by the scrutiny of one’s scientific peers. But that’s very different from the idea that ideas should be developed via paper-sized units of work which are each individually defended and justified. Historically speaking, the latter simply isn’t how the best science happened—Newton and Smith and Darwin developed their ideas via writing books and letters rather than peer-reviewed papers (and even Einstein didn’t encounter peer review until decades after his main breakthroughs). But the requirement to “publish or perish” is now so entrenched across academia that it produces strong streetlight effects.

[…] In this 1995 commentary on NIPS (now NeurIPS), a statistician frustrated by how “everyone wants to be a theorist” writes that “mathematical theory is not critical to the development of machine learning. But scientific inquiry is.” He characterizes scientific inquiry as “sensible and intelligent efforts to understand what is going on”, and gives overparameterization of neural networks as a central example of a good target for scientific inquiry. Yet only after the rise of deep learning did phenomena like grokking, deep double descent, and memorization of random labels render this omission too blatant to ignore (though I’m uncertain about how much real progress subsequent theoretical work has made).