Easy Evaluations as Comfort Blanket

You can't go home to the easy answers again

I recently found myself absorbed in a post comparing AI models based on numerical benchmarks. As I scanned the rows of data, a sense of calm snuck over me: It was a world of clear winners and losers, where progress was measured in neatly quantifiable increments. For a fleeting moment, I forgot about the messiness of taking the new capabilities and making products that fit into people’s lives and help them.

It felt good. For a second.

I had fallen prey to the seductive allure of simple rules. It was like returning to a childhood faith, where complex questions about existence had easy answers, and the path forward was clearly defined. The world made sense, even if it was a simplified, incomplete version of reality.

In both scenarios, the appeal lies in the simplicity of the narrative. Of wanting things to be easy to analyze. Both simple benchmarks and simplified religious doctrine offer a framework for understanding the world, providing a sense of order and control amidst the complexity.

It felt comforting for a second to go back to that. It felt alluring.

But I prefer the wilderness.