I haven’t read the paper in dept, but these are deeply unserious reasons for disparaging it.
Bad research is still bad. The main paper is 9 pages with wide margins, outside of references. This was a few college students publishing their CS research assignment
The length of this paper entirely within the norm for scientific publications. In fact, depending on the type of paper, they can be much shorter. For example, Nature expects submissions at around 2500 words or around 4300 words, depending on the format, while this paper is around 4800 words long (excluding references and SI).
The authors also obviously not “a few college students publishing their CS research assignment”: Gloaguen, Mündler, Raychev, Müller, and Vechev.
The “Limitations” section is rather telling:
Another interesting avenue opened by this work is how to improve the automatic generation of useful context files.
Okay, so they already admitted that they started with useless context files, which sabotages their own test.
You know how you fix useless context files? You iterate until they are useful context files!
The entire point of the study was to see what kind of context files were useful and which were useless, which according to the authors was not something that had been done before:
A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents’ task completion performance in two complementary settings
They found that,
while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful.
So when you say that they “admitted that they started with useless context files”, you are referring to the results of the study. Complaining that they didn’t base their study on the conclusions of that very study is just absurd
I haven’t read the paper in dept, but these are deeply unserious reasons for disparaging it.
The length of this paper entirely within the norm for scientific publications. In fact, depending on the type of paper, they can be much shorter. For example, Nature expects submissions at around 2500 words or around 4300 words, depending on the format, while this paper is around 4800 words long (excluding references and SI).
The authors also obviously not “a few college students publishing their CS research assignment”: Gloaguen, Mündler, Raychev, Müller, and Vechev.
The entire point of the study was to see what kind of context files were useful and which were useless, which according to the authors was not something that had been done before:
They found that,
So when you say that they “admitted that they started with useless context files”, you are referring to the results of the study. Complaining that they didn’t base their study on the conclusions of that very study is just absurd