Maxim SaplinThe International Conference on Learning Representations (ICRL) is hosted annually. Each year researchers from around the globe are invited to:
a) submit a research paper
b) review the submitted papers
Here're the numbers of submissions:
- 2024: 2272
- 2023: 1584
- 2022: 1095
- 2021: 860
- 2020: 687
The conference has become the subject of another research: Monitoring AI-Modified Content at Scale:
A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
And here's the most visual insight. For some reason (wonder which one 😊) there's an abnormal spike in the frequency of certain words in peer reviews: commendable, innovative, meticulous, intricate, notable, and versatile.
There're 3 more conclusions in the research:
- The reason for the anomaly in word frequencies is due to the use of LLM
- LLM generation went beyond just spell-checking
- The reviews flagged as touched by LLM were of worse quality
Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals.
Curios, what would be the outcomes of researching the papers, not the reviews...
The findings made me think along the following lines:
- Lazy and sloppy work gets amplified by the proliferation of GenAI tools
- Humanity is in its "garbage civilization" stage and last year it leveled up. Сonsumer society started this stage by filling physical space with tons of plastic, packaging, and unrepairable goods thrown away. Then we moved to social media and cameras in the pocket of every human and Instagram and the likes filling data centers with terabytes of digital garbage (and growing energy consumption in the real world). Now we have GenAI...
P.S.> none of this text has been generated or modified by LLM ;)
