In the Age of AI Scientists, What Will Researchers Do?
In the Age of AI Scientists, What Will Researchers Do?
A paper written by AI
It passed conference peer review.
AI has a long history of assisting science. It has provided support in various fields, such as identifying chemical structures and proving mathematical theorems. Furthermore, since the advent of large language models (LLMs), its role has expanded to include generating hypotheses, organizing literature, and writing experimental code. A prime example is “Coscientist,” published in Nature in 2023, which demonstrates an AI capable of designing experiments and carrying them out directly using automated equipment—all based solely on a given objective.
However, there was one slight drawback: there was no system that could carry the entire process on its own—from choosing a topic to submitting a paper. This paper aims to fill that very gap.
Sakana AI’s “The AI Scientist” is a system that generates ideas, runs experiments, writes papers, and even reviews them on its own, carrying out its work in the following four steps.
① Brainstorming If you enter the prompt “An ambitious AI Ph.D. student who wants to publish a meaningful paper” into the system, the AI will generate hypotheses and research directions on its own. It will self-assess the originality and feasibility of the ideas on a scale of 1 to 10 and automatically filter out topics that overlap with existing papers.
② Experimentation If an error occurs while running the code, it reads the error message itself and even handles the debugging. The most interesting part is that a vision-language model (VLM) directly reviews the graphs generated from the experimental results. It flags graphs with blurry axis labels or missing legends as “problematic” and has them redrawn; it also repeats the experiments multiple times and neatly records the mean and standard deviation.
③ Writing the Paper Based on these organized lab notes, it writes the manuscript according to the conference format (LaTeX). When citing related research, it shows meticulous attention to detail, going so far as to explain the reason for citing each paper individually. Once the draft is complete, it not only runs a grammar check but also verifies that the graphs and the accompanying text align, and revises the manuscript multiple times.
④ Self-review Finally, an AI reviewer—‘Automated Reviewer’—conducts an evaluation in accordance with the conference’s official review guidelines. After generating five independent reviews, it synthesizes them—much like a conference Area Chair would—to identify the paper’s strengths and weaknesses and determine whether it should be accepted.
A Better Model
→ A Better Paper
The research team had the system write papers by simply switching out the AI models within it, and then had the “Automated Reviewer”—the AI reviewer described earlier—grade the papers. Scores rose significantly as the models progressed from GPT-4 in 2023 to Claude Sonnet 4 and Gemini 2.5 in 2025.
The research team draws two conclusions from these results. First, using the latest model already produces papers that are on the verge of being accepted to the workshop, as judged by AI reviewers. Second, they predict that as models improve in the future, this system will improve as well.
However, the team did add a caveat: while progress has been made on “hallucination” issues—where AI is easily deceived or confidently provides incorrect answers—solving them has proven more difficult than expected. As long as these weaknesses remain, it is difficult to trust the system’s results at face value. They added that it is still unclear whether AI will be able to generate creative ideas, such as major conceptual leaps in science.
After going through a real conference review
To demonstrate the value of AI papers, the research team conducted the “AI Scientist Turing Test,” in which AI papers are peer-reviewed in the same way as human-authored papers. They secretly submitted three AI papers to the ICLR 2025 workshop, and one of them scored an average of 6.33, surpassing the acceptance threshold. Although they withdrew it immediately as planned, it effectively made it into the acceptance range.
However, it is difficult to conclude that “AI can write papers for top-tier conferences.” This is because the workshop in question had a high acceptance rate of 70% (compared to 32% for the main conference), making the bar low, and the research team itself acknowledged that the submitted papers did not meet the standards of the main conference. This experiment demonstrates both the potential and the practical limitations of AI scientists.
How will the work of researchers change?
The faster experiments and drafting become, the more time researchers can devote to fundamental questions, such as the value of their ideas or the reliability of their results. Therefore, when implementing AI systems, it is crucial not only to delegate tasks effectively but also to design “verification procedures”—such as maintaining experiment logs, reproducing key results, and validating evidence—alongside the system.
The more automation there is, the more important verification becomes.
This paper demonstrated that it is possible to automatically generate a manuscript by stringing together various stages of machine learning research and that such a manuscript can gain a certain degree of recognition during external peer review. However, the reliability of the results and the rigor of the methodology remain areas for further improvement.
When considering research automation in the future, there are factors to consider before asking how quickly it can be developed. These include whether the results are reproducible, whether the evidence is accurate, and whether it actually contributes to our knowledge.
Tags
- #artificial-intelligence
- #ai-papers
- #ai-scientist
- #sakana-ai
- #iclr
- #llm










