In the Age of AI Scientists, What Will Researchers Do?
Sent Oct 8, 2026
A paper written by AI has passed peer review at a conference
💡 This Week’s Newsletter Summary
- A team of researchers, including Sakana AI, created “The AI Scientist,” a system that comes up with ideas, runs experiments, writes papers, and even reviews them itself.
- A paper written by this system has passed the acceptance threshold in the blind review for the ICLR 2025 workshop.
- As the performance of the AI models used within the system (such as GPT, Claude, and Gemini) improved, the paper scores also increased. However, these scores were assigned by AI reviewers, not humans.
- However, it was a workshop with a 70% acceptance rate, and out of the three papers submitted, only one was accepted. None of the papers were of the standard of the main conference.
Why was this study conducted?
AI has a long history of assisting science. It has provided support in various fields, such as identifying chemical structures and proving mathematical theorems. Furthermore, since the advent of large language models (LLMs), its role has expanded to include generating hypotheses, organizing literature, and writing experimental code. A prime example is “Coscientist,” featured in Nature in 2023—an AI system that, given only a goal, can design experiments and carry them out directly using automated equipment.
However, there was one slight drawback: there was no system that could carry out the entire process—from selecting a topic to submitting a paper—on its own. This paper aims to fill that very gap.
This Is How an AI Scientist Conducts Research
“The AI Scientist” is a system that generates ideas, runs experiments, writes papers, and even reviews them itself, carrying out its work in the following four steps.

The AI Scientist’s Workflow
① Brainstorming
Within a given field, it continuously generates research directions and hypotheses and adds them to a list. The system is programmed to act like an “ambitious AI Ph.D. student who wants to publish meaningful papers.” For each idea, it assigns a title, rationale, and experimental plan, and self-scores it on a 10-point scale for interest, novelty, and feasibility. It then searches Semantic Scholar using various search terms to find existing papers and discards any ideas that are too similar to previous research.
② Conducting the Experiment
It writes code, runs it, and if an error occurs, it reads the error message and fixes it itself. It conducts preliminary experiments to verify that the idea works, adjusts the hyperparameters, and then proceeds with the experiment. Finally, it conducts ablation experiments, removing components one by one to observe the effects. At each stage, it runs multiple trials simultaneously and selects the best result to move on to the next stage.
The AI Scientist also checks the graphs. It shows the graphs generated from the experiment results to the Vision Language Model (VLM), and if the axis labels are unclear, the legend is missing, or the graph is misleading, it flags the experiment as problematic and has it corrected. It runs the same experiment multiple times with different random seeds to calculate the mean and standard deviation. All results are recorded like an experiment notebook and used for future planning and paper writing.
③ Writing the Paper
Based on its lab notes and graphs, it fills out the conference paper template (LaTeX) section by section. When writing about related research, it runs up to 20 searches on Semantic Scholar, and for each citation, it makes a separate note explaining why it is citing that particular paper. After writing the first draft, it revises it several times on its own. It incorporates the suggestions from the LaTeX syntax checker and has the graphs and captions double-checked to make sure they correspond.
④ Self-Review
The completed manuscript is evaluated by an AI reviewer called “Automated Reviewer” in accordance with the NeurIPS review guidelines. OpenAI’s o4-mini generates five independent reviews and then synthesizes them—much like a conference area chair—to determine whether the paper is accepted. As in a real conference, the reviews include assessments of soundness, presentation, contribution, an overall score (out of 10), as well as strengths and weaknesses.
Better Model → Better Paper
The research team had the system write papers by simply switching out the AI models within it, and then had the “Automated Reviewer”—the AI reviewer described earlier—grade the papers. Scores rose significantly as the models progressed from GPT-4 in 2023 to Claude Sonnet 4 and Gemini 2.5 in 2025.

Source: Lu et al., Nature 2026, Fig. 1b
The research team draws two conclusions from these results. First, using the latest model already produces papers that are on the verge of being accepted to the workshop, as judged by AI reviewers. Second, they predict that as models improve in the future, this system will improve as well.
However, they did add a caveat: while progress has been made on “hallucination” issues—where AI is easily deceived or confidently provides incorrect answers—solving them has proven more difficult than expected. As long as these weaknesses remain, it is difficult to trust the system’s results at face value. They added that it is still unclear whether AI will be able to generate creative ideas, such as major conceptual leaps in science.
After going through a real conference review
The fairest way to evaluate AI papers is to have them peer-reviewed just like human-authored papers. The authors called this the “AI Scientist Turing Test.”
The research team submitted to the ICBINB workshop at ICLR 2025. As the name “I Can’t Believe It’s Not Better” suggests, this is a forum for discussing cases where deep learning has not lived up to expectations. After obtaining IRB approval and securing the cooperation of ICLR and the workshop organizers, the research team submitted three papers written by AI. The reviewers were only aware that some of the papers were written by AI; they did not know which ones. Among the 43 papers reviewed, three were written by AI, and one of them scored 6, 7, and 6 (an average of 6.33), exceeding the acceptance threshold.
However, the research team had decided from the outset to withdraw any paper generated by AI, regardless of the peer review outcome, and they did in fact withdraw the paper before the meta-review. The organizers explained that if it had not been withdrawn, it would very likely have been accepted.
That said, it would be a stretch to claim that “AI is now producing ICLR-level papers.” While the acceptance rate for this workshop was 70%, the acceptance rate for the main ICLR conference was 32%, and the research team itself assessed that none of the three papers met the standards of the main conference.
How will the work of researchers change?
The faster we can run experiments and write drafts, the more time we’ll have to devote to other questions—questions like whether this idea is worth investigating, whether the results properly validate the hypothesis I’ve set forth, and on what grounds we can trust the results generated by AI.
If you implement a system like this in your lab, you need to design verification procedures alongside the processes for assigning tasks. This means keeping experiment logs and code, independently reproducing key results, and verifying that claims align with their supporting evidence.

The more automation there is, the more important verification becomes.
This paper demonstrated that it is possible to automatically generate a manuscript by stringing together various stages of AI research and that such a manuscript can gain a certain degree of recognition in external peer review. However, ensuring the reliability of the results and the rigor of the methodology remain challenges.
When considering research automation in the future, there are factors to consider before thinking about how quickly it can be developed. These include whether the results are reproducible, whether the evidence is accurate, and whether it actually contributes to our knowledge.
In an era where even research is automated,
The best researcher is not the one who “creates well,” but the one who “verifies most rigorously.”