FORRT AI in Metascience Online Conference

When AI writes the literature

The problem as it already shows up in the correction record, the pressure producing it, and two routes to a better literature.

• AI-Generated Content as an Emerging Driver of Journal RetractionsAnton Sokolov (Tallinn University of Technology)10 min
• When every hypothesis is published: agentic AI and the collapse of selection at productionMatt Spick (University of Surrey)20 min
• The Conceptual Armament Effect: Armed AI Reaches Above a Human 99th-Percentile PaperKenji Yamada (independent researcher, Japan)20 min
• How should AI-assisted research be documented? Seven guiding questionsMaximilian Frank (LMU Munich)20 min

Evidence synthesis: capability, comparison, assurance

From what frontier models can extract, through a head-to-head test of the tools built on them, to the safeguards that would make an AI-assisted review checkable.

• Can frontier LLMs code studies for meta-analysis?Shaheed Azaad (University of Münster)20 min
• Fast, but incomplete and inaccurate? Performance of AI-powered tools for systematic literature reviewsRene Bekkers (VU Amsterdam)20 min
• ReviewAid: confidence-scored, locally runnable LLM screening and extractionVihaan Sahu (Georgian National University SEU)10 min

Evaluating science, and its evaluators

What human review actually does, what a computational novelty signal might add, and a tool that checks a paper against what its authors said they would do.

• Gender and disciplinary differences in 39,280 SNSF grant peer review reportsStefan Müller (University College Dublin)20 min
• Tracing Scientific Novelty in Transformer Representations: Toward Computational Models of Novelty SignalsAnna Kruzenshtern (LUT University)20 min
• RegCheck: an LLM-assisted tool for comparing registrations with papersJamie Cummins (University of Bern)20 min

Conditions for legitimate adoption

What researchers do with these tools, what open infrastructure and open data would require, who gains or loses when the tools arrive, and the rules of use that follow.

• How health researchers actually use GenAI: findings from a global surveyDanny Maupin (University of Surrey)10 min
• Small open-weight models as the path to genuinely open AI-assisted metasciencePeder Mortvedt Isager20 min
• washopenresearch: an open dataset for studying data-sharing practices in WASH researchLars Schöbitz (ETH Zurich)20 min
• AI and researchers with Specific Learning Disabilities: access gains and new inequitiesChelsea Sawyer (University of Manchester)20 min
• The FOCUS Framework for Optimal and Careful Use of Generative AI in ScienceMohamed Helmy (University of Saskatchewan)20 min

Closing provocation

A challenge to the premise that complex problems are the kind of thing better tools solve.

• Rethinking how AI is approached to address wicked problems and global challenges in a polycrisis worldBatool Almarzouq (University of Edinburgh)20 min

Reproducing the record

Replication and reproduction as work actually done, with agents in the loop and with the paperwork that makes a result reusable, followed out to whether anyone downstream notices.

• Thirty agent-assisted replications, published as nanopublications: the honest debriefAnne Fouilloux (Lifewatch ERIC, Spain), Jean Iaquinta (Vitenhub AS, Norway)20 min
• A local, open-weights pipeline for semi-automated replication checklistsJohannes Breuer (CAIS Bochum & University of Duisburg-Essen), Mario Haim & Philipp Knöpfle (LMU Munich), Xinyue Zhao (CAIS Bochum)20 min
• Do failed replications make their way into public knowledge?Youyou Wu (University College London)20 min

Symposium: how can AI help us understand animal-to-human translation in drug development?

Four talks and a discussion on the oldest replication question in biomedicine — whether a finding survives the move from an animal model to a human trial. Chaired by Benjamin Ineichen.

• Miriam Kober, Tim Chopard, Vera Bernhard & Simona Doneva (University of Bern)90 min

Getting our hands dirty: experimenting with agentic reproductions and robustness checks

The practical counterpart to the talks either side of it. Take one published analysis, ask a coding agent to reproduce it and to check how robust it is, and see where it works and where it fails quietly. No prior experience needed.

• Lukas Wallrich (Birkbeck, University of London)50 min

Symposium: predicting replicability

Three talks and a panel on forecasting which findings will hold, and what such forecasts are for. Convened by Andrew Tyner.

• The Predicting Replicability Challenge: three rounds of automated credibility assessmentAndrew Tyner (Center for Open Science)20 min
• What Are Automated Replicability Predictions For?Lukas Röseler (University of Münster)20 min
• Multi-agent AI measures of methodological complexity help forecast study reproducibilityDaniele Fanelli (Heriot-Watt & LSE), Yuan Yingfang (Northumbria & Heriot-Watt)20 min

Machines that build and test the replication record

First the benchmarks that measure what these agents can actually do, then the two databases being built on that basis with humans kept in the loop, and last the question the day has been circling: can an agent run the checks at human quality?

• ReplicatorBench: an end-to-end benchmark for AI agents in research replicationBang Nguyen (University of Notre Dame), Dominik Soós (Old Dominion University)20 min
• Can AI reviewers catch the errors that matter? A 100-error planted benchmarkPaul Litvak (Robyn Dawes Institute)20 min
• Keeping humans in the loop: using LLMs to enable replication database validation at scaleHamidreza Behbood (University of Münster), Rohan Tondlekar (FORRT)10 min
• The Metascience Observatory: an AI-built replication database and Bird's Eye ReviewsDan Elton (The Metascience Observatory)20 min
• Can an agentic LLM run computational reproducibility checks at human quality?Yoel Inbar (University of British Columbia)20 min

• How to use PreCheck: An AI-Assisted Tool and Evaluation Framework for Preregistration QualityBence Palfi (Goldsmiths, London)90 min

• A personal literature wiki, written and maintained by an AI agentYu Heng Daryl Lee & Tao Ma (UCL)90 min

Does the finding survive another analyst?

The other way a result fails to hold. The study is not repeated badly; a defensible alternative analysis simply lands somewhere else. And the alternatives increasingly consult the same model.

• Shared model dependence as a source of correlated error in evidence synthesisAlyssa Columbus (Johns Hopkins & LMU Munich)20 min
• PaperRobust: a multi-agent framework for assessing analytical robustnessQian Ma (Penn State)10 min
• Multiverse Analysis in the Age of Large Language ModelsArdin Kropff, Tygo van Beek & Steven Verheyen (Erasmus University Rotterdam)10 min

• HubMeta: an AI-assisted workflow for systematic review and meta-analysisPiers Steel (University of Calgary), Hadi Fariborzi (Mount Royal University)90 min