How to Read a Paper in the Age of AI

Read in layers, with a goal, and verify by reconstruction. AI triages, tutors and argues; the judgment stays yours.

Ravi Kalia

2026-10-10

The last paper you cited

How much of it did you read?

And the AI summary, if there was one: did it come

• before your reading,
• after it,
• or instead of it?

Ask for hands on each of the three. Don’t comment on the split yet, and don’t treat any answer as the wrong one. The point is to notice which one it was.

Keep the three words in mind: before, after, instead. They are the whole argument of this talk in miniature. The protocol at the end puts AI before your own reading and after it, and never instead of it.

Everything between here and there is the case for that: what the classic reading guides say, how the AI tools map onto them, what the evidence says about how researchers use the tools, and where the tools fail.

AI reading moves fastest in peer review

over half of reviewers now use AI when reviewing, up from 24% in 2024

61% · 45% early-career against senior researchers

77% · 31% China against North America

Top uses: writing the report (59%), summarizing the article’s findings (29%), flagging possible misconduct (28%).

Across almost 5,000 researchers, 63% cite unclear guidelines or lack of training as barriers.

Frontiers survey of about 1,600 researchers, reported January 2026 (NISO); Wiley ExplanAItions, 2025 (NISO)

AI-assisted reading is now mainstream, and it is moving fastest in peer review, where reading closely is the job.

A Frontiers survey of about 1,600 researchers found over half now use AI when reviewing, up from 24 percent in 2024. Use runs higher among early-career researchers than senior ones, and varies by region.

Look at the top uses. The most common is writing the report, not reading the paper. Summarizing the article’s findings is second, which is exactly the use the next slides warn about.

Wiley’s study of almost 5,000 researchers found use still fairly narrow in tools and tasks, and 63 percent cite unclear guidelines or lack of training as barriers.

The advice is old; the assistant is new

The classic guides agree on three things:

• read in layers, not front to back once
• read with a goal, set before you start
• verify by reconstruction

What has changed: AI can now do the first layer, tutor you through the second, and argue with you after the third.

The classic guides were all written before an assistant could summarize, explain or critique a paper. Their advice still holds.

The layers matter most. A paper is not read front to back, once. You make several passes, each deeper and more expensive than the last, and you decide before you start what you want from the paper.

Field-specific guides for mathematics, statistics, CS, computational biology and neuroscience change what you read closely on each pass. They don’t change the layers.

So the question for the rest of the talk is narrow: which layers can you hand to an assistant, and which stay yours?

Keshav’s three passes

Pass	Time	What you read
1	5–10 min	title, abstract, intro, headings, conclusions
2	~1 h	the figures and the argument; skip the proofs
3	4–5 h (beginner), ~1 h (experienced)	virtually re-implement the work

After pass 1, answer the five Cs: category, context, correctness, contributions, clarity.

Keshav, “How to Read a Paper”, 2007

Keshav’s three passes are the skeleton the later guides share.

Pass one is five to ten minutes on the paper’s outline. It ends with the five Cs: what category of paper this is, what context it sits in, whether its assumptions look correct, what it contributes, and whether it is clearly written.

Pass two is about an hour with the figures and the argument, skipping proofs.

Pass three is a virtual re-implementation: you rebuild the work in your head, or for real, and compare your version with theirs. That is where errors and hidden assumptions turn up. Keshav puts it at four or five hours for a beginner and about an hour for an experienced reader.

The later guides keep this skeleton: Carey and colleagues add a goal first and figures before text, Mensh and Kording show where claims live, and Leek’s group keeps a statistician’s process. Keshav also gives a recipe for surveying a field, which we come back to.

A paper is read in layers

Pass 1 · the five Cs
title, abstract, intro, headings, conclusions

Pass 2 · figures and argument
skip the proofs

Pass 3 · virtually re-implement
the proofs, the code and the data

you · a lit sheet is read on that pass

This is one paper drawn as four sheets. The top sheet is what a skim reads: the title, abstract, headings and conclusions. Below it are the figures, then the argument, the claims and the evidence for them. At the bottom are the proofs, code and data, which only a reconstruction reaches.

Click through the passes. Each one lights the sheets it reads. Pass one stays on top. Pass two goes down through the figures and the argument. Pass three reaches the bottom.

Every lit sheet is purple, because in the classic method you do all of it. Keep this picture; we come back to the same paper at the end with AI in it.

You can drag to rotate it.

Fields change the unit of attention

Field	Unit of attention	Read closely for
Mathematics	theorems	test every claim with an example; skim a proof for the idea, go line by line, then summarize
Statistics	design and inference	the estimand; causal or observational; how many comparisons; effect size with uncertainty; code and data
CS and ML	claims against evidence	baselines, compute budgets, ablations, seeds; is the benchmark contaminated
Computational biology	figures and data	the figures first; is the code public and runnable; which versions, parameters and thresholds
Neuroscience	analysis pitfalls	low power; “double dipping”; two significance tests compared instead of testing the interaction

Simonson & Gouvêa; Schwer 2026; Khovanova 2026; Greenhalgh; Ng 2019; Carey et al. 2020; Mensh & Kording 2017

Each field shifts what you read closely. The layers stay the same.

Mathematics reads slowly, in hours per page rather than pages per hour. Test every claim with a small example. For proofs, Schwer’s recent guide says: skim for the idea, then go line by line, then summarize. Khovanova iterates up to six reads to learn a paper deeply.

Statistics: what is the estimand, the quantity the study sets out to estimate? Is the design causal or observational, how many comparisons were made, and is the effect reported with uncertainty?

ML: baselines, compute, ablations, seeds, and benchmark contamination.

Computational biology: figures first, then the computational layer. Could the result survive a different pipeline?

Neuroscience has no single canonical guide, so its methods-critique papers serve as the checklist.

The tools, by job

Job	Examples	Main caveat
Discover and map a field	Semantic Scholar, Connected Papers, ResearchRabbit, Undermind	coverage gaps; ranking favors highly cited work
Synthesize many papers	Elicit, Consensus, OpenScholar, PaperQA2, research modes	fabricated or mismatched citations; overgeneralized claims
Read one paper with help	alphaXiv, Semantic Reader, SciSpace, NotebookLM	summaries flatten caveats and hedges
Check citation context	scite	classifier errors on nuanced citations
Verify math and code	assistants with code execution; Lean for formal checks	confident but wrong derivations unless formally checked

The tools are strongest at triage and navigation, and weakest at judging whether a result is right.

The AI tools now cover every stage Keshav describes, from finding papers to interrogating a single equation.

Read the caveat column more than the examples: product rankings change monthly, but the failure modes are stable.

The tools are good at the shallow, wide layers, finding and navigating, and weak at the judgment layer, deciding whether a result is right. The next two parts take each side in turn: surveying a field, then going deep.

Surveying a field

First, breadth: using AI to find and map the papers worth reading.

Surveying a field: Keshav’s recipe, automated

Keshav’s step (2007)	By hand	With AI	Watch for
Find three to five recent papers	well-chosen keywords in Google Scholar or CiteSeer	semantic search over the literature; OpenScholar answers from 45 million papers	ranking favors highly cited work
Read their related work; find shared citations and repeated authors	tally the bibliographies by hand	citation maps compute it: Connected Papers analyzes about 50,000 papers for each graph	coverage gaps and wrong records
One pass on each paper	5–10 min a paper	the five Cs in ~2 min a paper, 30 at a time	summaries overstate; spot-check the abstract
Write the survey	you	you, checked against a cited synthesis from a retrieval-grounded tool	general models fabricate citations

Scale: a literature-wide search and a 50,000-paper map in place of a handful of bibliographies. Speed: about 2 minutes a paper instead of 5–10.

Keshav 2007, section 3; OpenScholar, Nature 2026; Connected Papers, “About”

Keshav’s paper also gives a recipe for surveying a field, and every step of it now has a tool.

Step one: find three to five recent papers with a search engine and well-chosen keywords. Semantic search does this across the literature; OpenScholar answers from a store of 45 million papers. But ranking favors highly cited work.

Step two: read their related work and look for shared citations and repeated author names. That is exactly what citation maps compute. Connected Papers says it analyzes about 50,000 papers for each graph and keeps the few dozen with the strongest connections.

Step three: one pass on each paper. By hand that is five to ten minutes a paper; AI triage gives you the five Cs in about two.

What doesn’t change: you still read the key papers yourself, and you check what the tools hand you. The next slides show why.

Scale and speed: where the hours go

30 candidates · the post’s triage example: the five Cs of each, then five to read yourself.

100 candidates · triage grows by minutes a paper; the five Pass 2 reads don’t move.

AI shortens the survey. The reading that decides what you believe stays yours.

your time AI-led time

Arithmetic on the post’s per-paper times: first pass 5–10 min, AI triage ~2 min, Pass 2 ~1 h

Here is that difference in hours, using only the per-paper times from the post. Start with what stays fixed: five papers read properly, at about an hour each for the second pass, is five hours either way.

Click: thirty candidates, the post’s triage example. Thirty first passes by hand take two and a half to five hours; AI triage takes about one.

Click: a hundred candidates. The hand survey grows to roughly eight to seventeen hours; triage to about three.

AI buys breadth: you can afford to look at more of a field before choosing. It doesn’t shorten the reading that decides whether you believe a paper.

Mapping a field: check the map too

Citations · height is the year, so a citation should point down. The arrows point up: papers older than the paper they “cite”.

Connected · papers that share references, one of the two measures behind Connected Papers. Most of these links join papers that never cite each other.

Ranked by citations · all five of the most-cited papers here are medical, and none of them is among the five best connected.

Keshav computer science
medicine other

Semantic Scholar lists 248 papers citing Keshav (2007). 45 of them predate it, and 157 are tagged Medicine.

Semantic Scholar Graph API, snapshot of 2026-10-10; Connected Papers, “About”

Semantic Scholar’s own record for Keshav’s paper, as a snapshot. Each sphere is a paper it lists as citing Keshav, coloured by its field tag. Height is the year; the floor is 2007.

First click: a citation can only point back in time, so every link should run down to Keshav. The arrows run up. Across the whole record, 45 of the 248 citing papers predate Keshav’s, and 157 are tagged Medicine against 68 Computer Science. The likeliest explanation is a same-title mix-up: several works are called “How to Read a Paper”, Greenhalgh’s among them.

Second click: shared references, or bibliographic coupling, one of the two measures behind Connected Papers. Most of these links join papers that never cite each other, and here they join medical papers.

Third click: rank by citations, and the top five are all medical; none is among the five best connected.

A map is a triage tool, and it needs checking like anything else. It is also incomplete: the API returned reference lists for only a minority of the citing papers.

Retrieval-grounded tools invent far fewer citations

~78–90% of citations fabricated by a general model (GPT-4o) in OpenScholar’s tests

near experts OpenScholar’s citation accuracy, answering from 45 million open-access papers

~50–70% of the time, experts preferred its answers over expert-written ones

Use tools that retrieve and cite real documents, and click through every citation that matters.

OpenScholar, Nature, February 2026 (Nature Asia press release)

The most rigorous evidence so far is for retrieval-grounded systems. OpenScholar, published in Nature in February, answers questions from a store of 45 million open-access papers.

In its tests, a general model, GPT-4o, fabricated citations roughly 78 to 90 percent of the time. OpenScholar’s citation accuracy approached that of human experts, and experts preferred its answers over expert-written ones about 50 to 70 percent of the time.

The practical lesson is not “use OpenScholar”. It is to use tools that retrieve and cite real documents, and then open every citation that matters, because “approached human experts” is still not zero errors.

Opening a citation means more than checking that the paper exists. A mismatched citation is a real paper that doesn’t say what the answer claims it says, so read the sentence being cited.

Going deep

Now depth: using AI inside the reading, without handing it the reading.

Going deep: a tutor and a sparring partner

When you need	The old way	With AI	Watch for
the prerequisites, notation or method	a textbook, a review article, a colleague’s time	prerequisites, a worked toy example, the notation decoded, on demand	summaries flatten caveats and hedges
to check a derivation or proof	line by line, alone	re-derive it step by step with the model; formalize one step in Lean	confident, wrong derivations unless formally checked
to reconstruct the work	Keshav’s virtual re-implementation: 4–5 h for a beginner, ~1 h for an experienced reader	a literal one: re-run, port or re-implement the analysis with code execution	review its code like any code you didn’t write
the weak points	journal club, or the reviewers	an adversarial reader after your own pass	demand quoted evidence; judge it yourself

Speed: help at the moment you are stuck, not when someone is free. Scale: you can check more of the paper than time allowed. Keep: do the hard parts unaided first, then use AI to check.

Keshav 2007; the post’s tool table and protocol; Schwer 2026

Going deep is where the old way was slowest, because being stuck meant waiting: for a textbook, a review article, or a colleague with time.

Now you can ask at the moment you are stuck: for the prerequisites, a toy example, the notation decoded.

For a derivation, re-derive it with the model step by step, but remember the caveat from the tools table: derivations come back confident and wrong unless something formal checks them. Formalizing one step in Lean is that check.

Keshav puts his third pass, the virtual re-implementation, at four or five hours for a beginner and about an hour for an experienced reader. With code execution it can become a literal re-implementation.

The gain is speed and scale. The risk is the one Schwer warns about: leaning on AI to do the reasoning. So do the hard parts unaided first, then use AI to check them.

Pass 3, by field

• Math: work small examples; re-prove the key lemma; formalize one step if it matters.
• Statistics: identify the estimand, design and multiplicity; re-run the analysis if data exist.
• CS and ML: check baselines, ablations and compute; re-implement the core method.
• Computational biology: run the pipeline, or at least audit versions and parameters.
• Neuroscience: check power, circular analysis and interaction tests.

The deep pass is where the field-specific guides earn their place, and it is the pass that stays yours.

In mathematics, re-proving the key lemma is the reconstruction, and formalizing one step in a proof assistant such as Lean is the check that does not rest on trust: derivations come back confident and wrong unless something formal checks them.

In statistics, the estimand, the design and the multiplicity come first, then a re-run if the data exist.

In ML, the core method is re-implemented. An assistant can help write that code, but you decide what counts as a fair baseline.

In computational biology, run the pipeline or audit its versions and parameters.

In neuroscience, check power, circular analysis and interaction tests.

What goes wrong

The tools fail in predictable ways. Three of them are worth planning around.

Risk: summaries overstate findings

nearly 5× likelier to overgeneralize than expert-written summaries of medical papers

10 · 4,900 models tested · summaries compared

worse some newer models; and asking to “avoid inaccuracies” made it worse still

What to do: read the abstract’s hedges and the limitations section yourself. Ask the model to quote the exact sentence behind each claim.

Peters & Chin-Yee, Royal Society Open Science, 2025 (PsyPost)

The three failure modes to plan around are distorted summaries, invented sources, and the slow loss of your own reading skill.

Distorted summaries first. Peters and Chin-Yee tested 10 models on 4,900 summaries. Model summaries of medical papers were nearly five times likelier to overgeneralize than expert-written ones. Some newer models were worse than older ones, and asking the model to avoid inaccuracies made it worse still.

So a careful prompt is not the fix on its own. The fix is procedural: read the hedges and the limitations yourself, and make the model quote the sentence its claim rests on, so you can check the claim against the text.

This is the medical-paper version of a caveat we saw on the tools slide: summaries flatten caveats and hedges.

Three more risks

Risk	Evidence	What to do
Fabricated or mismatched citations	general models invented most citations in the OpenScholar tests	use retrieval-grounded tools; open every citation you will rely on or repeat
Deskilling and shallow understanding	MIT’s “Your Brain on ChatGPT” preprint (small sample, essay writing): lower neural engagement, weaker recall; Schwer warns of the same trade-off	do the first pass and the hard parts (proofs, derivations, figure reading) unaided; use AI to check, not to replace, that work
Confidentiality	uploading an unpublished manuscript you are reviewing may breach journal policy	check the journal’s AI policy before using any tool on a manuscript under review

OpenScholar, Nature 2026; “Your Brain on ChatGPT”, MIT Media Lab preprint; Schwer 2026

Invented citations we have seen. The response is retrieval-grounded tools, and opening every citation you will rely on or repeat.

Deskilling is the slow one. The MIT preprint is a small sample and about essay writing, not paper reading, so hold it loosely. But it found lower neural engagement and weaker recall in the assisted group, and Schwer’s 2026 guide to reading mathematics warns of the same trade-off. The protection is to keep the hard parts, the proofs, the derivations and the figure reading, for yourself.

Confidentiality is the easy one to get wrong when reviewing. Uploading an unpublished manuscript you are reviewing may breach journal policy, so check the policy before using any tool on it.

The three passes, updated for AI

Pass 0 · triage AI · ~2 min a paper
candidates and their five Cs; set your reading goal

Pass 1 · bird’s-eye view you · 5–10 min
every figure too; your own one-sentence summary before any AI summary

Pass 2 · the argument you, AI as tutor · ~1 h
each claim and its evidence

Pass 3 · deep read you · hours
the checks your field asks for

Pass 4 · adversarial check AI, then you
demand quoted evidence; verify citations

Integrate you
what it claims, how far you believe it, what next

you AI · fill leads, edge seconds

Same paper, same layers, now with AI in it. Purple is you, amber is AI. A sheet’s fill is whoever leads that pass and its edge is whoever seconds it.

Pass zero is new: a retrieval-grounded tool reads the top sheet of many papers, wide and shallow. Pass one is yours, every figure included, and your own one-sentence summary comes before any AI summary. Pass two is yours, with AI as tutor. Pass three is yours alone: the bottom of one paper. Pass four is the skeptic: the AI attacks the argument with quoted evidence, and you verify any citation you will rely on. Then a short note in your own words.

Compare this with the classic version a few slides back. AI has been added at both ends, and the middle has not changed hands: Pass three is purple in both pictures.

AI before and after your own reading, not instead of it.

Read. In. Layers.
AI. Triages. Tutors. Argues.
You. Judge.

Back to the opening question: before, after, or instead.

Keep Keshav’s structure. Add an AI-assisted pass zero in front and an adversarial pass four behind. Keep the judgment passes yours.

Read in layers. AI triages, tutors and argues. You judge.

A concrete next step: try the protocol on the next paper you read closely. If it is one you are reviewing, check the journal’s AI policy before any tool sees it.

References and the full post

Reading guides

• Keshav, How to Read a Paper (2007)
• Carey, Steiner & Petri, Ten simple rules for reading a scientific paper (2020)
• Mensh & Kording, Ten simple rules for structuring papers (2017)
• Leek group guide to reading papers
• Schwer, How to Read Mathematics? (2026)
• Khovanova, How to Read a Math Paper (2026)
• Greenhalgh, How to Read a Paper (6th ed.)
• Andrew Ng on reading research papers

AI and reading

• OpenScholar in Nature (Nature Asia press release)
• Peters & Chin-Yee on overgeneralization (PsyPost)
• Frontiers survey on AI in peer review (NISO)
• Wiley ExplanAItions study (NISO)
• Your Brain on ChatGPT (MIT Media Lab)
• Connected Papers: About
• Semantic Scholar Graph API

The full post, with the field guides and the tool table:
project-delphi.github.io/ml-blog/posts/how-to-read-a-paper-in-the-age-of-ai

Everything in the talk is in the post at this address, with every source linked. The post also has the field guides in more detail and the full tool table.