The arXiv pipeline, with its receipts
Read every new AI paper on arXiv and answer three questions with evidence: where the authors work, who paid for the work, and which author records belong to the same person.
| Stage | What happens | How |
|---|---|---|
| List | Every new paper in cs.AI, cs.CL, cs.LG, cs.CV, cs.CR, cs.MA, cs.RO, cs.HC, cs.CY, stat.ML | arXiv bulk metadata (OAI-PMH) |
| Fetch | Full text of each paper | arXiv HTML, with the PDF's first page when the HTML drops affiliations; rate-limited; nothing kept on disk |
| Parse | Author block, affiliation lines, emails, names in characters | Deterministic parsing |
| Affiliations | Each line → institution and the place it states | Bilingual institution index |
| Funding | Every funding or in-kind acknowledgement, wherever it appears | Sentence classifier, funder extraction, grant-number parser |
| Scope | In / out / unknown under each saved definition | A filter over stored evidence, so changing a definition needs no re-run |
| Identity | Author records → people | Log-likelihood matcher using pf-names |
| Review | Checks, funder labels, overrides | Local web app; decisions survive re-runs |
"Hong Kong SAR, China" is not mainland China. Scope is decided from the place a line actually names, matched on whole words. That rule also fixed places turning up inside personal names: 传辉 ("Chuanhui") is not Anhui, and 京贤 ("Jingxian") is not Xi'an.
Institution names are ambiguous. The index combines a curated seed with about 8,800 organisations from the Research Organization Registry. An acronym counts only when it is unambiguous: HUST (Wuhan or Hanoi) and NTNU (Taipei or Trondheim) are excluded. Ordinary-word names count only when they make up a whole comma segment. A company match is dropped when the line states another country, so Huawei's Zurich lab is not China.
A funder's country depends on the sentence. "Ministry of Education" may be China's, India's or Korea's. Generic names are resolved sentence by sentence, from the place named, from national markers such as grant prefixes and programme names, or from the one country the sentence mentions.
Each link is graded, and the weights are calibrated on pairs of records that share an email address, set against random pairs. In a human review of 30 pairs the matcher had left unresolved, 20 were the same person and none were different people. The matcher errs on the side of caution by design: a false merge is worse than a missed one when the output feeds analysis of real people.
Confirmed
Needs a hard identifier, an email, ORCID or grant PI, or identical characters plus corroboration.
Probable
The ceiling for soft signals alone: name, co-authors, institutions and topic.
Unresolved
The evidence is not enough. Left for a human, never forced into a merge.
A local review app shows each machine decision with its evidence: the affiliation line, the funding sentence, and the name comparison with its reason. A reviewer can confirm, reject or relabel. Decisions are stored separately from the machine output, override it, and survive every re-run. The app binds to localhost, and researcher data never leaves the analyst's machine.