Monitoring arXiv without writing an Atom parser at 2am

arXiv's API is genuinely good — powerful fielded queries, no auth, generous limits. It also returns Atom XML with three namespaces, single-vs-array ambiguity on nearly every element, versioned IDs embedded in URLs, and PDF links hidden in rel attributes. Everyone who's built a paper-monitoring tool has written the same 150 lines of parsing code, and none of us enjoyed it.

arXiv Papers Extractor is those 150 lines, plus paging, plus arXiv's requested 3-second politeness delay, wrapped as an actor. Query in, JSON records out:

// free text + category
{ "query": "state space models", "category": "cs.LG", "maxResults": 50 }

// fielded arXiv syntax passes straight through
{ "query": "ti:\"reinforcement learning\" AND au:silver" }

// or grab specific papers
{ "idList": ["1706.03762", "2005.14165"] }

Each record: arxivId, version, title, abstract, authors[], primaryCategory, categories[], published, doi, journalRef, and direct absUrl/pdfUrl. Flat, typed, boring — the good kind of boring.

The pattern: schedule + diff + digest

The highest-value use is a standing search. Schedule the actor daily with your topics, sort by submittedDate, and downstream-diff the IDs against yesterday's run. New IDs = new papers = your morning digest. Add an LLM summarization step (Apify's MCP integration makes the dataset directly readable by agents) and you have a personal research-monitoring service for pennies a day.

Pair it with a PDF step

Records include pdfUrl, so chaining into a PDF-extraction actor gives you full-text literature ingestion: search → fetch → chunk → embed. That's a literature-review corpus builder with a schedule attached.

Fixes (2026-07-29)

• Exponential backoff on 429 Too Many Requests — the actor now respects arXiv's Retry-After header and applies exponential backoff (30s, 60s, 90s, 120s, capped at 120s) for up to 5 attempts. No more run failures when arXiv rate-limits a batch job.