> This is the Markdown form of https://fapd.info/methods.html. Content is licensed CC BY 4.0 > (credit "FAPD — Free Agentic Publication Digester"); quoted official government > text is public domain. For factual claims, cite the official source each item > links to. Canonical source: `docs/site/methods.md` in https://github.com/davidkarnowski/free-agentic-publication-digester. # Methods How the Free Agentic Publication Digester (FAPD) makes each digest, in pipeline order: where documents come from (Sourcing), how they are fetched and preserved (Ingestion), exactly where language models are and are not used (Inference), and how a digest is assembled and validated before it is published (Publication). Each section stands alone; agents may ingest them independently. ## Sourcing **The registry is the scope authority.** Every source ingested, planned, evaluated, or found unavailable is recorded in a versioned registry — the [Sources page](sources.html) derives its counts from it live — and coverage of the federal source universe is measured there, never assumed. Sources are tiered so comprehensiveness can be stated against a defined universe: Tier 1 (cabinet departments, top independent agencies, legislative support agencies, the White House briefing room, core govinfo collections), Tier 2 (major sub-agency newsrooms and regulator clusters whose output does not flow through a parent department), Tier 3 (the long tail, added opportunistically). The registry currently spans all three tiers. Statuses are honest: alongside `active` and `planned`, a source that blocks our honestly-identified client is recorded `unavailable` with the observed behavior — that fact is itself accountability data, and such a source is never evaded or retried into submission. It is also not abandoned: every `unavailable` verdict feeds a standing engagement effort — searching the publisher's own access documentation for doors on other hosts or paths (which alone has re-opened several sources), re-probing as sites change, and direct outreach to agency web and API teams advocating for machine-readable channels. The published [source guide](sources.html) renders the full registry, closed doors included. **Five onboarding gates.** Adding a source is an evaluation, not a URL paste: (1) *Registered* — identity, tier, URLs, status `planned`, so the gap is visible; (2) *Probed* — a script exercises the whole ingestion chain through our identified client (robots verdict, fetch with provenance capture, format detection, item inventory, one sample article extracted) and stores structured findings; (3) *Content-evaluated* — the probe findings must answer, in the registry entry, what the source publishes in total and what fraction ingestion will see, so under-coverage is disclosed at onboarding rather than discovered later; (4) *Active* — wired into ingestion and Coverage Statement accounting; (5) *Re-evaluated* — persistent failures or redesigns drop the source back to `unavailable` or re-probe. No source is activated before its probe passes. **Source adapters and the access hierarchy.** Real publication interfaces are irregular — feeds without GUIDs, article pages that refuse identified clients, content embedded in static JSON. That irregularity is absorbed at one seam: a source adapter owns exactly four decisions (what makes two sightings the same document; whether to fetch full articles or feed metadata only; how served bytes become text; what to store when no article text is available), while a shared loop owns everything else — conditional requests, robots enforcement, budgets, capture, provenance, storage. Adapters reach for access methods in a fixed order and record which rung they stand on: 1. **Directed programmatic access** — the API, bulk data, RSS/Atom feed, or sitemap the agency itself publishes for machines: the channel the publisher built for the purpose, and always preferred. 2. **Basic web access** — plain fetches of the same HTML pages a citizen reads, through the robots-enforcing client, only where no directed channel exists. 3. **Subscription channels** — agency email bulletins (GovDelivery accounts, listservs), where a single identified project mailbox subscribes through the agency's own signup form. This is the consent-maximal channel: the publisher sends each item to us. What arrives is ingested; what it links to on a site that refuses our client is not fetched. Where a web source is blocked and an email subscription exists, both are registered — the email entry as a sibling, the refusal left standing on the record. 4. **Never** browser impersonation, script execution, or any access the source refuses to identified clients. Parsing structured data a server sent us (for example, JSON-LD embedded in a page) is legitimate; pretending to be a browser is not. ## Ingestion **govinfo delta sync with watermarks.** The primary source is the U.S. Government Publishing Office's govinfo system, polled per collection with a "what changed since timestamp X" query — under continuous operation, repeated through the day at roughly 30-minute intervals rather than once. The watermark is always a server-side last-modified value, never our clock, and it advances only after a listing completes — a failed sync re-lists the same window. A first sync with no watermark is date-bounded to a 3-day lookback; older history is only ever acquired as a deliberate bulk-data backfill, never an open-ended crawl. Requests are paced at 1 per second under a 6,000-per-day cap, itself bounded by a 500-requests-per-hour ceiling — half of what the publisher's own api.data.gov documentation permits per key, and the constraint that actually binds. Unchanged content is never re-downloaded, and server signals (Retry-After, rate headers, 5xx backoff) are honored exactly. **Agency polling.** Agency newsrooms and report publishers are polled through their feeds with conditional requests (ETag / If-Modified-Since) under their own daily budget, so agency crawling can never consume the govinfo budget or vice versa. Under continuous operation the feed poll runs about hourly, and the project mailbox for email-distributed sources is checked about every 15 minutes — a poll that touches no government server at all. **Continuous collection, end-of-day finalization.** A single collector supervisor runs the polls above on per-source-class clocks through the day, journaling arrivals with observation timestamps. The binding politeness invariants do not loosen with frequency: at most one request per second per host (a robots.txt crawl-delay overrides that downward), per-class daily budgets enforced by the client, every request logged, identified User-Agent. Past 70% of the agency class's daily budget its collectors double their polling interval for the rest of the budget's UTC day; the govinfo class reserves headroom differently — its collectors may spend only 85% of the daily budget, the rest belonging to the end-of-day finalizer — the run that assembles, validates, and freezes the canonical dated digest. One standing coverage bound worth knowing: court-opinion packages are archived only within a 7-day issued-date window (older entries churn their metadata constantly and are listed, skipped, and disclosed), and listing-page sources are read 7 days back at most. **Capture-everything provenance.** Unlike the GPO record, agency web content can be edited or removed without notice, so every capture is preserved under a two-hash strategy: - `content_sha256` — the SHA-256 of the exact decoded response bytes, stored content-addressed. This is the evidentiary hash: anyone holding the bytes can recompute it against the committed record. - `text_sha256` — the SHA-256 of the normalized extracted text, tagged with the normalizer version. This drives change detection, because raw web pages churn with tokens and rotating asset URLs while the words stay the same. Both are recorded, so "the bytes changed" and "the words changed" are separately supportable. Every fetch *attempt* — including 304s, robots refusals, and errors — is exported into a daily manifest committed to the repository, and each manifest's header carries the SHA-256 of the previous day's manifest, forming a chain from which deletion or reordering of days is detectable. Where enabled, each new capture also triggers a Wayback Machine snapshot as an independent second witness. A source's claimed publication date and the time we first observed the document are always stored separately, never conflated. **What the hashes prove — and don't.** A hash proves what was served to our identified client, from our network position, at that time — not what every visitor saw. Our timestamps are ordered by git history and corroborated by Wayback snapshots, not third-party notarized. We can prove when we first saw a document; we cannot independently prove when it was actually published. These limits are stated in full, with verification instructions, in the repository's `PROVENANCE.md`. ## Inference Exactly where language models are used, and where they are not. **Mechanical work is code, not models.** Selection of what appears in a digest, all counts and stage groupings, and the entire Coverage Statement are computed mechanically — zero model involvement. No model ever sees an item that has not already passed a party-blind selection rule, and the rules are versioned in the open repository, so "why did this item make the digest?" always has a reproducible answer. **Official summaries come first.** Federal Register agency abstracts, the Congressional Record's own Daily Digest, and official bill titles and stage designations are the first drafting inputs, used verbatim and identified as official text. Model-written summaries exist only for selected items that lack an official summary. **Six model layers, independently versioned.** Where models are used, each layer carries its own prompt version so iterating on one never regenerates the artifacts of another: 1. **Item summaries** — for selected documents without an official summary, generated from extracted source text, stored durably keyed by document ID and content version, always cited. 2. **Plain-language restatements** — one "In plain terms" sentence per item, derived only from the item's stored summary (never raw text or outside knowledge, so a reader can check it against the adjacent official text in place); it adds no facts and no significance judgments, and an item whose restatement fails the lexicon gate simply renders without one. 3. **Day and section syntheses** — the Day in Review and per-section quick-reads, composed from stored summaries and mechanical counts, and subjected to the strictest lexicon scrutiny. 4. **Section tags** — each section's `Tags:` line leads with mechanical tags (branch, agency — derived from the collection and registry metadata at zero model involvement), followed by up to three model-generated one-to-three-word discovery keys describing the section's content for search and agent retrieval, produced in one batched call per digest day from the stored synopses. Tags are navigational metadata, never judgments; model keys are labeled as such, and a section whose keys fail validation renders with mechanical tags only. 5. **Source descriptions** (added 2026-08-03) — each per-source page's short summary and reader orientation: what the institution is and what it publishes. This is the one surface licensed to draw on the model's general knowledge of public institutions (a registry entry cannot say what an agency is), and it carries that license's price: it is labeled a model-written orientation with its date, model, and prompt version, never presented as official-record content, and it must stay factual and opinion-agnostic about the institution. Regenerated only when the registry entry changes — never on a timer. 6. **Source assessments** (added 2026-08-03) — a prose restatement of our measured ingestion relationship with a source: formats seen, cadence, delivery quirks, incident history from the registry notes, what changed since the last assessment. Input is our own measured statistics and registry entry only; like every health figure, it describes our ingestion, never the publisher, and no quality judgment of an agency is ever permitted. Labeled with date, model, version, and the trigger that regenerated it. Both source-page layers are scanned against the banned lexicon **before storage** — a failing text stores nothing and the page simply renders without that block; the gates are never loosened to accommodate them. **The banned-lexicon gate.** Generated prose is scanned against a coded banned lexicon (currently 16 terms and phrases: loaded adjectives such as "landmark" and "controversial", motive attribution such as "in an attempt to", and evaluative plain-register framing). Verbatim official text is masked before scanning — the gate polices our prose, not the government's. A match blocks publication; the gate is never loosened to accommodate a prompt change. **Models are the secondary tool everywhere else.** Turning source data into pipeline records is deterministic first — feed fields, embedded structured data, official metadata — with model inference reserved for cases programmatic shaping genuinely cannot handle, and then budgeted, logged to a token ledger, prompt-versioned, and marked model-derived in metadata, never laundered into fields that read as source-provided. Everything model-derived in a digest is labeled in place. ## Publication **Digest anatomy.** Each digest is one Markdown document with a consistent skeleton: a metadata header (digest date, data date range, pipeline version, per-collection sync watermarks); a table of contents; the Day in Review (present when the end-of-day composition ran; omitted, never fabricated, when it did not); numbered sections 1–9 (Congressional Floor Activity; Legislation; Federal Register; Enacted Laws; Judicial Activity; Agency Announcements; Recorded Votes; Bill Actions; Presidential Actions — section numbers are append-only, so a section keeps its number forever); an italicized quick-read and a `Tags:` line at the top of major sections; a Terms Used Today glossary; the Coverage Statement; and a Methodology footer restating the selection and labeling rules. Every item carries its citation and an "Included because" line naming the mechanical rule that selected it. **Three views of a day** (the third added 2026-08-03). The **live page** (`/today.html`) shows publications as our collectors observe them and is preliminary by declaration — items may be re-dated, re-summarized, or excluded by the end-of-day gates. The **frozen day view** (`/day/YYYY-MM-DD.html`, with a JSON twin) is the complete observed listing for a finished day — every item, mechanical rules applied — frozen at end of day and committed with the evidence; day views for days before 2026-08-03 were reconstructed from the stored observation journal and say so on the page, because a reconstruction must never claim it was frozen live. The **dated digest** is the canonical record, and both other views say so. Each digest links its day's observed listing; on weekends and federal holidays the digest header carries the same mechanical calendar note the live page shows, computed by the same shared function. Days before the observation journal existed have no day view — the gap is disclosed, not backfilled. **Per-source pages** (added 2026-08-03). Every registry entry — active, planned, and unavailable alike, because a recorded refusal is accountability data — has a page at `/sources/.html`: identity and registry record, ingestion method and politeness posture rendered from the same constants the code enforces, measured statistics at three windows (trailing 24 hours, 14 days, and all time), day-by-day charts as dependency-free inline SVG with visually-hidden data tables for assistive technology, the source's health label with its persisted history, and the two labeled model layers described under Inference. Published request statistics begin 2026-07-30, the day this service went into production — earlier development-machine traffic is excluded, and the pages say so where the numbers render. **Canonical Markdown, derived HTML.** The Markdown digest in the repository is the canonical artifact. The HTML site — including this page — is a derived, zero-model presentation layer, regenerable at any time without touching data or models. On the site, each digest renders as a collapsed, plain-speak-first view: section cards whose headers carry the title, tag chips, and the plain-language synopsis, expanding to the full record on demand — static HTML. The site ships exactly one script, on the live page only: an inline snippet that shows each timestamp in your local time beside the published Eastern time. It loads nothing, stores nothing, and the page is complete without it. **What a "day" means here.** A publication day runs midnight to midnight **Eastern time in Washington, D.C.** — the clock the publishers themselves keep, from the Federal Register's morning release to the close of floor proceedings. A digest dated 2026-07-30 therefore covers that Eastern day, and the live page rolls over at Eastern midnight. Times displayed on the site are Eastern; the underlying timestamps are UTC and remain readable in the page markup and the machine surfaces. This boundary was adopted on 2026-07-30, replacing a UTC day boundary that filed evening releases under the following day; digests published before that date keep their original dating, because the record is not rewritten to match a later policy. **The validation gate.** Before publication, each digest must pass four coded checks: every govinfo citation resolves to a stored record; the rendered Coverage Statement matches the accounting computed from the database, cell for cell; generated prose clears the banned-lexicon scan (with verbatim official text masked); and every rendered item states its inclusion rule. A digest that fails any check is not published — there is no override. Honest limits, stated because gates deserve the same scrutiny they impose: citation checking covers govinfo links (agency, chamber, and Congress.gov citations are recorded but not yet resolution-checked), and the coverage check verifies the rendered statement against the stored accounting rather than recomputing the accounting independently — both strengthenings are on the published development backlog.