Registry Loading Status

Reports are automatically generated on a schedule, committed to this repository, and published to GitHub Pages. They track migration progress from the legacy PDS Solr registry to the new OpenSearch-based registry.

Note

Reports update automatically whenever the scheduled workflow runs (typically daily). The burnup chart is published to GitHub Pages on every push to main or develop.

How These Reports Work

The baseline for what bundles and collections should exist in the registry is the PDS Keyword Search, which is populated by the Engineering Node (EN) each time a formal data release occurs. This is the authoritative inventory of PDS archive data — a product that EN has released is the agreed-upon source of truth for what belongs in the archive.

These reports compare that baseline against what is actually accessible through the PDS Registry API to surface gaps.

A product is missing when it appears in the PDS Keyword Search but is not yet present in the new OpenSearch-based registry. A product is staged when it is in the new registry but its archive_status has not yet been advanced to archived — it has been loaded but requires operator action to complete the transition.

For detailed guidance on interpreting the reports, filtering by node, and understanding the superseded flag, see the docs/status README on GitHub.

Burnup Chart

The Burnup Chart shows cumulative products loaded into the new registry over time, broken down by PDS node. It includes:

• Overall and per-node loading progress vs. target

• Both “all versions” and “latest version per LID” views

• An interactive date-range filter (defaults to the last 12 months)

📊 Open Burnup Chart in full page

Status Reports

All CSV reports and the full metrics summary are in the docs/status directory on GitHub. The README in that directory contains the latest metrics table and detailed field descriptions.

Missing Products

Products present in the legacy Solr registry but not yet loaded into the new OpenSearch registry.

File	Description
missing_bundles_in_registry.csv	All missing Product_Bundle records (all versions)
missing_collections_in_registry.csv	All missing Product_Collection records (all versions)

Each row includes NODE_ID, LIDVID, PRODUCT_CLASS, and a SUPERSEDED flag (true if a newer version of the same LID exists anywhere in the dataset).

Loaded Products

All products currently in the new OpenSearch registry, regardless of archive status. Queried with search_after pagination so counts are not capped at 10,000.

File	Description
loaded_bundles_in_registry.csv	All Product_Bundle records in new OpenSearch
loaded_collections_in_registry.csv	All Product_Collection records in new OpenSearch

Note

Loaded counts will exceed the legacy Solr counts because the new registry also contains products harvested directly from non-Solr sources (e.g., PSA/ESA ~900 bundles, ~4,000 collections).

Staged Products

Products loaded into the new registry but with archive_status = staged (loaded but not yet transitioned to archived). These require operator action.

File	Description
staged_bundles_in_registry.csv	Staged Product_Bundle records
staged_collections_in_registry.csv	Staged Product_Collection records

Historical Counts

Append-only snapshots of missing and staged totals per script run. Use these files to plot burndown charts externally.

File	Description
counts_history.csv	Missing/staged totals per run (append-only)
burnup_history.csv	Cumulative loaded counts by date (all versions)
burnup_history_latest.csv	Cumulative loaded counts by date (latest version per LID only)
burnup_by_node.csv	Per-node cumulative loaded counts (all versions)
burnup_by_node_latest.csv	Per-node cumulative loaded counts (latest version per LID only)

Report Generation

Reports are generated by scripts/generate_registry_status_reports.py. To regenerate locally without committing:

./venv/bin/python scripts/generate_registry_status_reports.py --no-commit

Requires AWS/Cognito credentials configured in ~/.pds/.registry-client or .env. See Developer for environment setup details.