Build 14.0 Plan

The Planetary Data System (PDS) is a long-term archive of digital data products returned from NASA’s planetary missions, and from other kinds of flight and ground-based data acquisitions, including laboratory experiments. But it is more than just a facility - the archive is actively managed by planetary scientists to help ensure its usefulness and usability by the world wide planetary science community.

The following page outlines the planned PDS software enhancements, bug fixes, and impacted requirements for the upcoming Build .

A more detailed breakdown of these tasks by date and priority can be found in our Zenhub Roadmap (login required).

Key Dates

Key Dates for Discipline Nodes

Date

Name

Responsible Party

Description

08/14/2023

SCR Freeze Date

DDWG

Freeze date for identifying SCRs to be implemented in this release.

08/15/2023 - 09/16/2023

Standards Documents Updates Due

Document Authoring Team

Updates to PDS4 standards documents due to EN for review.

10/17/2023 - 11/07/2023

Beta Testing Period

DNs, IPDA

Changes to PDS4 IM and system components available for beta testing.

10/17/2023 - 11/07/2023

dLDD Integration and Test

dLDD Stewards

Auto-generated dLDDs should be reviews by dLDD Stewards, and final updates to dLDDs should be completed during this time.

09/28/2023 - 10/26/2023

Standards Documents Review Due

Document Review Team

Review updates to PDS4 standards documents.

12/01/2023

System Release

EN

PDS4 system is operationally deployed and website is updated accordingly.
Key Dates for Engineering Node

Date

Name

Description

09/12/2023

Delivery to I&T

The PDS4 IM and all software updated during this build cycle are delivered to EN I&T.

09/25/2023

Test Readiness Review (TRR)

PDS EN internal review to verify readiness to enter I&T phase.

11/20/2023

Delivery and Deployment Review (DRR)

PDS internal review with PDS Software Working Group to verify readiness to deploy build system.

12/01/2023

Operational Deployment

PDS4 system is operationally deployed and website is updated accordingly.

For a more detailed schedule, see https://pds-engineering.jpl.nasa.gov/content/schedules.

Planned PDS4 Information Model Changes

This section details the planned changes to the PDS4 Information Model for this build. These changes are deliverables by the PDS Data Design Working Group (DDWG) to the Engineering Node for implementation.

Planned Software Changes

This section details the planned software changes for this build.

Planned tasks for the next release cycle are grouped by software component, and sub-tasks are included where known at the time of this plan generation. Each section also includes additional information, such as:

  • Software description

  • Link to 🔍 User Guide

  • Link to 💻 Github Repo

  • Link to 🚑 Issue Tracking

  • Link to 📒 Backlog - future work not planned for this release cycle

Don’t see the tool you are looking for? This means there is no active development planned for this build cycle. Head to the PDS Tool Registry to see all possible tools available for PDS data.

Have a new feature or tool you think we should be working on? Create a ticket in the applicable Github repo or in our PDS EN Operations repo and we will see how that can fit into our plans moving forward.

validate

Validates PDS4 product labels, data and PDS3 Volumes

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release
  • validate#534 B14.0 Content Validation Improvements: Additional Table Types, Additional File Areas

    • validate#7 Update to support ComplexLSB8 data types and investigate Floating point exception

    • validate#190 Validation fails to catch real value in ASCII_NonNegative_Integer field (Table_Delimited)

    • validate#211 Update validate with additional data type support for image transformations

    • validate#217 As a user, I want to validate content for all possible PDS4 table types

    • validate#343 As a user I want to see the name of a table/array in errors, if one is specified

    • validate#431 warning.table.characters_between_fields missing for last record in table

    • validate#436 Improve error messages for overlapping objects in a label

    • validate#476 Check for duplicate LIDVIDs in collection inventory

    • validate#480 Validate does not calculate overlaps correctly when Header is not first object in file

    • validate#508 CRLF/LF checks do not seem to be working as expected for Table_Character

  • validate#606 Support for Encoded Video and Encoded Audio

    • validate#604 As a user, I want to validate MP4/H.264 encoded video as observational data

    • validate#605 As a user, I want to validate MP4/H.264/AAC encoded audio as observational data

  • validate#607 Support for Improved Datetime Checks

    • validate#520 Refactor regex handling to use Google’s RE2J library and compile

    • validate#608 Update datetime regex for content validation

  • validate#610 B14.0 Enhancements to Initial Version of Registry Referential Integrity Validator

    • validate#619 Improve new referential integrity checker timeout tolerance

    • validate#620 verbosity flag does not appear to output INFO messages

    • validate#622 Update to use non-zero exit code if any ERROR in execution

    • validate#643 Document how to use validate-refs tool in Validate Operation User Guide

  • validate#629 Add Check for Schematron/Schema Version Mismatch

    • validate#628 As a user, I want to throw a WARNING when a product’s schematron version does not match the schema version

pds4-information-model

The software tools and data necessary for generating the Information Model including PDS4 ontology, data, and information model.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

pds-github-util

github utility functions to enforce the PDS engineering node software life cycle.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

software-issues-repo

Issue tracking repository as a centralized entry point for general PDS software bugs and feature requests.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

pds-api

PDS web APIs specifications and user’s manual

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release
  • pds-api#237 Support Searching for Past Versions of Products with Differing LIDs

    • pds-api#148 As a user, I want to search for past versions of a product where the LID changed during the product history

  • pds-api#255 Enable Authorized Access to Restricted Data

operations

Tickets for the PDSEN Operations Team

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

doi-ui

The web interface for the PDS DOI Service providing the ability management PDS archive DOIs. See the DOI Service for more details on the available capabilities. https://nasa-pds.github.io/doi-service/

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

  • doi-ui#184 Develop DOI Service User Guide and Training

cloud-tasks

PDS Cloud Migration documentation, issue, tracking and simple tools for assisting in the PDS hybrid cloud study and migration efforts.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

planetarydata.org

Website and related services for the International Planetary Data Alliance, nominally run at https://planetarydata.org/

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

devops

Parent repo for PDS DevOps activities

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release
  • devops#37 Automatically deploy an up to date I&T platform where all software can be tested

    • devops#36 For registry components, add a github action which runs the integration test when something is pushed on a dev branch

    • registry-sweepers#6 Create the docker image and push it to docker hub as part of the CICD

    • devops#41 Push docker image on AWS ECR in github-actions, in addition to docker hub

    • devops#44 Ask new Application developer profile on JPL AWS

    • devops#45 Have a github action self-hosted runner on NGAP

    • devops#46 Fully terraform the registry application

nucleus

None

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

  • nucleus#17 Deploy Baseline Nucleus for SBN-PSI Catalina Sky Survey Pilot

  • nucleus#38 Develop Logging and Monitoring Strategy

  • nucleus#41 Determine DAG definition standard

registry

PDS Registry provides service and software application necessary for tracking, searching, auditing, locating, and maintaining artifacts within the system. These artifacts can range from data files and label files, schemas, dictionary definitions for objects and elements, services, etc.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

  • registry#47 Support LBLX extension in Registry Loader tools

  • registry#99 Populate EN Registry with PDS3 Data Sets

    • registry#62 As a user, I want to ingest legacy PDS3 data sets from EN legacy registry

    • registry#63 Deploy harvest / registry-mgr as cron to regularly upload PDS EN data to registry

    • operations#270 Deployment of Registry Loader Tools and Initial Ingestion of Engineering Node Registry on Latest AWS Deployment

    • registry#168 As a manager, I want to see the progress of data sets ingested into registry vs. legacy registry

  • registry#178 Improved Fault Tolerance for Registry and Registry API

    • registry-common#26 [SECURITY] Fix code scanning alert - TrustManager that accepts all certificates

    • registry-api#292 Queries for data products with lots of metadata attributes crash in browser

    • registry-api#297 As a EN Operator, I want to the registry API to be fault tolerant

    • registry#176 As a EN Operator, I want to the registry to be fault tolerant

    • registry-api#300 Update Java params and Docker image to expand JVM memory beyond 50%

    • harvest#119 As a developer, I want to know what version of Harvest was used to load a product

  • registry#181 Support Registration and Search of Products Replicated Across Archives

    • registry#93 As a user, I want to be able to ingest mirrored archive data

    • registry-api#306 If >1 products with the same LIDVID are registered by different nodes, we should respond with one of them

  • registry#185 Implement Registry Multi-tenancy

    • registry-mgr#66 Update to utilize new multi-tenancy approach

    • registry#179 Implement authorization handshake with API Gateway and Lambda for multi-tenancy approach

    • harvest#118 Update to utilize new multi-tenancy approach

    • registry-api#304 Update to utilize new multi-tenancy approach

    • validate#621 Re-test validate-refs using populated multi-tenant registry

    • registry#188 Update Terraform scripts to support multi-tenancy

  • registry#186 Enhance Terraform Scripts for Continuous Deployment and MCP

registry-api

Web API service for the PDS Registry, providing the implementation of the PDS Search API (https://github.com/nasa-pds/pds-api) for the PDS Registry.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release
  • registry-api#285 Implement Faceting

    • registry-api#283 As a client developer, I want to facet on 1 or more fields in the registry

    • registry-api#284 As an API user, I want to know the unique values for a specific API field.

    • registry-api#323 As a user, I want the /properties endpoint to display counts for products having each property

    • registry-api#324 As a user, I want the /properties endpoint to display range/enum type and values for each property

    • registry-api#325 As a user, I want the /properties endpoint to include a link to explanatory documentation (i.e. DD or similar)

data-upload-manager

Data Upload Manager (DUM) component for managing the interface for data uploads to the Planetary Data Cloud from Data Providers and PDS Nodes.

User Guide

Github Repo

Issue Tracking

Backlog

Stable Release

Dev Release

Other Information

Reference Documents

This section details the controlling and applicable documents referenced for this release. The controlling documents are as follows:

  • PDS Level 1, 2 and 3 Requirements, April 20, 2017.

  • PDS4 Project Plan, July 17, 2013.

  • PDS4 System Architecture Specification, Version 1.3, September 1, 2013.

  • PDS4 Operations Concept, Version 1.0, September 1, 2013.

  • PDS General System Software Requirements Document (SRD), Version 1.1, September 1, 2013.

  • PDS Harvest Tool Software Requirements and Design Document (SRD/SDD), Version 1.2, September 1, 2013.

  • PDS Preparation Tools Software Requirements and Design Document (SRD/SDD), Version 0.3, September 1, 2013.

  • PDS Registry Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.

  • PDS Report Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.

  • PDS Search Service Software Requirements and Design Document (SRD/SDD), Version 1.0, September 1, 2013.

    • PDS Search Scenarios, Version 1.0, September 1, 2013.

    • PDS Search Protocol, Version 1.2, March 21, 2014.

    • PDAP Search Protocol, Version 1.0, March 21, 2014.

  • PDS Security Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.

  • PDS NSSDC Delivery Software Requirements and Design Document (SRD/SDD), Version 0.1, October 29, 2019.

System Requirements, Dependencies, and Assumptions

See individual applications for specific dependencies and system requirements.