Build 14.1 Plan
The Planetary Data System (PDS) is a long-term archive of digital data products returned from NASA’s planetary missions, and from other kinds of flight and ground-based data acquisitions, including laboratory experiments. But it is more than just a facility - the archive is actively managed by planetary scientists to help ensure its usefulness and usability by the world wide planetary science community.
The following page outlines the planned PDS software enhancements, bug fixes, and impacted requirements for the upcoming Build .
A more detailed breakdown of these tasks by date and priority can be found in our Zenhub Roadmap (login required).
Key Dates
Date |
Name |
Responsible Party |
Description |
---|---|---|---|
02/12/24 |
SCR Freeze Date |
DDWG |
Freeze date for identifying SCRs to be implemented in this release. |
02/13/24 - 04/12/24 |
Standards Documents Updates Due |
Document Authoring Team |
Updates to PDS4 standards documents due to EN for review. |
04/23/24 - 05/21/24 |
Beta Testing Period |
DNs, IPDA |
Changes to PDS4 IM and system components available for beta testing. |
04/23/24 - 05/21/24 |
dLDD Integration and Test |
dLDD Stewards |
Auto-generated dLDDs should be reviews by dLDD Stewards, and final updates to dLDDs should be completed during this time. |
04/12/24 - 05/22/24 |
Standards Documents Review Due |
Document Review Team |
Review updates to PDS4 standards documents. |
06/07/24 |
System Release |
EN |
PDS4 system is operationally deployed and website is updated accordingly. |
Date |
Name |
Description |
---|---|---|
03/21/24 |
Delivery to I&T |
The PDS4 IM and all software updated during this build cycle are delivered to EN I&T. |
04/8/24 |
Test Readiness Review (TRR) |
PDS EN internal review to verify readiness to enter I&T phase. |
05/30/24 |
Delivery and Deployment Review (DRR) |
PDS internal review with PDS Software Working Group to verify readiness to deploy build system. |
06/07/24 |
Operational Deployment |
PDS4 system is operationally deployed and website is updated accordingly. |
For a more detailed schedule, see https://pds-engineering.jpl.nasa.gov/content/schedules.
Planned PDS4 Information Model Changes
This section details the planned changes to the PDS4 Information Model for this build. These changes are deliverables by the PDS Data Design Working Group (DDWG) to the Engineering Node for implementation.
pds4-information-model#663 CCB-367: Superseded_by / Supersedes reference type
Planned Software Changes
This section details the planned software changes for this build.
Planned tasks for the next release cycle are grouped by software component, and sub-tasks are included where known at the time of this plan generation. Each section also includes additional information, such as:
Software description
Link to 🔍 User Guide
Link to 💻 Github Repo
Link to 🚑 Issue Tracking
Link to 📒 Backlog - future work not planned for this release cycle
Don’t see the tool you are looking for? This means there is no active development planned for this build cycle. Head to the PDS Tool Registry to see all possible tools available for PDS data.
Have a new feature or tool you think we should be working on? Create a ticket in the applicable Github repo or in our PDS EN Operations repo and we will see how that can fit into our plans moving forward.
validate
Validates PDS4 product labels, data and PDS3 Volumes
- validate#695 Update Validate Test Suite to Support Larger Data Sets
validate#633 Reintegrate cucumber test for ticket #599
validate#659 move test data to git large file storage repo
- validate#696 Use New Registry API for Context Product Validation
validate#592 As a user, I want to when a context product has been deprecated
validate#675 Cutover to using new Registry API for generating context products json
pds4-information-model
The software tools and data necessary for generating the Information Model including PDS4 ontology, data, and information model.
- pds4-information-model#700 Document Nuances for Reusability of Generic LDD Classes/Attributes
pds4-information-model#469 As a namespace steward, I want to provide a more specific definition/nuance for a generic attribute
pds4-information-model#701 Implement Cucumber Framework for LDDTool
- pds4-information-model#703 B14.1 Information Model SCR Implementation
pds4-information-model#663 CCB-367: Superseded_by / Supersedes reference type
pds4-information-model#704 B14.1 Information Model Delivery to I&T
pds4-information-model#705 B14.1 Standards Documents Updates
harvest
Standalone Harvest client application providing the functionality for capturing and indexing product metadata into the PDS Registry system (https://github.com/nasa-pds/registry).
- harvest#129 Add Support for LBLX File Extension
harvest#130 As a user, I want to ingest data products with labels having .lblx file extension
- harvest#131 Enhance support for searching lid/lidvid references
registry#215 Verify fields types are changed when LDD fields are reloaded
software-issues-repo
Issue tracking repository as a centralized entry point for general PDS software bugs and feature requests.
- software-issues-repo#67 Migrate PDS JIRA to Atlassian Cloud JIRA
software-issues-repo#68 Complete URS request for pds-jira.jpl.nasa.gov
software-issues-repo#69 Complete JIRA Open Source form for PDS JIRA
software-issues-repo#70 Export and Migrate JIRA to Atlassian Cloud
- software-issues-repo#80 Upgrade Legacy Registry Tools and Services to Fix Vulnerability
operations#431 Deploy and Test legacy registry components on OL8
registry-pds3-catalog#4 Upgrade Catalog Tool and dependencies to support Harvest/Registry upgrades
ds-view#3 Upgrade Dataset View and dependencies to support Harvest/Registry upgrades
search-ui-legacy#1 Upgrade Keyword Search and dependencies to support Harvest/Registry upgrades
registry-mgr-legacy#3 Upgrade legacy registry mgr with upgraded log4j libraries and Solr 8.11.2
- software-issues-repo#86 Complete Tasks and Artifacts Supporting NASA A&A Process
software-issues-repo#54 As a user, I want to ensure no passwords/secrets are committed to a PDS repo
software-issues-repo#87 Develop Contingency Plans
software-issues-repo#88 Develop Disaster Recovery Plan
software-issues-repo#89 Develop Incident Response Plan
software-issues-repo#90 Update Software Management / Configuration Management Plans
software-issues-repo#91 Complete Contingency Plan Test
software-issues-repo#92 Develop Boundary Diagram for PDS-managed systems
software-issues-repo#95 Develop Continuous Monitoring Plan
software-issues-repo#93 B14.1 Bi-Annual Triage CodeQL Static Code Analysis Scan Results
software-issues-repo#94 B14.1 Prep for I&T
portal-wp-tasks
PDS Web Design System
portal-wp-tasks#42 Deploy Dev Wordpress Environment in MCP
- portal-wp-tasks#44 Develop Initial Headless Wordpress Docker Deployment
portal-wp-tasks#43 As a developer, I want to deploy a local instance of headless wordpress and latest code using Docker
portal-wp-tasks#45 Why headless over vanilla Wordpress?
- portal-wp-tasks#49 Implement Header
portal-wp-tasks#50 Implement Navbar
portal-wp-tasks#51 Implement Titlebar (logo, node navigation, search)
portal-wp-tasks#55 Updated header components
portal-wp-tasks#56 Implement Initial Investigation Landing Page Template and Associated Components
portal-wp-tasks#57 Implement Initial Bundle/Collection Landing Pages and Associated Components
portal-wp-tasks#58 Implement Initial Search Results Page and Associated Components
operations
Tickets for the PDSEN Operations Team
- operations#379 Phase 2: Wordpress migration for pds-engineering website
operations#338 Request last re-sync of most recent files then verify
operations#339 Test redirects
operations#343 Install Feedback widget in WordPress PDS Engineering site
operations#362 update information file for Metrics’ Multi-Selection Visualization Chart
operations#391 Migrate and verify latest content for PDS Eng WP
operations#392 Incorporate JPL logo into header for PDS Eng WP
operations#399 Submit new URS request
operations#424 Add Data Registration Validation and Deep Archive Execution to Nominal Data Release Process
operations#460 B14.1 dLDD Build and I&T
operations#456 B14.1 EN Deployment & Release
roundup-action
Do a “roundup”, a/k/a PDS-style continuous integration and delivery
- roundup-action#114 Develop CI/CD for React Projects
wds-react-legacy#95 Setup NPM Audits as part of CI on React projects
roundup-action#118 Develop React template repo
roundup-action#119 Develop Branch Testing Github Action
roundup-action#120 Develop Unstable GitHub Action CI/CD to I&T Staging venue
roundup-action#121 Develop Stable GitHub Action CI/CD to I&T production venue
roundup-action#122 Develop PR Review GitHub Action CI/CD to Developer Staging venue
roundup-action#123 Develop Initial Architecture and Action Flow for React CI/CD Deployments
doi-ui
The web interface for the PDS DOI Service providing the ability management PDS archive DOIs. See the DOI Service for more details on the available capabilities. https://nasa-pds.github.io/doi-service/
doi-ui#184 Develop DOI Service User Guide and Training
search-api-notebook
Jupyter notebooks for demonstrating and utilizing the Planetary Data System (PDS) Search API
- search-api-notebook#24 Geographic data extraction and visualization
search-api-notebook#12 Test extracting lat/long bounding box info from python component with STAC API
search-api-notebook#14 Connect leaflet-cartocosmos component to the TBD data bounding box attributes in the registry-api
search-api-notebook#25 Use GDAL Digital Elevation Model visualization features
planetary-data-cloud
PDS Cloud Migration documentation, issue, tracking and simple tools for assisting in the PDS hybrid cloud study and migration efforts.
- planetary-data-cloud#35 MCP Pilot: Deploy Registry
registry#218 Add registry costs to the PDS EN Cloud spreadsheet
registry#222 Setup OIDC Authentication for MCP
registry#223 OpenSearch Serverless Setup in MCP
planetary-data-cloud#61 Design and Implement Initial Archive Browser Beta
planetary-data-cloud#69 Design and Implement Minor Planet Center Cloud Backup Solution
planetary-data-cloud#71 CSS MVP: Identify Requirements and Prep for CSS in AWS Open Data Registry
- planetary-data-cloud#76 Design and Implement Warm Storage Data Backup Solution
planetary-data-cloud#75 As a user, I want a warm backup of my archive data to be available in the event of corruption of of primary archive data
nucleus#73 As a user, I want to store a copy of my archive data in AWS Deep Archive / Glacier in another region from the primary copy
planetary-data-cloud#77 B14.1 Bi-Annual Cost Model Review
- planetary-data-cloud#79 Migrate Existing JPL AWS Infra to MCP
planetary-data-cloud#81 Migrate PDS EC2 instances from JPL AWS to MCP
devops
Parent repo for PDS DevOps activities
- devops#37 Automatically deploy an up to date I&T platform where all software can be tested (Part 2)
registry-api#269 Add registry/docker compose integration tests to github action on dev branches
devops#36 For registry components, add a github action which runs the integration test when something is pushed on a dev branch
registry-sweepers#6 Create the docker image and push it to docker hub as part of the CICD
devops#41 Push docker image on AWS ECR in github-actions, in addition to docker hub
devops#44 Ask new Application developer profile on JPL AWS
devops#45 Use Github Actions Github-Hosted Runner to deploy to NGAP
devops#46 Fully terraform the registry application
devops#47 Other CICD improvements
devops#48 Use JPL AWS jenkins to launch terraform scripts
nucleus
Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
nucleus#17 CSS MVP: Deploy Baseline Nucleus for Catalina Sky Survey
- nucleus#66 MCP Pilot: Pilot Nucleus Deployment for CSS and MESSENGER MDIS Data
nucleus#52 Work with IMG to connect to S3 Bucket with MESSENGER data
nucleus#53 Deploy and Test Nucleus DAG for MESSENGER processing
nucleus#54 Validate and Load all PDS4 MESSENGER data products with Nucleus
nucleus#61 Manually validate and load MESSENGER MSGRMDS_4001 and MESSDEM_1001 data
nucleus#68 Setting-up MWAA Environment on MCP
nucleus#74 Support processing of PDS products with fitz file format in Nucleus
nucleus#62 Develop Support For Off-Nominal Data Loads
registry
PDS Registry provides service and software application necessary for tracking, searching, auditing, locating, and maintaining artifacts within the system. These artifacts can range from data files and label files, schemas, dictionary definitions for objects and elements, services, etc.
- registry#155 Registry Multi-tenancy Design and implementation with Cognito
registry#179 Implement authorization handshake with API Gateway and Lambda for multi-tenancy approach
- registry#178 Improved Fault Tolerance for Registry and Registry API
registry-common#26 [SECURITY] Fix code scanning alert - TrustManager that accepts all certificates
harvest#101 Unexpected WARNINGs related to http requests
registry-api#292 Queries for data products with lots of metadata attributes crash in browser
registry-api#300 Update Java params and Docker image to expand JVM memory beyond 50%
harvest#119 As a developer, I want to know what version of Harvest was used to load a product
registry-api#337 Enhance robustness of API Monitoring
registry-sweepers#25 Enhance provenance sweeper with more graceful failure when registry contains zero documents
harvest#125 Improve Fault Tolerance of Harvest for Forbidden Access error and Timeout
registry#211 As an operator, I want to be notified of when Registry storage capacity exceeds 75% capacity.
registry-api#361 As a user, I want my API request to execute successfully even when the registry contains corrupted documents
registry#224 Write documentation and Training for Registry monitoring
registry#225 Registry-Sweepers Multiple Updates Detected
registry-sweepers#69 Registry-Sweepers Error: contained no hits when hits were expected
registry-api#378 Investigate and fix intermittent Registry-API Errors
harvest#133 Data too large error from very large data products
harvest#134 Too many requests error to OpenSearch
- registry#181 Support Registration and Search of Products Replicated Across Archives
registry-api#306 If >1 products with the same LIDVID are registered by different nodes, we should respond with one of them
registry#245 As a user, I want to register data product(s) that is mirrored across 2 or more organizations
- registry#185 Implement Registry Multi-tenancy without Cognito in the loop
registry-mgr#66 Update to utilize new multi-tenancy approach
harvest#118 Update to utilize new multi-tenancy approach
registry-api#304 Update to utilize new multi-tenancy approach
validate#621 Re-test validate-refs using populated multi-tenant registry
registry#188 Update Terraform scripts to support multi-tenancy
registry#212 Design scalable, multi-tenant opensearch
registry#217 Write a migration plan to multitenant registry
registry-sweepers#60 Registry-Sweeper ECS Enchancements (Post Multi-Tenancy)
registry#223 OpenSearch Serverless Setup in MCP
registry#230 Enable Support for Changing Field Types
- registry#231 Design and Implement Support for Alternate Data File Paths
registry#96 As a user I want to change the file paths of products in the registry
harvest#123 As a Node Operator, I want to specify an alternate file paths for 1 or more archive products
registry#209 As a user, I want to manually update a product’s file location (file_ref)
registry-api
Web API service for the PDS Registry, providing the implementation of the PDS Search API (https://github.com/nasa-pds/pds-api) for the PDS Registry.
- registry-api#285 Implement Faceting
registry-api#283 As a client developer, I want to facet on 1 or more fields in the registry
registry-api#284 As an API user, I want to know the unique values for a specific API field.
registry-api#323 As a user, I want the /properties endpoint to display counts for products having each property
registry-api#324 As a user, I want the /properties endpoint to display range/enum type and values for each property
registry-api#325 As a user, I want the /properties endpoint to include a link to explanatory documentation (i.e. DD or similar)
planetary-data-engine
Free-text search capability for planetary data, services, tools, and information
- planetary-data-engine#5 Complete Sinequa Useability Analysis
planetary-data-engine#9 Make requirement analysis & trade study for PDS Web Search
planetary-data-engine#8 Working with Jupyter Notebook to show various features to be used by UI team
- planetary-data-engine#7 Develop Query Test Suite and Success Criteria
planetary-data-engine#14 Create test cases for evaluation of PDS search quality
planetary-data-engine#15 Design and develop a new tool to evaluate the quality of a set of search results
data-upload-manager
Data Upload Manager (DUM) component for managing the interface for data uploads to the Planetary Data Cloud from Data Providers and PDS Nodes.
data-upload-manager#4 Develop Cost Model
- data-upload-manager#13 CSS MVP: Deploy to MCP and Test Uploads to Nucleus
data-upload-manager#31 Upload test data set with manual trigger of Nucleus
data-upload-manager#32 Test upload subset of CSS with manual trigger of Nucleus
registry#238 Assist with DUM infra setup in MCP
nucleus#71 Support CSS demo
data-upload-manager#29 Develop Directory Write Locking Mechanism
Other Information
Reference Documents
This section details the controlling and applicable documents referenced for this release. The controlling documents are as follows:
PDS Level 1, 2 and 3 Requirements, April 20, 2017.
PDS4 Project Plan, July 17, 2013.
PDS4 System Architecture Specification, Version 1.3, September 1, 2013.
PDS4 Operations Concept, Version 1.0, September 1, 2013.
PDS General System Software Requirements Document (SRD), Version 1.1, September 1, 2013.
PDS Harvest Tool Software Requirements and Design Document (SRD/SDD), Version 1.2, September 1, 2013.
PDS Preparation Tools Software Requirements and Design Document (SRD/SDD), Version 0.3, September 1, 2013.
PDS Registry Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
PDS Report Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
- PDS Search Service Software Requirements and Design Document (SRD/SDD), Version 1.0, September 1, 2013.
PDS Search Scenarios, Version 1.0, September 1, 2013.
PDS Search Protocol, Version 1.2, March 21, 2014.
PDAP Search Protocol, Version 1.0, March 21, 2014.
PDS Security Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
PDS NSSDC Delivery Software Requirements and Design Document (SRD/SDD), Version 0.1, October 29, 2019.
System Requirements, Dependencies, and Assumptions
See individual applications for specific dependencies and system requirements.