Built by Maven

Installation

This document describes how to install the Registry software contained in the registry-mgr-legacy package. The following topics can be found in this section:

System Requirements

This section details the system requirements for installing and operating the Registry.

Java Runtime Environment

The Registry was developed using Java and will run on any platform with a supported Java Runtime Environment (JRE). The software was specifically compiled for and tested in Java version 1.8 (64-bit). The following commands test the local Java installation in a UNIX-based environment:

NOTE: This software only works with 64-bit Java installation.

% which java
/usr/bin/java

% java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

The first command above checks whether the java executable is in the environment's path and the second command reports the version. If Java is not installed or the version is not at least 1.8, Java will need to be downloaded and installed in the current environment. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Java software can be downloaded from the Oracle Java Download page. The suggested software package is the Java Standard Edition (SE) 8, either the JDK or the JRE package. The JDK package is not necessary to run the software but could be useful if development and compilation of Java software will also occur in the current environment.

Memory Requirements

      The Registry requires >2GB memory available in order to perform normal operations

Unpacking the Package

Download the registry-mgr-legacy package from Github (TAR.GZ ZIP). The binary distribution is available in identical zip or tar/gzip packages. The installation directory may vary from environment to environment but in UNIX-based environments it is typical to install software packages in the /home/pds directory and in Windows-based environments it is typical to install software packages in the C:\Program Files directory. Unpack the selected binary distribution file with one of the following commands:

The top-level directory where the package is installed will be referenced in these instructions as $REGISTRY_HOME.

% unzip registry-mgr-legacy-2.3.6-bin.zip
or
% tar -xzvf registry-mgr-legacy-2.3.6-bin.tar.gz

Note: Depending on the platform, the native version of tar may produce an error when attempting to unpack the distribution file because many of the file paths are greater than 100 characters. If available, the GNU version of tar will resolve this problem. If that is not available or cannot be installed, the zipped package will work just fine in a UNIX environment.

The commands above result in the creation of the registry-mgr-legacy-2.3.6 directory with the following directory structure:

  • LICENSE

    The Apache License, Version 2.0

  • README.txt

    A README file with some high level information on the project and how to view more detailed documentation.

  • VERSION.txt

    A VERSION file contains the version of the project.

  • bin/

    This directory contains the scripts for installing or uninstalling.

  • build/

    This directory contains the scripts for installing or uninstalling the project in Docker.

  • dist/

    This directory contains the jar for the Registry API (PDS Search Protocol).

  • collections/

    This directory contains the Apache Solr configuration for collections installed by default by the Registry.

  • lib/

    This directory contains JARs used by some of the registry utility tools.

Configuring the Environment

In order to execute the Harvest Tool, the local environment must first be configured appropriately. This section describes how to setup the user environment on UNIX-based and Windows machines.

UNIX-Based Environment

This section details the environment setup for UNIX-based machines. The binary distribution includes a couple shell scripts that must be executed from the command-line. Setting the REGISTRY_HOME environment variable to where the Registry is installed can help when using Harvest in the future, but is not required.

The following command demonstrates how to set the REGISTRY_HOME environment variable (in Bourne shell), by appending to its current setting. You may want to append this to the end of your $HOME/.bash_profile (or equivalent) to avoid having to reset this in the future: 

          % export REGISTRY_HOME=/path/to/registry-mgr-legacy-2.3.6

In addition, the shell scripts require that the Java be installed on the machine, you can both check it is installed or set the JAVA_HOME environment variable: 

          # Check if it is installed globally
          % which java
          
          # If not, you can set JAVA_HOME
          % export JAVA_HOME=/path/to/java/home

The system administrator for the local machine may need to be consulted for this location. The path specified should have a bin sub-directory that contains the java executable. This variable may also be defined within the scripts. Edit the scripts (files without the .bat extension) and change the line in the example above to represent the local Java installation.

Windows Environment

This section details the environment setup for Windows machines. The binary distribution includes a couple batch scripts that must be executed from the command-line. Setting the PATH environment variable to the location of the files, enables the batch scripts to be executed from any location on the local machine.

The following command demonstrates how to set the PATH environment variable, by appending to its current setting: 

          C:\> set PATH = %PATH%;C:\Program Files\registry-mgr-legacy-2.3.6\bin

In addition, the batch scripts require that the JAVA_HOME environment variable be set to the appropriate location of the Java installation on the local machine. This may have already been set when Java was installed. However, if it hasn't, then run the following command to set the JAVA_HOME environment variable: 

          C:\> set JAVA_HOME = C:\path\to\java\home

The system administrator for the local machine may need to be consulted for this location. The path specified should have a bin sub-directory that contains the java executable. This variable may also be defined within the scripts. Edit the scripts (files with the .bat extension) and change the line in the example above to represent the local Java installation. Additional methods for setting Windows environment variables can be found in the Windows System Properties document.

Install Solr and Registry - Docker

NOTE: Docker installation is currently only available for Linux / Mac OS X users. Windows Users please use the Standalone Installation

This section details how to install and deploy a dockerized instance of the PDS Registry.

The following Registry installation procedure is for Linux and Mac operating systems. Please contact Engineering Node if you run into any issues.

  1. Install and Run Docker from Docker Installation Guide website if your system doesn't have Docker.

  2. Move to the directory that registry installer script resides. 
    % cd $REGISTRY_HOME/bin
  3. Execute the Registry Docker Installer script and you should see the following output if successful: 
    % ./registry_installer_docker.sh install

Installing...
Building Registry Docker Image.                               SUCCESS
Starting Registry Docker Container                            SUCCESS
Waiting for Solr server to start.
Creating a Registry Service Blob collection (registry)        SUCCESS
Creating a XPath collection (xpath)                           SUCCESS
Creating a Search collection (data)                           SUCCESS
  4. Next, you can Verify Successful Installation and potentially configure for Operational Deployment for setting this up on an operational, public server.

Install Solr and Registry - Standalone

This section of the Registry Installation guide is for installing Solr and the Registry as a standalone application instead of using Docker. The Docker installation method is the preferred method for ease of deployment across multiple discipline nodes, but not required.

Note: The following procedure can be followed by either Linux or Windows users, however the paths and some of the commands will need to be modified if not performed in a Cygwin or equivalent POSIX-compatible environment.

  1. Download Solr

    Unpack Solr package and note the path to the Solr instance.
    Note: It is preferred that the Solr package have same parent directory as Registry / Harvest.

    % tar -xvf solr-8.2.0.tgz
% pwd
/usr/local/

% ls
harvest-x.x.x/
registry-mgr-legacy-2.3.6/
solr-8.2.0/

# If different location, note location of SOLR
% cd solr-8.2.0/
% pwd
/usr/local/solr-8.2.0/
  2. Move to the directory that registry installer script resides (Note: See Unpacking the Package for configuring the REGISTRY_HOME variable or find out what that path represents.) 
    % cd /usr/local/registry-mgr-legacy-2.3.6/bin
  3. Execute the registry installer software: 
    % ./registry_installer_standalone.sh
  4. Enter the Solr installation location that is done in previous step. 
    Enter location of Solr installation: /usr/local/solr-8.2.0

    You should see output similar to the following: 

    STARTING Registry Installer in standalone mode.

  Registry   .....
       ( v 2.3.6 )
       ( installing on platform: Mac OS X )
       ( IP Address:- 192.168.1.120 )
       ( Host Name:- MY HOST )

Enter location of SOLR installation: /usr/local/solr-8.2.0
Copied a directory from /usr/local/registry/registry-2.3.6/bin/../dist to /usr/local/solr-8.2.0/contrib/pds/lib
Copied a directory from /usr/local/registry/registry-2.3.6/bin/../collections/data to /usr/local/solr-8.2.0/server/solr/configsets/data/conf
Copied a directory from /usr/local/registry/registry-2.3.6/bin/../collections/registry to /usr/local/solr-8.2.0/server/solr/configsets/registry/conf
Copied a directory from /usr/local/registry/registry-2.3.6/bin/../collections/xpath to /usr/local/solr-8.2.0/server/solr/configsets/xpath/conf

Starting a SOLR server... Waiting up to 180 seconds to see Solr running on port 8983...
*** [WARN] *** Your open file limit is currently 10240.
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
*** [WARN] ***  Your Max Processes Limit is currently 1418.
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Waiting up to 180 seconds to see Solr running on port 8983 [/]
Started Solr server on port 8983 (pid=6007). Happy searching!


Return status from the SOLR server startup = 0
The SOLR server is started successfully.
Creating registry collection...
Created collection 'registry' with 3 shard(s), 1 replica(s) with config-set 'registry'
Return status from creating registry collection = 0
Registry collection is created successfully.
Creating xpath collection...
Created collection 'xpath' with 3 shard(s), 1 replica(s) with config-set 'xpath'
Return status from creating xpath collection = 0
Xpath collection is created successfully.
Creating search collection (data)...
Created collection 'data' with 3 shard(s), 1 replica(s) with config-set 'data'
Return status from creating search collection (data) = 0
Search collection (data) is created successfully.
  5. Next, you can Verify Successful Installation and potentially configure for Operational Deployment for setting this up on an operational, public server.

Verifying Successful Installation

  • To verify installation, first, check the docker image is running: 
    % docker ps

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
0ee274f1fa5f        registry:2.2.0   "/bin/bash -c 'solr …"   5 minutes ago       Up 5 minutes        0.0.0.0:8983->8983/tcp   registry
  • Next, you can check the solr endpoint is up, however, how to test that will depend upon your server configuration:
    • If running on your local machine, open up your favorite browser and go to http://localhost:8983/solr to view the Solr admin interface.
    • If running on a server and the port is open, replace local host with your domain, e.g. http://pds.nasa.gov:8983/solr
    • If running on a server and the port is not open, you can run a curl command to verify Solr is active: 
      % curl http://localhost:8983/solr/admin/cores?action=STATUS

{
"responseHeader":{
"status":0,
"QTime":4},
"initFailures":{},
"status":{
"data_shard1_replica_n1":{
  "name":"data_shard1_replica_n1",
  "instanceDir":"/var/solr/data/data_shard1_replica_n1",
  "dataDir":"/var/solr/data/data_shard1_replica_n1/data/",
  "config":"solrconfig.xml",
  "schema":"managed-schema",
  "startTime":"2019-10-25T04:06:13.058Z",
  "uptime":551826,
  "lastPublished":"active",
  "configVersion":0,
  "cloud":{
    "collection":"data",
    "shard":"shard1",
    "replica":"core_node3"},
  "index":{
    "numDocs":0,
    "maxDoc":0,
    "deletedDocs":0,
    "indexHeapUsageBytes":0,
    "version":2,
    "segmentCount":0,
    "current":true,
    "hasDeletions":false,
    "directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/var/solr/data/data_shard1_replica_n1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5566110; maxCacheMB=48.0 maxMergeSizeMB=4.0)",
    "segmentsFile":"segments_1",
    "segmentsFileSizeInBytes":69,
    "userData":{},
    "sizeInBytes":69,
    "size":"69 bytes"}},
    .
    .
    .
  • You can learn more about the Registry Solr Collections installed, or continue onto Next Steps.

Operational Deployment

Once you are ready to deploy the Registry operationally, you will want to secure Solr by setting up a reverse proxy in your Apache HTTP Server httpd.conf file:

ProxyPass /services/registry/ http://localhost:8983/solr/
ProxyPassReverse /services/registry/ http://localhost:8983/solr/
ProxyPass /services/registry http://localhost:8983/solr
ProxyPassReverse /services/registry http://localhost:8983/solr

<!-- Secures the Solr admin interface to your local subnet. Change your IP/subnet information accordingly. -->
<Location "/services/registry/#/">
Order Deny,Allow
Deny from all
Allow from 123.45.0.0/255.255.0.0 123.56.0.0/255.255.0.0 .jpl.nasa.gov
</Location>

You should now be able to go to http://my.website.com/services/registry and see the Solr Admin interface.

Next Steps

Install Harvest Tool

The Harvest Tool can be used to populate the Registry with data. The Harvest Tool runs locally at the Discipline Node to crawl the local data repository in order to discover products and index associated metadata with the Registry.

Uninstall

In the event you want to uninstall the Registry either to install a newer version or for another reason:

  • For Docker Installation: 
    % ./registry_installer_docker.sh uninstall
Uninstalling...

Are you sure you want to uninstall the Registry and Search and all associated indices? (y/n) y
Removing the Registry Blob collection from the SOLR.
Removing the Registry XPath collection.
Removing the Search collection.
Stopping the SOLR instance.
Removing Registry Docker Images.
Removing 'solr_data' volume
  • For Standalone Installation: 
    $ ./registry_installer_standalone.sh uninstall
STARTING Registry Installer in standalone mode.

  Registry   .....
       ( v 2.3.6 )
       ( installing on platform: Mac OS X )
       ( IP Address:- 192.168.1.120 )
       ( Host Name:- MY_HOST )

Enter location of SOLR installation: /usr/local/solr-8.2.0
Deleting registry collection...
{
  "responseHeader":{
    "status":0,
    "QTime":198},
  "success":{"192.168.1.120:8983_solr":{"responseHeader":{
        "status":0,
        "QTime":47}}}}


Deleted collection 'registry' using command:
http://192.168.1.120:8983/solr/admin/collections?action=DELETE&name=registry
Return status from deleting registry collection = 0
Registry collection is deleted successfully.
Deleting xpath collection...
{
  "responseHeader":{
    "status":0,
    "QTime":187},
  "success":{"192.168.1.120:8983_solr":{"responseHeader":{
        "status":0,
        "QTime":54}}}}


Deleted collection 'xpath' using command:
http://192.168.1.120:8983/solr/admin/collections?action=DELETE&name=xpath
Return status from deleting xpath collection = 0
Xpath collection is deleted successfully.
Deleting search collection (data)...
{
  "responseHeader":{
    "status":0,
    "QTime":171},
  "success":{"192.168.1.120:8983_solr":{"responseHeader":{
        "status":0,
        "QTime":37}}}}


Deleted collection 'data' using command:
http://192.168.1.120:8983/solr/admin/collections?action=DELETE&name=data
Return status from deleting search collection (data) = 0
Search collection (data) is deleted successfully.

Found 1 Solr nodes:

Solr process 6007 running on port 8983
{
  "solr_home":"/usr/local/solr-8.2.0/server/solr",
  "version":"8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe - ivera - 2019-07-19 15:11:04",
  "startTime":"2019-10-25T04:41:04.215Z",
  "uptime":"0 days, 0 hours, 20 minutes, 2 seconds",
  "memory":"84.4 MB (%16.5) of 512 MB",
  "cloud":{
    "ZooKeeper":"localhost:9983",
    "liveNodes":"1",
    "collections":"0"}}


Stopping the SOLR server...
Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 6007 to stop gracefully.

The SOLR server is stopped successfully.

Common Errors

Failed to create a registry service collection (.system)...

This issue could happen for several reasons, but we will need to look through the logs to determine the issue:

  • Look at Solr Console log to determine where the issue may lie: 
    # STANDALONE INSTALLATION
open $SOLR_HOME/server/logs/solr.log
open $SOLR_HOME/server/logs/solr-8983-console.log
  • If the log output states: 
    Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap
    TBD solution

HTTP 404 - Requested Resource Not Available

This error means the application failed to install correctly. The following are potential mitigation strategies:

  • Verify domain and port are correct for the local Solr installation.
  • View $SOLR_HOME/server/logs/solr.log for information on potential causes of the failed deployment.

Other Errors

For other errors that arise while testing the installation, see the Operate - Common Errors