Command Line API

This section contains details on the interfaces for the command line tools.

pds-ingress-client

Client side script used to perform ingress request to the DUM service in AWS.

usage: pds-ingress-client [-h] [-c CONFIG_PATH]
                          -n {atm,eng,geo,img,naif,ppi,rs,rms,sbn}
                          [--prefix PREFIX] [--weblogs LOG_TYPE]
                          [--force-overwrite] [--include PATTERN]
                          [--exclude PATTERN] [--num-threads NUM_THREADS]
                          [--log-path LOG_PATH]
                          [--manifest-path MANIFEST_PATH]
                          [--report-path REPORT_PATH] [--dry-run]
                          [--log-level {warn,warning,info,debug}] [--version]
                          file_or_dir [file_or_dir ...]

Positional Arguments

file_or_dir

One or more paths to the files to ingest to S3. For each directory path is provided, this script will automatically derive all sub-paths for inclusion with the ingress request.

Named Arguments

-c, --config-path

Path to the INI config for use with this client. If not provided, the default config (/github/workspace/src/pds/ingress/conf.default.ini) is used.

-n, --node

Possible choices: atm, eng, geo, img, naif, ppi, rs, rms, sbn

PDS node identifier of the ingress requestor. This value is used by the Ingress service to derive the S3 upload location. Argument is case-insensitive.

--prefix, -p

Specify a path prefix to be trimmed from each resolved ingest path such that is is not included with the request to the Ingress Service. For example, specifying –prefix “/home/user” would modify paths such as “/home/user/bundle/file.xml” to just “bundle/file.xml”. This can be useful for controlling which parts of a directory structure should be included with the S3 upload location returned by the Ingress Service.

--weblogs, -w

Denotes the upload request as being for LOG_TYPE web logs. All uploaded files will be routed to a special S3 location reserved for web log files. LOG_TYPE denotes the type of web logs being uploaded, and becomes part of the destination upload path. If provided, –prefix must be provided as well.

--force-overwrite, -f

By default, the DUM service determines if a given file has already been ingested to the PDS Cloud and has not changed. If so, ingress of the file is skipped. Use this flag to override this behavior and forcefully overwrite any existing versions of files within the PDS Cloud.

Default: False

--include, -i

Specify a file path pattern to match against when determining which files should be included with an Ingress request. Unix-style wildcard patterns are supported. Include patterns are always applied prior to any Exclude patterns. This argument can be specified multiple times to configure multiple include patterns. Include patterns are evaluated in the order they provided.

Default: []

--exclude, -e

Specify a file path pattern to match against when determining which files should be excluded from an Ingress request. Unix-style wildcard patterns are supported. Exclude patterns are always applied after any Include patterns. This argument can be specified multiple times to configure multiple exclude patterns. Exclude patterns are evaluated in the order they provided.

Default: []

--num-threads, -t

Specify the number of threads to use when uploading files to S3 in parallel. By default, all available cores are used.

Default: -1

--log-path

Specify a file path to write logging statements to. These will include some of the messages logged to the console, as well as additional messages about the status of each file/batch transfer. By default, the log file is created in a temporary location if this parameter is not provided. If provided, this argument takes precedence over what is provided for OTHER.log_file_path in the INI config.

--manifest-path

Specify a file path to a JSON manifiest of all files indexed for inclusion in the current ingress request. If the provided path is not an existing file, then the manifest will be written to that location. If the path already exists, this script will read the manifiest, and skip checksum generation for any paths that are already specified. If not provided, no manifiest is written or read.

--report-path, -r

Specify a path to write a JSON summary report containing the full listing of all files ingressed, skipped or failed. By default, no report is created.

--dry-run

Derive the full set of ingress paths without performing any submission requests to the server.

Default: False

--log-level, -l

Possible choices: warn, warning, info, debug

Sets the Logging level for logged messages. If not provided, the logging level set in the INI config is used instead.

--version

Print the Data Upload Manager release version and exit.