Package gov.nasa.pds.registry.common.es.dao

Class DataLoader

java.lang.Object

gov.nasa.pds.registry.common.es.dao.DataLoader

public class DataLoader
extends Object

Loads data from an NJSON (new-line-delimited JSON) file into Elasticsearch. NJSON file has 2 lines per record: 1 - primary key, 2 - data record. This is the standard file format used by Elasticsearch bulk load API. Data are loaded in batches.

Author:

karpenko

Constructor Summary

Constructors

Constructor	Description
DataLoader(ConnectionFactory conFactory)	Constructor

Method Summary

Modifier and Type	Method	Description
int	loadBatch(List<String> data)	Load data into Elasticsearch
int	loadBatch(List<String> data, Set<String> errorLidvids)	Load data into Elasticsearch
int	loadBatch(List<String> data, Set<String> errorLidvids, int retries)	Load data into Elasticsearch
void	loadFile(File file)	Load data from an NJSON (new-line-delimited JSON) file into Elasticsearch.
void	loadZippedFile(File zipFile, String fileName)	Load data from a zipped NJSON (new-line-delimited JSON) file into Elasticsearch.
void	setBatchSize(int size)	Set data batch size

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DataLoader

public DataLoader(ConnectionFactory conFactory)
           throws Exception

Constructor

Parameters:

conFactory - instance of class gov.nasa.pds.registry.common.ConnectionFactory

Throws:

Exception - an exception

Method Details

setBatchSize

public void setBatchSize(int size)

Set data batch size

Parameters:

size - batch size

loadFile

public void loadFile(File file)
              throws Exception

Load data from an NJSON (new-line-delimited JSON) file into Elasticsearch.

Parameters:

file - NJSON (new-line-delimited JSON) file to load

Throws:

Exception - an exception

loadZippedFile

public void loadZippedFile(File zipFile,
 String fileName)
                    throws Exception

Load data from a zipped NJSON (new-line-delimited JSON) file into Elasticsearch.

Parameters:

zipFile - Zip file with an NJSON data file.

fileName - NJSON data file name in the Zip file.

Throws:

Exception - an exception

loadBatch

public int loadBatch(List<String> data,
 Set<String> errorLidvids)
              throws Exception

Load data into Elasticsearch

Parameters:

data - NJSON data. (2 lines per record)

errorLidvids - output parameter. If not null, add failed LIDVIDs to this set.

Returns:

Number of loaded documents

Throws:

Exception - an exception

loadBatch

public int loadBatch(List<String> data,
 Set<String> errorLidvids,
 int retries)
              throws Exception

Load data into Elasticsearch

Parameters:

data - NJSON data. (2 lines per record)

errorLidvids - output parameter. If not null, add failed LIDVIDs to this set.

retries - number of times to retry the request if an exception is thrown.

Returns:

Number of loaded documents

Throws:

Exception - an exception

loadBatch

public int loadBatch(List<String> data)
              throws Exception

Load data into Elasticsearch

Parameters:

data - data NJSON data. (2 lines per record)

Returns:

Number of loaded documents

Throws:

Exception - an exception