ingest_dads_logs.pl

This utility enables the sending of DADS logs files through the ingest pipeline. After running this tool, you must start all the processes for a reingest pipeline. The reingest pipeline must be clear before running this tool. At least one file in each dataset must have the extension ".log". There is no cataloging needed for DADS logs.

Usage
>ingest_dads_logs.pl -o <data_source> [-i <input_dir>]

Context

Inputs

-o {source}
This is the four-character source of the ingest override value used as the value for the ads_data_source field in the DADS catalog archive_data_set_all table. This is a way for Ops to distinguish between test data, normally processed pipeline data, and reingested data. E.g. REGR, TEST, RING, DLSG, etc.

-i {input_dir}
This is the optional location of the data. If not provided, current working directory is assumed.

Data

At least one file in each dataset must have the extension ".log". There is no catalog tool for DADS logs.

Outputs

No real output, but sets files up for the reingest pipeline. Dumps messages to the screen which could be redirected into a file.

Verbosity

For verbose messaging, especially for debugging, use the standard OPUS environment variable MSG_REPORT_LEVEL. There is no more -v option. So to turn messaging up for a run of the command-line tools, set the MSG_REPORT_LEVEL on the command line before you run the tools. This will only affect processes run in the same window thereafter, not the entire account you are running in.
E.g. setenv MSG_REPORT_LEVEL MSG_ALL

Note that MSG_REPORT_LEVEL and STDB_REPORT_LEVEL settings are changed right before calls to genreq, InteractiveIngResponse, and update_db_tool. In the case of genreq and ingrsp, this was done to make debugging these comand-line tools easier. It was thought we did not need to debug those processes since they were not changing. However, we may wish to reconsider this change in the future.

See_also

Examples 
#:225:sherbert@tufnel> ingest_dads_logs.pl 
2010280143916-E-ERROR-Too few arguments
Usage: 
>ingest_dads_log.pl -i <input_dir> -o <data_source>
   
   All parameters are required but the order of [-flag value] pairs does
   not matter.

   <input_dir>     is the location of the data to be archived.  
                   Only data from one type of archive_class should be here.
   
   <data_source>   is the four character data source override value to be
                   set in the regression pipeline.

   Note that the DADS log datasets must be in the default directory.  At least
   one file in each dataset must have the extension ".log".

#:249:sherbert@tufnel> ingest_dads_logs.pl -i dlg -o TEST2010280144903-I-INFO-Starting... ingest_dads_log.pl -i dlg -o TEST
2010280144903-I-INFO-get_regr_path-Obtained null path name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/null.path
2010280144903-I-INFO-get_regr_path-Found INGEST_PATH_NAME: regr
2010280144903-I-INFO-check_pipeline-Pipeline test: osf_test -p regr -pr dataset
2010280144906-I-INFO-check_pipeline-regr is idle.
2010280144906-I-INFO-ingest_dads_log.pl-chdir'd to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/dlg 
2010280144906-I-INFO-get_datasets-Adding ingest_therappe_2004_06_18 to dataset list.
2010280144906-I-INFO-check_datasets-Checking that every dataset has a file with extension .log.
2010280144907-I-INFO-get_path_dirs-Obtained path file name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/regr.path
2010280144907-I-INFO-get_path_dirs-Found INGEST_TRIGGER_ROOT: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/
2010280144907-I-INFO-get_path_dirs-Found INGEST_SOURCE_DIR: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/
2010280144907-I-INFO-get_root_dir-Found INGEST_PATH_ROOT: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/
2010280144907-I-INFO-set_data_source-Deleted old file from /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/.
2010280144907-I-INFO-set_data_source-Created /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/source_override.test
2010280144907-I-INFO-copy_datasets-Copying with data_id Dlg from ./ to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/dlg/.
2010280144907-I-INFO-copy_datasets-Using trigger directory /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/hst/.
2010280144907-I-INFO-copy_datasets-Copying dataset ingest_therappe_2004_06_18.
2010280144907-I-INFO-copy_datasets-Created /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/hst/ingest_therappe_2004_06_18.Dlg.
2010280144907-I-INFO-Completed...ingest_dads_log.pl -i /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/dlg -o TEST
Last Updated: OPUS 2010.4