ingest_hst_cal_oms.pl

Usage

>ingest_hst_cal_oms.pl -c <archive_class> -o <data_source> [-i <input_dir>]

Context

Inputs

-c {archive_class}

This is the archive_class of the data to be ingested. Must be either CAL or OMS.

-o {source}

This is the four-character source of the ingest override value used as the value for the ads_data_source field in the DADS catalog archive_data_set_all table. This is a way for Ops to distinguish between test data, normally processed pipeline data, and reingested data. E.g. REGR, TEST, RING, DLSG, etc.

-i {input_dir}

This is the optional location of the data. If not provided, current working directory is assumed.

Data

Data from different instruments may be combined in the input directory. However, data from OMS class must be processed separately from CAL class data, and must not be mixed in the same input dir. Take careful note of the files in your input dir and make sure none are from a different archive class (e.g. rootname_pdq.fits files should be handled separately). The tools trust the user to only put the correct set of data into the input directory. Of course, just to be confusing, ASN class data is sort of a special subset of both CAL and OMS. I.e. association files can go into the same directory as the files for the members of that association.

For OMS, you can now mix rootname.ext and rootname_ext.fits input. ASN files are always supposed to be rootname_asn.fits. For example, this is the OMS input that AUTO regr_00202 uses:

j8tlgfkmj.jif        o5el08020_asn.fits   o8tt7gmcj_jif.fits
j8tlgfkmj.jit        o5el08020_jif.fits   o8tt7gmcj_jit.fits
n6296q020.jif        o5el08020_jit.fits   u59g0108j.jif
n6296q020.jit        o5el08020_jwf.fits   u59g0108j.jit
n6296q020_asn.fits   o5el08020_jwt.fits

Outputs

No real output, but sets files up for the reingest pipeline. Dumps messages to the screen which could be redirected into a file.

Verbosity

For verbose messaging, especially for debugging, use the standard OPUS environment variable MSG_REPORT_LEVEL. There is no more -v option. So to turn messaging up for a run of the command-line tools, set the MSG_REPORT_LEVEL on the command line before you run the tools. This will only affect processes run in the same window thereafter, not the entire account you are running in.
E.g. setenv MSG_REPORT_LEVEL MSG_ALL

Note that MSG_REPORT_LEVEL and STDB_REPORT_LEVEL settings are changed right before calls to genreq, InteractiveIngResponse, and update_db_tool. In the case of genreq and ingrsp, this was done to make debugging these comand-line tools easier. It was thought we did not need to debug those processes since they were not changing. However, we may wish to reconsider this change in the future.

In the case of update_db_tool, no extra reporting is necessary as this tool already does a wonderful job of making itself clear. Any extra reporting is redundant and makes it more difficult to see what the tool is doing.

See_also

• catalog_hst_cal_oms.pl
• ingest_hst_non_cal.pl
• catalog_hst_non_cal.pl
• ingest_dads_logs.pl
• Reingest_Tools.html

Examples

#:144:sherbert@tufnel> unsetenv MSG_REPORT_LEVEL
#:145:sherbert@tufnel> unsetenv STDB_REPORT_LEVEL

#:210:sherbert@tufnel> ingest_hst_cal_oms.pl 
2010280141753-E-ERROR-Too few arguments
Usage: 
>ingest_hst_cal_oms.pl -i <input_dir> -c <archive_class> -o <data_source>
   
   All parameters are required but the order of [-flag value] pairs does
   not matter.  

   <input_dir>     is the location of the data to be archived.  
                   Only data from one type of archive_class should be here.
                   This is the optional parameter.  Defaults to current dir,
                   if omitted.

   <archive_class> is the either CAL or OMS.  

   <data_source>   is the four character data source override value to be 
                   set in the regression pipeline.   

#:211:sherbert@tufnel> ingest_hst_cal_oms.pl -c OMS -o TEST -i oms >& keepMe_OMS_ing.out

#:213:sherbert@tufnel> cat keepMe_OMS_ing.out
2010280141947-I-INFO-Starting... ingest_hst_cal_oms.pl -i oms -c OMS -o TEST
2010280141947-I-INFO-get_regr_path-Obtained null path name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/null.path
2010280141947-I-INFO-get_regr_path-Found INGEST_PATH_NAME: regr
2010280141947-I-INFO-check_pipeline-Pipeline test: osf_test -p regr -pr dataset
2010280141950-I-INFO-check_pipeline-regr is idle.
2010280141950-I-INFO-ingest_hst_cal_oms.pl-chdir'd to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/oms 
2010280141950-I-INFO-get_cal_oms_datasets-Adding j8tlgfkmj to dataset list.
2010280141950-I-INFO-get_path_dirs-Obtained path file name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/regr.path
2010280141950-I-INFO-get_path_dirs-Found INGEST_TRIGGER_ROOT: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/
2010280141950-I-INFO-get_path_dirs-Found INGEST_SOURCE_DIR: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/
2010280141950-I-INFO-set_data_source-Deleted old file from /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/.
2010280141950-I-INFO-set_data_source-Created /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/source_override.test
2010280141950-I-INFO-request_datasets-Processing with data_id Fgs with resource fgsreq.
2010280141950-I-INFO-request_datasets-Using trigger directory /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/hst/.
2010280141950-I-INFO-request_datasets-Processing dataset j8tlgfkmj.
2010280141953-I-INFO Initialized OPUS environment. (1)
2010280141953-I-INFO Registered callbacks. (1)
2010280141953-I-INFO GENREQ----------------------------------- (1)
2010280141953-I-INFO Spawning command: osf_present -p regr.path -f j8tlgfkmj -t Fgs (1)
2010280141957-I-INFO Processing type is GENERIC (1)
2010280141957-I-INFO 2 files are collected for archiving. (1)
2010280141957-I-INFO Copy .//j8tlgfkmj.jif to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//oms/j8tlgfkmj_jif.fits (1)
2010280141957-I-INFO Copy .//j8tlgfkmj.jit to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//oms/j8tlgfkmj_jit.fits (1)
2010280141957-I-INFO Archive files collected successfully. (1)
2010280141957-I-INFO-request_datasets-Dataset j8tlgfkmj successfully processed by genreq.
2010280141957-I-INFO-request_datasets-Created /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/hst/j8tlgfkmj.Fgs.
2010280141957-I-INFO-Completed...ingest_hst_cal_oms.pl -i /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/oms -c OMS -o TEST