catalog_hst_cal_oms.pl

This utility enables the cataloging of HST CAL and OMS archive class data without a pipeline.

Usage

>catalog_hst_cal_oms.pl -c <archive_class> [-i <input_dir>]

Context

Inputs

-c {archive_class}

This is the archive_class of the data to be ingested. Must be either CAL or OMS.

-i {input_dir}

This is the optional location of the data. If not provided, current working directory is assumed.

Data

Data from different instruments may be combined in the input directory. However, data from OMS class must be processed separately from CAL class data, and must not be mixed in the same input dir. Take careful note of the files in your input dir and make sure none are from a different archive class (e.g. rootname_pdq.fits files should be handled separately). The tools trust the user to only put the correct set of data into the input directory. Of course, just to be confusing, ASN class data is sort of a special subset of both CAL and OMS. I.e. association files can go into the same directory as the files for the members of that association.

For OMS, you can now mix rootname.ext and rootname_ext.fits input. ASN files are always supposed to be rootname_asn.fits. For example, this is the OMS input that AUTO regr_00202 uses:

j8tlgfkmj.jif        o5el08020_asn.fits   o8tt7gmcj_jif.fits
j8tlgfkmj.jit        o5el08020_jif.fits   o8tt7gmcj_jit.fits
n6296q020.jif        o5el08020_jit.fits   u59g0108j.jif
n6296q020.jit        o5el08020_jwf.fits   u59g0108j.jit
n6296q020_asn.fits   o5el08020_jwt.fits

Outputs

No real output, but runs catalog and housekeeping software for the input data. Dumps messages to the screen which could be redirected into a file.

Verbosity

For verbose messaging, especially for debugging, use the standard OPUS environment variable MSG_REPORT_LEVEL. There is no more -v option. So to turn messaging up for a run of the command-line tools, set the MSG_REPORT_LEVEL on the command line before you run the tools. This will only affect processes run in the same window thereafter, not the entire account you are running in.
E.g. setenv MSG_REPORT_LEVEL MSG_ALL

Note that MSG_REPORT_LEVEL and STDB_REPORT_LEVEL settings are changed right before calls to genreq, InteractiveIngResponse, and update_db_tool. In the case of genreq and ingrsp, this was done to make debugging these comand-line tools easier. It was thought we did not need to debug those processes since they were not changing. However, we may wish to reconsider this change in the future.

In the case of update_db_tool, no extra reporting is necessary as this tool already does a wonderful job of making itself clear. Any extra reporting is redundant and makes it more difficult to see what the tool is doing.

See_also

Examples

#:223:sherbert@tufnel> catalog_hst_cal_oms.pl 
2010280143338-E-ERROR-Too few arguments
Usage: 
>catalog_hst_cal_oms.pl -i <input_dir> -c <archive_class>  

   The order of [-flag value] pairs does not matter.  
   Tool now responds to MSG_REPORT_LEVEL settings*

   <input_dir>     is the location of the data to be archived.  
                   Only data from one type of archive_class should be here.
                   This is the optional parameter.  Defaults to current dir,
                   if omitted.

   <archive_class> is either CAL or OMS.

   * MSG_REPORT_LEVEL and STDB_REPORT_LEVEL settings are turned off before 
     calls to ingrsp and update_db_tool (housekeeping).


#:224:sherbert@tufnel> catalog_hst_cal_oms.pl -i oms -c OMS
2010280143442-I-INFO-Starting... catalog_hst_cal_oms.pl -i oms -c OMS
2010280143442-I-INFO-get_regr_path-Obtained null path name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/null.path
2010280143442-I-INFO-get_regr_path-Found INGEST_PATH_NAME: regr
2010280143442-I-INFO-check_pipeline-Pipeline test: osf_test -p regr -pr dataset
2010280143445-I-INFO-check_pipeline-regr is idle.
2010280143445-I-INFO-catalog_hst_cal_oms.pl-chdir'd to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/oms 
2010280143445-I-INFO-get_cal_oms_datasets-Adding j8tlgfkmj to dataset list.
2010280143445-I-INFO-catalog_datasets-Processing with task catalog and resource cat_fgs.
2010280143445-I-INFO-catalog_datasets-Starting catalog processing for dataset j8tlgfkmj.
2010280143445-I-INFO-catalog_datasets-Using command: catalog -r cat_fgs j8tlgfkmj.fgs  2>&1 
2010280143449-I-INFO Starting CATALOG processing. (1)
2010280143449-I-INFO Registered callbacks. (1)
2010280143449-I-INFO Processing fgs file for j8tlgfkmj (1)
2010280143449-I-INFO CATALOG Process is being initialized (1)
2010280143449-I-INFO No SEPARATE_MEMBERS value found in resource file. Defaults to: 1 (1)
2010280143449-I-INFO No ASSOC_IS_MEMBER value found in resource file. Defaults to: 1 (1)
2010280143449-I-INFO Loaded keyword rules (1)
2010280143450-I-INFO process_dataset ----- CATALOG Starting: j8tlgfkmj  : Data_id: fgs --- (1)
2010280143450-I-INFO process_dataset processing j8tlgfkmj: data id: fgs (1)
2010280143450-I-INFO Loading keywords from .//j8tlgfkmj.jif fits_id=0 (1)
2010280143450-I-INFO Loaded up to group 1 from .//j8tlgfkmj.jif (1)
2010280143450-I-INFO Loading keywords from .//j8tlgfkmj.jit fits_id=0 (1)
2010280143451-I-INFO Loaded up to group 1 from .//j8tlgfkmj.jit (1)
2010280143451-I-INFO Inserting j8tlgfkmj group 0 into oms_data (1)
2010280143451-I-INFO Inserting j8tlgfkmj group 0 into oms_summary (1)
2010280143451-I-INFO process_dataset ----- CATALOG Ending Observation: j8tlgfkmj ----- (1)
2010280143451-I-INFO-catalog_datasets-Completed catalog processing for dataset j8tlgfkmj.
2010280143451-I-INFO end of process_file_event (1)
2010280143451-I-INFO Halt event: INTERACTIVE registered (1)
2010280143451-I-INFO-Processing with data_id fgs with resource hkp_fgs.
2010280143451-I-INFO-hkp_datasets-Starting housekeeping data update for dataset j8tlgfkmj. 
2010280143451-I-INFO-Using command: update_db_tool -r hkp_fgs -t fgs j8tlgfkmj 2>&1
2010280143454-I-INFO Starting processing. (1)
2010280143454-I-INFO Initialized OPUS environment. (1)
2010280143454-I-INFO Loading queries from /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs//hkp_hst_query.lis (1)
2010280143454-I-INFO ***** loading queries from /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs//hkp_hst_query.lis (1)
2010280143454-I-INFO ***** query loading completed. (1)
2010280143454-I-INFO Registered callbacks. (1)
2010280143454-I-INFO ***** begin update_db_tool for j8tlgfkmj data_id fgs ***** (1)
2010280143454-I-INFO Trying query named UPDATE_ADS_WITH_OMS (1)
2010280143454-I-INFO Title: Update ADS table with oms_data fields (1)
2010280143454-I-INFO 0 records updated for 
update dadsauto1..archive_data_set_all set
      ads_instrument = 'ACS',
      ads_program_id = oms_program_id,
      ads_obset_id = oms_obset_id,
      ads_obsnum = oms_obsnum,
      ads_start_time = oms_startime,
      ads_end_time= oms_endtime,
      ads_pep_id = oms_proposid
from dadsauto1..oms_data
where oms_data_set_name = 'J8TLGFKMJ' and
      ads_data_set_name = 'J8TLGFKMJ' and
      ads_archive_class='OMS' and 
      ads_best_version='Y'
2010280143454-F-INFO An exception of type Bad_val has occurred.
Zero records are not allowed for this query (1)
2010280143454-E-ERROR-hkp_datasets-update_db_tool failed for dataset j8tlgfkmj. at /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/override//bin/sparc_solaris/ingest_tools_pkg.pl line 838.

Note: this failure is because this is an example for demonstration purposes only. I did not actually ingest the data in an OPUS reingest pipeline before running the catalog tool. This is a good cautionary tale to remind the user of the necessity of running the reingest pipeline between the ingest and catalog tools.

Last Updated: OPUS 2010.4