ingest_hst_non_cal.pl

Usage

>ingest_hst_non_cal.pl -d <data_id> -o <data_source> [-i <input_dir>]

Context

Inputs

-d {data_id}

The <data_id> is the OSF data_id used in normal processing. It is generally not equal to the archive class. If it is set to an invalid value, such as "xxx", then a list of valid values with the name of the related archive class is printed. This tool supports all HST data_ids except for classes CAL and OMS that have associations.

-o {source}

This is the four-character source of the ingest override value used as the value for the ads_data_source field in the DADS catalog archive_data_set_all table. This is a way for Ops to distinguish between test data, normally processed pipeline data, and reingested data. E.g. REGR, TEST, RING, DLSG, etc.

-i {input_dir}

This is the optional location of the data. If not provided, current working directory is assumed.

Data

OPUS 2010.4 changes should allow the data to be ingested in either the name that comes out of the originating OPUS pipeline, or the one that DADS delivers. The data format matters. In other words, SMS data can be in either yd5a11550.pod format, or yt5813360_pod.fits format. PDQ data that is a text file called o6n901fbq.pdq should work, and PDQ data that has already been converted to fits and is called o6n901fbq_pdq.fits will also work. OMS FITS data called o6n901fbq.jif will work just as well as OMS FITS data called o6n901fbq_jif.fits.

Outputs

No real output, but sets files up for the reingest pipeline. Dumps messages to the screen which could be redirected into a file.

Verbosity

For verbose messaging, especially for debugging, use the standard OPUS environment variable MSG_REPORT_LEVEL. There is no more -v option. So to turn messaging up for a run of the command-line tools, set the MSG_REPORT_LEVEL on the command line before you run the tools. This will only affect processes run in the same window thereafter, not the entire account you are running in.
E.g. setenv MSG_REPORT_LEVEL MSG_ALL

Note that MSG_REPORT_LEVEL and STDB_REPORT_LEVEL settings are changed right before calls to genreq, InteractiveIngResponse, and update_db_tool. In the case of genreq and ingrsp, this was done to make debugging these comand-line tools easier. It was thought we did not need to debug those processes since they were not changing. However, we may wish to reconsider this change in the future.

In the case of update_db_tool, no extra reporting is necessary as this tool already does a wonderful job of making itself clear. Any extra reporting is redundant and makes it more difficult to see what the tool is doing.

See_also

• catalog_hst_non_cal.pl
• catalog_hst_cal_oms.pl
• ingest_hst_cal_oms.pl
• ingest_dads_logs.pl
• Reingest_Tools.html
• PR 51848(original PR)

Examples

#:142:sherbert@tufnel> printenv MSG_REPORT_LEVEL
MSG_ALL
#:143:sherbert@tufnel> printenv STDB_REPORT_LEVEL
STDB_INFO

#:141:sherbert@tufnel> ingest_hst_non_cal.pl -i dmp -d XXX 
2010280131919-D-DIAG-Running /project/tufnel/dsb/lisa/bin/sparc_solaris/ingest_hst_non_cal.pl 
2010280131919-D-DIAG-ingest_hst_non_cal.pl-number of arguments is 4.  We fail if < 4. 
2010280131919-D-DIAG-open_opus_db-Opening connection to ROBBIE opusdev2 
2010280131920-D-DIAG-check_hst_data_id------subBeg----- 
2010280131920-D-DIAG-report_valid_ids------subBeg----- 
2010280131920-D-DIAG-report_valid_ids-query:
SELECT acm_data_id, acm_archive_class
FROM archive_class_map
WHERE acm_data_id=lower(acm_data_id) and acm_archive_class NOT IN ('CAL','OMS','FUS','FUR') 


The following data_id values may be used with this tool.
acm for archive class ACM
adm for archive class DIA
anc for archive class ANC
ast for archive class AST
cdb for archive class CDB
cdm for archive class DIA
csi for archive class CSI
ctb for archive class CTB
dia for archive class DIA
dlg for archive class DLG
dmp for archive class DMP
edi for archive class EDT
edj for archive class EDT
edl for archive class EDT
edn for archive class EDT
edo for archive class EDT
edt for archive class EDT
edu for archive class EDT
edx for archive class EDT
epc for archive class EPC
msc for archive class MSC
mtl for archive class MTL
ndm for archive class DIA
orb for archive class ORB
pdq for archive class PDQ
pod for archive class POD
prb for archive class PRB
rpt for archive class RPT
sdm for archive class DIA
sms for archive class SMS
ssi for archive class SSI
wdm for archive class DIA
2010280131920-E-ERROR-check_hst_data_id-The data_id xxx is not found in archive_class_map. at /project/tufnel/dsb/lisa/bin/sparc_solaris/ingest_tools_pkg.pl line 117.

#:144:sherbert@tufnel> unsetenv MSG_REPORT_LEVEL
#:145:sherbert@tufnel> unsetenv STDB_REPORT_LEVEL

#:147:sherbert@tufnel> ingest_hst_non_cal.pl -i dmp -d dmp -o LTST >& keepMe.out

#:148:sherbert@tufenl> cat keepMe.out
2010280134602-I-INFO-Starting... ingest_hst_non_cal.pl -i dmp -d dmp -o LTST
2010280134602-I-INFO-get_regr_path-Obtained null path name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/null.path
2010280134602-I-INFO-get_regr_path-Found INGEST_PATH_NAME: regr
2010280134602-I-INFO-check_pipeline-Pipeline test: osf_test -p regr -pr dataset
2010280134622-I-INFO-check_pipeline-regr is idle.
2010280134622-I-INFO-ingest_hst_non_cal.pl-chdir'd to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/dmp 

2010280134622-I-INFO-get_datasets-Adding gtja92336 to dataset list.
2010280134622-I-INFO-get_path_dirs-Obtained path file name: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/defs/regr.path
2010280134622-I-INFO-get_path_dirs-Found INGEST_TRIGGER_ROOT: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/
2010280134622-I-INFO-get_path_dirs-Found INGEST_SOURCE_DIR: /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/
2010280134622-I-INFO-set_data_source-No old files found in /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/.
2010280134622-I-INFO-set_data_source-Created /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/source/source_override.ltst
2010280134622-I-INFO-request_datasets-Processing with data_id Dmp with resource genreq.
2010280134622-I-INFO-request_datasets-Using trigger directory /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/hst/.
2010280134622-I-INFO-request_datasets-Processing dataset gtja92336.
2010280134625-I-INFO Initialized OPUS environment. (1)
2010280134625-I-INFO Registered callbacks. (1)
2010280134625-I-INFO GENREQ----------------------------------- (1)
2010280134625-I-INFO Spawning command: osf_present -p regr.path -f gtja92336 -t Dmp (1)
2010280134628-I-INFO Processing type is GENERIC (1)
2010280134628-I-INFO 6 files are collected for archiving. (1)
2010280134628-I-INFO Copy .//gtja92336.bdx to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//dmp/gtja92336.bdx (1)
2010280134628-I-INFO Copy .//gtja92336.eix to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//dmp/gtja92336.eix (1)
2010280134628-I-INFO Copy .//gtja92336.pkx to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//dmp/gtja92336.pkx (1)
2010280134628-I-INFO Copy .//gtja92336.shx to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//dmp/gtja92336.shx (1)
2010280134629-I-INFO Copy .//gtja92336.trx to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//dmp/gtja92336.trx (1)
2010280134629-I-INFO Copy .//gtja92336.ulx to /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr//dmp/gtja92336.ulx (1)
2010280134629-I-INFO Archive files collected successfully. (1)
2010280134629-I-INFO-request_datasets-Dataset gtja92336 successfully processed by genreq.
2010280134629-I-INFO-request_datasets-Created /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/pipe/cache/ingest/regr/trigger/hst/gtja92336.Dmp.
2010280134629-I-INFO-Completed...ingest_hst_non_cal.pl -i /info/tufnel/lisa/test_sherbert_20100902_1/regr_00202/dmp -d dmp -o LTST