GREEN_ENTRIES - come from the process resource
file for a particular pipeline process
ORANGE_ENTRIES - come from entries in the OTFR
request and response files
TAN_ENTRIES - come from the OPUS_DEFINITIONS_DIR:opus_login.csh
login script
RED_ENTRIES - come from the OPUS blackboard system
Hopefully the use of these different colors will make the source of each parameter more clear.
The OTFR system consists of a single OTFR pipeline, which in the middle bridges off to an HST science pipeline (see figure 1). The OTFR pipeline consists of a set of stages (processes) that will accept OTFR requests from DADS and return OTFR responses. An OTFR request is a simple text file which includes the name of the dataset that is to be re-processed, and the directory location in DADS to which the resulting output products produced by OTFR will be returned. An OTFR response consists of a response file (again a simple text file, almost exactly matching the request file except for a status return message) along with calibrated and raw datasets and products, produced by the stages of the OTFR and HST science pipelines.
Each stage in the OTFR and HST science pipelines performs a part of the processing necessary to re-process observations. The processing is split up to allow multiple copies of pipeline stages to be running. This can improve the throughput of processing by keeping the CPU and I/O systems loaded. It also can improve the robustness in the event of hardware problems, by having different copies of processes running on different machines. If one machine goes down with a hardware failure, it is possible that other machines in the cluster could continue to process data.
Currently only STIS and WFPC-2 data will be processed in the OTFR system. Eventually other instruments will be added (NICMOS, ACS, etc).
The OTFR machines will need to put and get files to and from the existing DADS VMS-based distribution workstations. File transfers via FTP will occur between the VMS and UNIX workstations in several different pipeline stages, so network connectivity disruptions will likely cause problems in the pipeline processing.
Currently all pipeline stages are implemented as either UNIX shell scripts or Perl scripts. These scripts use a combination of system commands, Perl utilities, OPUS programs and STSDAS programs to perform the necessary processing, so software installations and updates to any of these software systems can cause changes in the pipeline behavior.
Database access to both OPUS and DADS databases occurs in many locations
in the OTFR and HST science pipelines. The particular databases and
tables are described in the details that follow. Disruptions in database
connectivity will obviously cause problems in the pipeline processing.
The characters "xxx" above imply the stage will have an instrument specific
OPUS
process resource file , i.e. POLWF2, POLNIC, POLSTI, etc., which supplies
instrument specific parameters and directory names to the process being
run. This allows a single piece of code to run using different input parameters.
Summary:
Awakens periodically (DELTA_TIME) to check the OTFR request directory (REMOTE_REQUEST_DIR) on a disk local to a DADS VMS machine (RMT_MACHINE) for OTFR request files (REQUEST_FILEMASK). Any OTFR request files found will be copied back to a local directory (OUTPATH_HOLD) on the OTFR UNIX machine via a FTP process (RMT_LOGIN, FTP_PFILE). Once this copy is verified as successful, the OTFR request files will be deleted from the OTFR request directory on the DADS VMS machine, so that they are not picked up multiple times. Then the request files are renamed from the local incoming directory (OUTPATH_HOLD) to the local polling directory (OUTPATH) where they can be picked up by the next pipeline process.This process serves OTFR request files to the OTFR UNIX system, by obtaining them from the DADS VMS system. Failures in this process will stop OTFR requests from entering the OTFR pipeline.
DADS will place OTFR request files in the OTFR request directory if the OTFR system has signalled that it is "online".
There will only be one copy of this process running. This
should be enforced by a restrictions
file in the OPUS PMG. This ensures that two separate GETREQ processes
do not conflict over copying and deleting requests in the OTFR request
directory.
Summary
Checks for OTFR request files (FILE_OBJECT1) in the local request directory (FILE_DIRECTORY1) filled by GETREQ. For each request found, creates a directory under the instrument calibration directory (OUTPATH) in which the particular request will be worked, and copies the OTFR request there. The request will have a new file extension addition (FILE_PROCESSING). Reads the OTFR request and creates Observation Status Files (OSFs) for each DATASET entry in the request (current plan is only one DATASET entry per request). The OSFs will be used to control and track the processing of the request while it is in the OTFR pipeline. A trailer (log) file is started that will document the processing steps performed by OTFR on this request.The OTFR Request File will be named with a unique pseudo-TIMESTAMP and a DATASET_NAME, e.g. 3112234912345_u2440101t.req . Since the DATASET_NAME could appear in more than one request, the TIMESTAMP must be guaranteed by DADS (who will be naming these files) to be unique for every OTFR request sent to the pipeline. The TIMESTAMP consists of several fields: DDHHMMSSnnnnn, where DD is the day of the month, HHMMSS are the hour, minute, and second at which the request was generated, and nnnnn is a unique number assigned by DADS that prevents duplicate timestamps. The OTFR request file will contain, at least,
When an OTFR request file is found in the polling directory (FILE_DIRECTORY1), a subdirectory named with the values of the TIMESTAMP_DATASET will be created under the calibration directory (OUTPATH) for this instrument. By grouping the subdirectories for each instrument under an instrument-specific directory, this allows segregation and disk management of processing by instrument.DATASET_NAME=U2440101T FILE_COUNT=0 TIMESTAMP=3112234912345 DIRECTORY=DISK$LEO:[LEO.MSWAM.OTFCAL.DADS] END_FILE
If the subdirectory already exists, then this will be considered a duplicate request, and an OSF (Observation Status File) will be created for the request indicating the duplication ('d') in the PO column. The duplication OSF will also be given a special class (CLS = req) and a special DCF number (DCF = 999), so that these OSFs can be sorted together in the OMG. These entries on the OMG will hopefully catch the operator's attention. A message will also appear in the POLxxx process log file explaining the error. These duplications are likely only to be seen in the test environment, where the same OTFR requests can be run repeatedly using some tricks. DADS should not normally feed duplicate requests to the OTFR system.
The OTFR request file will be copied from the polling directory into this request-unique subdirectory. It is given a file extension addition (FILE_PROCESSING) at this time as an artifact of the OPUS architecture. This copy of the request will provide later pipeline stages access to needed information. For example, the DIRECTORY field in the request will need to be accessed in order to send the OTFR products back to DADS. The copy of the request in the local polling directory will be deleted automatically by the OPUS architecture to avoid processing the request multiple times.
The contents of the OTFR request are echoed to the process log file and the request trailer file, so that later searching of these log files can be used in request debugging. The request trailer file will be named TIMESTAMP_DATASET.trl, and will be created in the unique request-specific subdirectory. During processing, the OMG/ViewTrailer function should be able to display the contents of this file to indicate the steps performed so far in processing this request.
POLxxx will use the information in the OTFR request filename and the file contents to create OSFs on the OTFR blackboard. Here is an example OSF:
The most relevant details here are the dataset identifier made up of the timestamp and datasetname (31512014412345_u2440101t) and the OSF status field (cw____________________). These indicate to the OPUS architecture that a particular dataset is waiting (w) for a particular kind of processing. Since the 'w' is in the second pipeline stage, the LS (LSTxxx) process will be the next process to peform work on this request. The OSF status field is filled from the OSF_START_STATUS resource entry. The data-id field of the OSF (in this case set to 'wf2') is filled from the POL_DATATYPE resource entry.33C243B1-cw_____________________.31512014412345_u2440101t-wf2-000-____
If a request does not contain a DATASET_NAME entry, then the request is considered corrupted, is renamed with a special file extension addition (FILE_ERROR), has an OSF created for the "bad" request (PO = 'b') to bring the error to the operator's attention, and will remain in the polling directory (FILE_DIRECTORY1) until further action is taken.
If OSF creation fails for a DATASET_NAME entry in the OTFR request, then the OTFR request will be renamed with a special file extension addition (FILE_ERROR) and this process will go ABSENT, indicating a problem to the OTFR operator in the PMG. This error is considered severe, because it likely indicates a problem in the OPUS architecture and would probably affect the processing of all datasets, so that is why the program is brought down.
A separate POLxxx OPUS resource file will exist for each instrument so that path-dependent locations for the instrument calibration directory can be used. This implies that a separate file polling process will need to be started in the OTFR pipeline for each instrument, though a single polling directory can be used to hold all requests if the file-polling mask is created with instrument-specific characters in it to differentiate between the instruments.
Once all OSFs are created, an entry is written to the ARCH_SERVER::ARCH_DB:otfr_request_stats database relation for keeping statistics on the key processing times for this OTFR request. Later pipeline tasks will update this database record with additional processing times, so that after the request has completed OTFR (successfully or not) this record will document how long processsing took, and whether or not processing problems were encountered.
Summary
Creates a file listing exposure names and POD file names needed to process this DATASET, including any member names if this is an association, or any special files (e.g. NIC SAA darks). If EDT files should be used for processing instead of POD files, an indication of this will appear in the generated file.This process begins by testing DATASET (from the triggering OSF) to see if it is an association. This is performed by checking the last character of the name, and if it is a digit [0-9] or letters [a-i], and this instrument supports associations, then it is assumed to be an association. The dataset name is then converted to the name of the primary product by changing the last character to zero (0), and that name is searched for in the DSQUERY::OPUS_DB:asn_members database relation. Any collected members found for the association will be saved in a list of exposures. If the association has no collected members, this is considered an error, and the OSF is marked for failure.
An additional check is made for NICMOS datasets. If the dataset name appears in the DSQUERY::OPUS_DB:nic_saa_link database relation, then this dataset has a corresponding SAA dark association. The members of that association are then obtained by another query to DSQUERY::OPUS_DB:asn_members, and those names are added to the exposures list. If no collected members are found for the NIC SAA dark association, again the OSF is marked for error.
If the DATASET requested was not an association, then the name of that dataset is the only entry in the exposures list.
A file named like the OTFR request, but given a .podlist extension. (e.g. 1412020112345_n12345678.podlist) is created in the request-specific subdirectory to hold the data sources required for all exposures needed to re-process this DATASET. This file will indicate how to generate archive requests to retrieve the needed files for re-processing. There are currently two choices for the source of the exposure data: POD or EDT. Normally POD will be used, but there are some datasets for which valid POD files do not exist in the archive, and for these the archived EDT set will be used for re-processing.
Now the data source for each entry in the exposures list must be determined. A check is made of the ARCH_SERVER::ARCH_DB:otfr_special_processing relation, and if the exposure name appears there with osp_data_source = EDT, then this exposure is not supposed to be re-processed from POD files, but from archived EDT files instead. Usually this indicates there was a problem with the original POD files, and repaired EDT files were created and archived instead. In this case an EDT entry is made in the .podlist file for this exposure.
If no special processing records are found, then the exposure is searched for in the DSQUERY::OPUS_DB:podnames relation. This search is performed for the first 8 characters of the dataset name, with the last character wild-carded. This is done because the DATASET listed in the archive is not necessarily the same exact name found in the podnames relation. The last character of the dataset name can differ depending on how many times the dataset was sent from DDF/PACOR to STScI. All POD names found by this query are returned, and if different last characters in any exposure names were found a second query is made. This one goes to the DSQUERY::OPUS_DB:executed table to see what the prefered version of the exposure is. This name and the corresponding POD files are written to the .podlist file.
The .podlist file should now contain entries describing the names and data sources of all of the "raw material" files (PODs and EDT sets) needed to re-process DATASET.
Summary:
This stage will perform an archive retrieval for each entry in the .podlist file. The files retrieved will be written back to the request-specific subdirectory.This process will convert the list of POD and EDT entries in the .podlist file into a set of archive retrieval commands for retrieving the data files needed for re-processing. For this sample entry in a .podlist file
Each archive retrieval command contains the root name to search the archive for, the archive class of the files, and the destination directory for the retrieved files. If any EDT entries appear in the .podlist file, similar commands are used, substituting the EDT dataset name, the EDT archive class, and the same destination directory. Before beginning any retrieval commands, a number of set-up steps are performed.
Files retrieved from the archive using direct acquire come back with this format:
The extra _030101190449_pod on the filename is the GMT timestamp and the archive class. These extra items will be removed before the file is passed on to the next pipeline stage.
If any of the archive retrieval commands for any of the .podlist entries fails, the OSF status will be set to 'e'.
Summary:
This stage will perform "pod-whacking", or segmentation of POD files into one exposure per POD file, for each POD entry in the .podlist file.NOTE: This segmentation is only needed for instruments where a POD file can contain data for more than one exposure (e.g. STIS, NICMOS). It does NOT apply to instruments that have a one-to-one mapping from PODs to exposures (e.g. WFPC-2). For instruments that don't need pod-whacking, this stage does a simple copy of the archive-retrieved POD file to the POD file name required for submission to the science pipeline. A copy is done so that the file ownership is changed from the archive (which placed the files there) to the OTFR pipeline operator.
This process will read the contents of the .podlist file, and for every POD entry found it will run the podwhacker OPUS utility to extract exposure data into one POD file per exposure. The newly created POD file will be named exposure_podname.pod, and will be written to the OTFR request-specific directory. Any EDT entries in the .podlist file will be skipped over.
Pod segmentation is needed for several reasons
Summary:
This stage will reserve exposure and association names on a separate blackboard to guarantee that no collisions will occur in the HST science pipeline between OTFR requests that happen to be processing the same files.This task will first look for an already-existing reservation list file (.rsvlist) in the request-specific subdirectory. If found, this is an indication that a previous reservation attempt failed and this attempt is a re-try. The list of exposure and association names in the reservation list file is read in. If no such file is found, then one is created by reading all of the exposure names in the .podlist file, and performing a database look-up against DSQUERY::OPUS_DB:asn_members to find all association_id values listed for each exposure name in the .podlist file. The reservation list file is formed by combining the list of .podlist exposure names and any association ids found in the database.
Once the reservation list is either read in or created new, an attempt is made to create an OSF on a separate blackboard (RESERVATION_PATH) for each reservation list entry. If every OSF creation attempt succeeds, then this OTFR request can proceed, since the required names that will be used in the science pipeline have been reserved. If any one of the OSF creation attempts fails for either an exposure or association name, then a collision has occurred. This indicates that another OTFR request has already reserved one or more of these names for science processing, and that the current OTFR request will have to try again later to see if the existing reservations have been cleared. If any of the OSF creations happened to succeed for the current OTFR request before the failed creation occurred, the OSFs that were successfully created are deleted, so that the reservations needed for this request are not "partially" made.
The reservation OSFs will be deleted by the CLNxxx process once the OTFR request is finished processing.
Summary:
This stage will attempt to move files for each entry in the .podlist file from the OTFR request-specific directory to the appropriate directory of the HST science pipeline.This process opens and reads the .podlist file, and attempts to move files for each .podlist entry from the OTFR request-specific directory (where the files were placed by an archive retrieval), to one of the input directories for the HST science pipeline. Once moved, these exposures will be ready to begin science processing.
If a .podlist file entry specifies an EDT data source, then the entire set of files matching the EDT name (EDT_EXT_MASK) will be moved from the OTFR request-specific directory to the appropriate science pipeline directory (EDT_RENAME_DIRECTORY). Once the EDT files are successfully moved, an OSF is created in the science pipeline path (DATA_REDUCTION_PATH, OSF_DATA_TYPE, OSF_START_STATUS) that will cause science processing to begin for these data.
If a .podlist file entry specifies a POD data source, then the corresponding "whacked" POD file will be moved from the OTFR request-specific directory to the appropriate science pipeline directory (POD_RENAME_DIRECTORY). This move is slightly different from the EDT file move, in that POD files can be immediately picked up by the science pipeline for processing. Because of this, the POD file is first moved to the destination directory with an altered file extension to keep it from getting picked up right away, while its contents may not be complete. Once the file has been completely and successfully moved, it is renamed in place to remove the altered file extension, which makes it visible to the science pipeline.
One other change is made for old-style WFPC-2 POD files. These can be named starting with "w". These old pod file names do not follow the convention expected by the current science pipeline software. If one of these old filenames is detected, the file is renamed to prepend it with "lz_a2_", which will allow the current science pipeline software to process it.
For EDT and POD, the file move is performed so that it will fail if files with identical names are already found in the science pipeline area. This is by design so that two OTFR requests that happen to want the same files processed by the science pipeline do not interfere with each other. This type of interference is expected to be rare, except for the case of shared STIS wavecals. For POD files this is accomplished by setting the file permissions to owner:read-only, and using a special form of the move command (mv -i) that will fail when run in batch mode with already existing destination files. For EDT sets, the file permissions are not altered, but the special move command is still used. The end result of one of these collisions is that one of the requests will be unable to move files into the science pipeline right away, but the OSF for this failed move will be set so that the move is re-tried at a later time (see PODTRG). Once the first set of files clears the science pipeline, the second set of files with the same name will be allowed in.
Once all file moves are successful, the ARCH_SERVER::ARCH_DB:otfr_request_stats database relation is updated with a timestamp indicating when the datasets for this OTFR request entered the science pipeline.
Summary:
The OTFR design includes several planned "sleep" stages during pipeline processing, when a request cannot proceed until some other event has occurred. Two specific cases are:
This process takes care of identifying OSFs on the OTFR blackboard
that are "sleeping" and retriggers them according to the resource file
selections (SEARCH_OSF, SET_OSF). This is a generic process
that can handle new retriggering behavior by just making additions to the
process resource file.
Summary:
This stage attempts to collect products from the science pipeline in order to satisfy an OTFR request. When the needed products are ready in the science pipeline, they are copied back to the OTFR request-specific directory. A timeout scheme is implemented so that a collector will not wait indefinitely for science pipeline products to be ready before closing-out an OTFR requestThere are two styles of collectors: one for the multi-file association instruments (NICMOS, ACS), and one for the others (WFPC-2, STIS, etc.). The multi-file association collector has the added complication of collecting science output for all association members and the requested product. The .podlist file contains the names of the exposures that drive the multi-file association collection. The triggering OSF supplies the name of the DATASET for the other type of collector.
Both collectors begin by checking for the presence of a timeout file. This file will appear in the OTFR request-specific subdirectory and be named TIMESTAMP_DATASET.timeout. It is created the first time the collector starts working on a particular OTFR request, and is used to limit the amount of time a collector will wait for needed output from the science pipeline. The age of this empty file is checked each time the collector is retriggered, and if the age of the file is greater than TIMEOUT_INTERVAL, then the collector picks up copies of any needed science output that is ready, and proceeds to flush this OTFR request from the pipeline. Any science output that is not ready by this time is considered lost.
If timeout has not occurred, then the status of the calibration stage in the science pipeline path (DATA_REDUCTION_PATH) is checked for each needed science output by testing the OSF. Only the DATASET is tested for WFPC-2 and STIS. All association members and the primary product are tested for NICMOS and ACS (OSFs are not created in the science pipeline for sub-products). If the status value(s) for calibration show COMPLETE (c), NOT PERFORMED (n), or FLUSHED (f), then science processing is considered done, and all needed science output files are copied back from the science pipeline directory (CAL_DIR) to the OTFR request-specific directory. The list of needed science output files is obtained from the .podlist contents and the requested DATASET.
The ARCH_SERVER::ARCH_DB:otfr_request_stats database relation is updated with the timestamp when collection completed, regardless of whether normal completion or a timeout occurred.
Summary:
This stage performs a GEIS-to-FITS conversion for WF2 and other pre-SM97 instruments, converting all files that appear to be GEIS or ASCII files to FITS format. This stage will also be run for post-SM97 instruments (STIS, NICMOS, ACS) to perform conversion of text files (trailer file, etc.) to FITS format.
Any calibration trailer files (*.tra) returned from the science pipeline processing are renamed to *.trl before being converted to FITS. This is because stwfits recognizes certain file extensions, and .tra is not one of them.
The construction of the FITS filename follows these rules for all files matching the filemask ?????????.??? (this leaves out the DATASET_???.FITS files created for STIS, NICMOS, and ACS) :
1.If the filename ends in ".??d"
then the file is assumed to be a GEIS data file and is skipped. It is assumed
that this file will be converted when the
corresponding GEIS header
file is converted.
2.If filename ends in ".??h"
then the file is assumed to be a GEIS header file. The ending "h" in the
file extension will be converted to "f" and the FITSNAME will be "DATASET_??f.fits"
The stwfits tasks requires use of a format file.
A specific one in the IRAF software tree is referenced in terms of the
environment variable TABLESDISK
and a
relative directory name in the FORMAT_DIR
process resource parameter.
OPUS Resource File Entries
Summary:
This stage will be used to copy back FITS datasets to a directory on the DADS staging disks. All *.fits files in the working directory for this request will be copied back.A FTP copy (RMT_MACHINE, RMT_MACHINE_TYPE, RMT_LOGIN, FTP_PFILE) of all FITS files in the OTFR request-specific directory will be made back to the DIRECTORY specified in the original OTFR request. Any files already existing there will be overwritten with new versions. The OTFR pipeline will delete the data (in a later pipeline stage, CLNxxx) from its disks without waiting for any kind of response from DADS, once a verified copy exists on the DADS disk.
Summary:
This stage will be used to create an OTFR response file and copy it back to the DADS system via FTP.The response file will be formed by renaming and editing the local copy of the OTFR request (REQUEST_FILEEXT). The filename extension of the OTFR request will be changed (RESPONSE_FILEEXT). The FILE_COUNT field in the response file will be updated to include all FITS files in the working directory.
The OSF for DATASET (or the primary product name, if DATASET is an association product) is searched for in the science pipeline path using the OPUS tool osf_test, to return the status column of the calibration stage (CA) of the pipeline. This indicates either success, failure, or non-action. If this status column = 'n' or 'f', then calibration was bypassed, and a message indicating this (RESPONSE_STATUS_NOCAL) will be placed in the STATUS entry in the OTFR response. If calibration occurred normally, a different status (RESPONSE_STATUS_OK) message will appear in the OTFR response.
The OTFR response will be copied, via FTP copy (RMT_MACHINE, RMT_MACHINE_TYPE, RMT_LOGIN, FTP_PFILE), back to a response directory (RESPONSE_DIRECTORY)on the DADS system.
The ARCH_SERVER::ARCH_DB:otfr_request_stats database table will be updated with the completion timestamp for this OTFR request.
Summary:
This process cleans up a completed OTFR request by deleting any .podlist-specific science output in the science pipeline, any processing files left in the request-specific TIMESTAMP_DATASET subdirectory, the reservation OSFs on the reservation blackboard, and removing the subdirectory itself if no files are left in it.
The following processes can run in both the OTFR science pipeline and
the pre-archive science pipeline. They are either new with the introduction
of OTFR, or have different behavior now that OTFR is being implemented:
The operator can use the facilities of the PMG to halt all OTFR processes, and the facilities of the OMG to delete all Observation Status Files for the currently processing requests in the OTFR pipelines. Note that each of these paths and pipelines must be halted and cleared using the OMG and PMG.
clear_otfr.sh otfr_pathname otfsci_pathname reserve_pathname
where otfr_pathname is the name of the path in which the OTFR processes are running (e.g. "otfr"), otfsci_pathname is the name of the path in which the OTFR science pipeline processes are running (e.g. otfsci), and reserve_pathname is the name of the path where science reservation OSFs are created. The script will search the path file for the locations of the processing directories (e.g. wf2_recalib, stis_recalib, etc.), and check for OTFR files in those directories. If files are found, a prompt is issued to the operator to be sure that they want to delete the files. The incoming request directories on the OTFR side (e.g. local_otfr_request_incoming, local_otfr_request_dir) are also searched. A separate deletion prompt is issued for each location where files are found.
The OTFR science pipeline will be cleared by searching the OTFR science path for entries matching DUDDC?, DUDDS?, and POD_ARCH_*. These are the typical locations of science files and POD files. Again, prompts are issued for any files found and the operator must confirm before each set of deletions will occur.
Additional prompts are then issued for cleaning out any OSFs remaining in the otfr, otfsci, and reserve blackboards. These could have been cleared using the OMG, as mentioned above, but these extra deletion checks are made in case any OSFs remain.
Process resource files are usually set up to "pull" certain parameter values "through a path file". This means that in the process resource file, instead of directly providing a value for a parameter, the value is given as the name of an entry in a path file. By using different path files, you can change the values of certain items used by the processes in a pipeline.
In the OTFR default path file, here are the parameters that are referenced by existing entries in the process resource files:
wf2_recalib - directory under which WFPC-2 processing directories are created
stis_recalib - directory under which STIS processing directories are created
nic_recalib - directory under which NICMOS processing directories are created
acs_recalib - directory under which ACS processing directories are created
pod_dir - science path mnemonic for pod input dir
acs_sis_dir - science path mnemonic link for ACS EDT dir
acs_cal_dir - science path mnemonic link for ACS CAL dir
nic_sis_dir - science path mnemonic link for NICMOS EDT dir
nic_cal_dir - science path mnemonic link for NICMOS CAL dir
stis_sis_dir - science path mnemonic link for STIS EDT dir
stis_cal_dir - science path mnemonic link STIS CAL dir
wf2_sis_dir - science path mnemonic link WFPC2 EDT dir
wf2_cal_dir - science path mnemonic link WFPC2 CAL dir
response_file_ext - file extension for the OTFR response file
request_file_ext - file extension for the OTFR request file, as it comes from DADS
request_work_ext - file extension for the OTFR request file as it progresses through the OTFR pipeline
data_reduction_path - name of the OTFR science pipeline path
reservation_path - name of the path in which exposure and association reservation OSFs are created
error_notify_interval - time in HOURS that a dataset will sit in an error state before DADS is notified
notify_polling_interval - polling time for checking for newly expired notify events
error_flush_interval - time in DAYS that a dataset will sit in an error state before the datatset is returned to DADS, as is
flush_polling_interval - polling time for checking for newly expired flush events
wf2_cal_file_ext = filespec for identifying WFPC-2 calibration products (for deletion during flushing)
stis_cal_file_ext = filespec for identifying STIS calibration products (for deletion during flushing)
nic_cal_file_ext = filespec for identifying NICMOS calibration products (for deletion during flushing)
acs_cal_file_ext = filespec for identifying ACS calibration products (for deletion during flushing)
edt_ext_mask - file extension mask for finding EDT files for a dataset
osf_start_status - OSF status column values for data reduction pipeline when EDT set is used
dads_tool_dir - NFS location of direct acquire tool
DADS_machine - remote DADS machine to use for FTP operations
DADS_machine_type - type of remote DADS machine (VMS or UNIX)
DADS_login - login id to use during FTP session on remote DADS machine
DADS_request_dir - directory on remote DADS machine where OTFR requests are picked up
DADS_response_dir - directory on remote DADS machine where OTFR responses are sent
local_otfr_request_incoming - directory on local OTFR system OTFR request are copied from the DADS system
local_otfr_request_dir - directory on local OTFR system where OTFR requests begin processing
These are the variables defined in the OTFR default opus_login.csh file:
setenv DSQUERY - OPUS database server
setenv OPUS_DB - OPUS database
setenv SPSS_DB - SPSS database (assumed on DSQUERY server)
setenv ARCH_SERVER - DADS database server
setenv ARCH_DB - DADS databasesetenv STSDASDISK - points to the STSDAS software tree
setenv TABLESDISK - points to the TABLES software tree
setenv STLOCAL - points to the STLOCAL software treesetenv nref - location of NICMOS reference files
setenv ntab - location of NICMOS reference tables
setenv oref - location of STIS reference files
setenv otab - location of STIS reference tables
setenv jref - location of ACS reference files
setenv jtab - location of ACS reference tables
setenv uref - location of WFPC-2 reference files
setenv xref - location of FOC reference files
setenv yref - location of FOS reference files
setenv zref - location of HRS reference files
setenv xtab - location of FOC reference tables
setenv ytab - location of FOS reference tables
setenv ztab - location of HRS reference tables
setenv mtab - location of multiple-instrument reference tables
setenv crwfpc2comp - location of WFPC2 component lookup tables
setenv crotacomp - location of WFPC2 component lookup tables
setenv ttab - location of throughput tables
Each stage of the OTFR pipeline has four (4) error
processes monitoring it:
So the progression is
The names of these error-handling processes follow this naming convention.
Note that the timer intervals (ERROR_NOTIFY_INTERVAL, ERROR_FLUSH_INTERVAL) in the process resource files are entered in different units. The ERROR_NOTIFY_INTERVAL is in units of HOURS, while the ERROR_FLUSH_INTERVAL is in units of DAYS. These seemed to be the most likely timescales on which these intervals would be set. Comments in the process resource file serve to remind the user of the units, if you ever want to change the values.
Errors in different OTFR pipeline stages will result in different flushing behavior.
Error in Pipeline stage
Action
-----------------------
------
PO - poll for OTFC_request
Send only OTFR_response back to DADS
LS - copy FITS files from DADS
RQ - submit an archive request
WK - segment a POD file
RV - reserve exposure names for science
MO - move POD/EDT to science pipeline
Errors in early pipeline stages result in only an OTFR_response file being returned to DADS, since no useful work has yet been done. The FILE_COUNT parameter in the response file will be set to zero (0) to indicate that no FITS files have been sent back by the OTFR pipeline.
Error in Pipeline stage
Action
-----------------------
------
CO - update FITS keywords
Send OTFR_response and any FITS files
2F - convert GEIS/ASCII to FITS
that have been created so far back
to DADS.
Errors in intermediate stages result in a response file and any existing FITS files being returned to DADS.
Error in Pipeline stage
Action
-----------------------
------
RE - return FITS files to DADS
No action taken. Error requires
RS - create OTFC_response for DADS
operation intervention.
(i.e. flushing has failed)
If a dataset hits an error in a pipeline stage that cannot be flushed (e.g. RE, the stage that copies files back to DADS), the error will remain in the OTFR pipeline until it is attended to by the operator. These are considered cases in which the flush operation itself cannot be performed because of some problem. No automated action seems possible.
