EDPS FOF Conversion Pipeline Reference Manual
Contents
FOF Pipeline Overview
The need for a FOF Pipeline
The FOF Pipeline is the Pipeline within the EDPS that extracts telemetry parameters of interest
from FOF files received from CCS and packages the parameters into intermediary telemetry files that
are subsequently used by the EDPS FGS and Astrometry pipelines to produce their associated output products.
In addition, the FOF pipeline SAVENG process extracts Save file and Breathing file related telemetry
parameters and packages them into the Save and Breathing files that get copied to the STScI Engineering Team.
Intended Audience and Scope
This Reference Manual is intended primarily to give OPUS Operations Personnel
information needed to understand the operation of the EDPS FOF Conversion Pipeline.
This includes information on the FOF Pipeline Input and Output products, the
directories and files used, and the processes involved. Hopefully, as envisioned, this
manual will be a dynamic document that is updated when FOF Pipeline changes are made.
FOF Pipeline Dependencies
Being one of five pipelines that compose the EDPS, The FOF pipeline is not a stand-alone
pipeline in and of itself. In order for the FOF Pipeline to function, processes
contained in other EDPS pipelines must first execute correctly. In regards to how
they provide for the FOF pipeline, and for the FOF pipeline processing to be
successful, the following EDPS pipelines must execute in the following sequence:
- PASS Data Receipt(PDR) This pipeline receives Science Mission Schedule (SMS)
information from the PASS system and uses the information to populate database relations
that processes in the FOF pipeline rely on. The relations provide information such as whether
or not any observations or guide star acquisitions occur within the time span of a given engineering
telemetry (FOF) file. Processes in the FOF pipeline use this information to determine if they have
work to do; i.e., if they have to extract telemetry parameters of interest from the FOF files and
package the parameters for input to the FGS and AST EDPS pipelines.
- Engineering Data Receipt(EDR) This pipeline receives compressed engineering telemetry
(FOF) files from CCS and uncompresses them so they can be read by processes in the FOF Conversion
pipeline.
- FOF Conversion(FOF) If it is determined that telemetry parameters of interest are
contained in the uncompressed engineering telemetry files produced by the EDR pipeline, processes
in the FOF pipeline extract the telemetry parameters of interest and store them into intermediary
files (tyyyydddhhmm.ast), and (tyyyydddhhmm.fgs). These files are subsequently
used as input for the EDPS Astrometry and FGS pipelines respectively. In addition, telemetry
parameters of interest to the STScI Engineering team are packaged into Save/Breathing files
(eng_save_yydddhhmm.fof) and (breathing_yydddhhmm.fof) and copied to the
Engineering Team.
Also, the FOF Pipeline and the OPUS EDPS in general for that matter are highly dependent upon
database interaction; see FOF Pipeline Database Usage. In order for the
FOF pipeline database interaction to occur correctly, the processes in the pipeline should be
executed in a pipeline mode as opposed to an interactive mode. Typically, as specified by the
OK_TO_UPDATE_DATABASE mnemonic in process resource files, processes that run in interactive mode
are prohibited from performing database updates. Therefore, to ensure correct operation of the pipeline,
!!! ALL FOF PIPELINE PROCESSES SHOULD BE RUN ONLY IN THE PIPELINE MODE !!!.
FOF Pipeline Nominal Operational Description
After the EDPS PDR and EDR pipelines have successfully executed for the time period covering a specific
SMS, the FOF Pipeline is ready to extract telemetry parameters of interest from input FOF files covering
the same time period. FOF pipeline processing is triggered by the appearance of uncompressed FOF telemetry
files in the FOF_INPUT_DIR directory. Then, the pipeline FGSENG, ASTENG, and
SAVENG process are triggered to extract from the input FOF files telemetry parameters that they have specific
interest in. Telemetry parameters extracted by the FGSENG and SAVENG processes are packaged into intermediary
telemetry files in the FOF_FGS_DIR and FOF_AST_DIR
directories respectively. The intermediary telemetry files are used as input to the EDPS FGS and Astrometry
pipelines. Telemetry parameters extracted by the SAVENG process are packaged into Save files and Breathing
files in the FOF_SAV_DIR directory. SAVENG also copies the files to the STScI
Engineering Team. Once all telemetry parameters of interest are extracted from an input FOF file, pipeline
clean-up occurs. This clean-up involves deleting the copies of the Save and Breathing files, and deleting the
input FOF files along with their corresponding OSFs.
FOF Pipeline Directories and Files
- OPUS_HOME_DIR
contains FOF Process Status Files (PSFs) and Process Log Files
- OPUS_HOME_DIR_LOCK
contains transient Lock Files used to avoid PSF collisions
- OPUS_OBSERVATIONS_DIR
contains FOF Observation Status Files (OSFs)
- OPUS_OBSERVATIONS_DIR_LOCK
contains transient Lock Files used to avoid OSF collisions
- OPUS_DEFINITIONS_DIR
contains Path files, Stage Files, Process Resource Files, OPUS Manager Data Files, and other configuration
related files:
fof.path_template - defines directory mnemonics used by the FOF Pipeline.
Also defines mnemonics such as OPUS_DB, DSQUERY, and OK_TO_UPDATE_DATABASE for database access
null.path - defines directory mnemonics used by FOF processes when they are
being executed in an interactive mode.
fof_pipeline.stage - defines the 8 stages in the FOF Pipeline; the processes
that execute within each stage; and the possible status values for each stage
- FF - FOF Telemetry File Polling Stage
- FE - FGS related Telemetry Extraction Stage
- AE - Astrometry related Telemetry Extraction Stage
- SE - Save/Breathing File related Telemetry Extraction Stage
- CK - Intermediary Telemetry File rename Stage
- FS - FTP Save/Breathing File Stage
- CL - Delete Save/Breathing File Stage
- DL - Delete Telemetry File/OSF Stage
FOFPOL.RESOURCE - defines characteristics of the FOFPOL Process
FGSENG.RESOURCE - defines characteristics of the FGSENG Process
ASTENG.RESOURCE - defines characteristics of the ASTENG Process
SAVENG.RESOURCE - defines characteristics of the SAVENG Process
CHKFOF.RESOURCE - defines characteristics of the CHKFOF Process
FTPSAV.RESOURCE - defines characteristics of the FTPSAV Process
DELSAV.RESOURCE - defines characteristics of the DELSAV Process
DL_FOF.RESOURCE - defines characteristics of the DL_FOF Process
FOFDEL.RESOURCE - defines characteristics of the FOFDEL Process
pmg_restrictions.dat - used to restrict the number of copies of certain processes that
the PMG is allowed to start up
opus.env - contains user-configurable parameters that establish the format of PSTAT and
OSF blackboard entries, which blackboard implementation to use, and the directory in which PSTAT
entries are stored for FILE blackboard implementations
opus_corba_objs - contains information needed when a CORBA Blackboard implementation is
being used for interprocess communication between the OPUS Managers and the processes in a pipeline
FOF_INPUT_DIR
contains uncompressed Engineering Telemetry (FOF) files produced by the EDPS EDR Pipeline ZIPFOF process
- tyyyydddhhmm.fof - Naming Convention of FOF files before they have been found by
the FOF pipeline Polling Process (FOFPOL)
- tyyyydddhhmm.fof_done - Naming Convention of FOF files after they have been found
by the FOF pipeline Polling Process (FOFPOL)
- tyyyydddhhmm.fof_done_bad - Naming Convention of FOF files that have been
determined to be of insufficient quality by the FOF pipeline FGSENG, ASTENG, or SAVENG processes.
FOF_FGS_DIR
contains intermediary telemetry files that contain telemetry parameters of interest to the
EDPS FGS pipeline. The telemetry parameters were extracted from input FOF files and stored in
the intermediary files by the FOF pipeline FGSENG process.
- tyyyydddhhmm.fgs_good - Naming Convention of telemetry files that contain FGS
related telemetry parameters that were extracted from a FOF file that has been
determined to be of sufficient quality by the FOF pipeline FGSENG process.
- tyyyydddhhmm.fgs_bad - Naming Convention of telemetry files that contain FGS
related telemetry parameters that were extracted from a FOF file that has been
determined to be of insufficient quality by the FOF pipeline FGSENG process.
- tyyyydddhhmm.fgs - Naming Convention of telemetry files that contain FGS related
telemetry parameters that were extracted from a FOF file that has been determined to be of
sufficient quality by the FOF pipeline FGSENG, ASTENG, and SAVENG processes.
FOF_AST_DIR
contains intermediary telemetry files that contain telemetry parameters of interest to the
EDPS Astrometry (AST) pipeline. The telemetry parameters were extracted from input FOF files
and stored in the intermediary files by the FOF pipeline ASTENG process.
- tyyyydddhhmm.ast_good - Naming Convention of telemetry files that contain AST
related telemetry parameters that were extracted from a FOF file that has been
determined to be of sufficient quality by the FOF pipeline ASTENG process.
- tyyyydddhhmm.ast_bad - Naming Convention of telemetry files that contain AST
related telemetry parameters that were extracted from a FOF file that has been
determined to be of insufficient quality by the FOF pipeline ASTENG process.
- tyyyydddhhmm.ast - Naming Convention of telemetry files that contain AST related
telemetry parameters that were extracted from a FOF file that has been determined to be of
sufficient quality by the FOF pipeline FGSENG, ASTENG, and SAVENG processes.
FOF_SAV_DIR
contains Save Files and Breathing files that get copied to the STScI Engineering Team.
The data in the files was extracted from input FOF files by the FOF pipeline SAVENG process.
- (eng_save_yydddhhmm.fof) - Naming Convention of Save files containing telemetry parameters
that were extracted from a FOF file that has been determined to be of sufficient quality by the FOF
pipeline SAVENG process.
- (eng_save_yydddhhmm.fof_bad) - Naming Convention of Save files containing telemetry parameters
that were extracted from a FOF file that has been determined to be of insufficient quality by the FOF
pipeline SAVENG process.
- (breathing_yydddhhmm.fof) - Naming Convention of Breathing files containing telemetry parameters
that were extracted from a FOF file that has been determined to be of sufficient quality by the FOF
pipeline SAVENG process.
- (breathing_yydddhhmm.fof_bad) - Naming Convention of Breathing files containing telemetry parameters
that were extracted from a FOF file that has been determined to be of insufficient quality by the FOF
pipeline SAVENG process.
FOF Pipeline Processes
FOF Pipeline Processes Overview
The FOF Pipeline is composed of the following 9 processes:
- FOFPOL - polls the FOF_INPUT_DIR directory for
uncompressed Engineering Telemetry Files produced by the EDPS EDR ZIPFOF pipeline process and creates a
'fof' class OSF for each telemetry file found.
- FGSENG - queries database to find out if any observations (other than
astrometry observations) or guide star acquisitions occur during the time frame of the telemetry
files found by the FOFPOL process. If observations or acquisitions do occur, FGS related telemetry
parameters are extracted from the FOF files and stored into intermediary files in the
FOF_FGS_DIR directory. These intermediary files are subsequently used
by the EDPS FGS pipeline.
- ASTENG - queries database to find out if any astrometry observations occur
during the time frame of the telemetry files found by the FOFPOL process. If astrometry observations
do occur, astrometry related telemetry parameters are extracted from the FOF files and stored into
intermediary files in the FOF_AST_DIR directory. These intermediary files
are subsequently used by the EDPS Astrometry pipeline.
- SAVENG - extracts Save/Breathing file related telemetry parameters from
the FOF files found by the FOFPOL process and produces Save/Breathing files in the
FOF_SAV_DIR directory.
- CHKFOF - renames the intermediary telemetry files produced by the FGSENG and
ASTENG processes in the FOF_FGS_DIR and FOF_AST_DIR
directories respectively. The files are renamed so they can be recognized and processed by the EDPS FGS
and Astrometry pipelines.
- FTPSAV - copies (via FTP) to the STScI Engineering team, the Save and
Breathing files produced by the SAVENG process in the FOF_SAV_DIR directory.
- DELSAV - deletes the Save and Breathing files from the
FOF_SAV_DIR directory after the files have been successfully copied to the STScI
Engineering team.
- DL_FOF - deletes FOF files from the FOF_INPUT_DIR
directory after the files have been successfully processed by the FGSENG, ASTENG, and SAVENG processes.
- FOFDEL - deletes 'fof' class OSFs for the FOF files in the
FOF_INPUT_DIR directory after the files have been successfully processed by
the FGSENG, ASTENG, and SAVENG processes.
FOF Pipeline Processes Details
FOFPOL Process Details
- FOFPOL Process Description
This process resides in the (FF) stage of the FOF Pipeline. It is a
file-triggered process and its purpose is to poll the
FOF_INPUT_DIR directory for uncompressed FOF
telemetry data files produced by the EDPS EDR Pipeline ZIPFOF Process. For
each FOF telemetry file found, FOFPOL creates a 'fof' class OSF in the
OPUS_OBSERVATIONS_DIR and sets the OSF to trigger
the FGSENG process in the next stage of the pipeline. Also, "_done" is
appended to the name of each FOF telemetry file found to keep the file from
being rerocessed by FOFPOL.
- FOFPOL Process Triggers
Input Trigger:
FOFPOL is a file poller process that is triggered by the appearance of uncompressed FOF telemetry files
in the FOF_INPUT_DIR directory
Output Trigger:
FOFPOL triggers the FGSENG Process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c w _ _ _ _ _ _ fof
FOFPOL Process I/O
Input to and Output from the FOFPOL process is as follows:
INPUT:
FOF_INPUT_DIR:
tyyyydddhhmm.fof - uncompressed FOF Telemetry Files generated by the EDPS EDR Pipeline ZIPFOF Process
OUTPUT:
FOF_INPUT_DIR:
tyyyyddhhmm.fof_done - renamed FOF Telemetry files
OPUS_OBSERVATIONS_DIR:
'fof' class OSF's for FOF Telemetry files
FOFPOL Process Modes
The FOFPOL process invokes a script to do file polling and it executes in
a Pipeline mode only. The script that is invoked is indicated in the
'COMMAND' mnemonic in the FOFPOL resource file.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r fofpol (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the FOFPOL process follows
opus_definitions_dir:your.path = path file to use
FOFPOL Process Resource File
This example fofpol.resource file is current as of OPUS build 2011.4, expected
1st QTR 2012.
!-------------------------------------------------------------------------------
!
! fofpol.resource
!
! External poller using xpoll
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! PR
! DATE NUMBER User Description
! -------- ------ --------- -------------------------------------
! 10/24/01 44684 Goldst Created initial version
! 03/01/02 45016 Goldst Change DATA_ID and SET_STATUS values
! 01/15/03 47209 J.Baum Add FILE_DUPLICATE.DANGLE
! 06/24/04 51412 Sherbert For PR 50437 Remove DISPLAY_ORDER
! 10/31/11 67820 LThompson change SET_STATUS from cwwww to cw to trigger only
! FE so FE,AD,SE,CK stages are triggered
! sequentially, instead of concurrently
!-------------------------------------------------------------------------------
PROCESS_NAME = fofpol
TASK = <xpoll -p $PATH_FILE -r fofpol>
DESCRIPTION = 'Polls for FOF telemetry files'
COMMAND = eng_poll.pl
SYSTEM = FOF
CLASS = fof
FILE_RANK = 1
FILE_DIRECTORY1 = FOF_INPUT_DIR
FILE_OBJECT1 = *.fof
FILE_PROCESSING.DANGLE = _proc
FILE_SUCCESS.DANGLE = _done
FILE_ERROR.DANGLE = _bad
FILE_DUPLICATE.DANGLE = _duplicate
XPOLL_STATE.11 = FILE_SUCCESS
XPOLL_STATE.13 = FILE_ERROR ! cannot create OSF
XPOLL_STATE.15 = FILE_DUPLICATE ! duplicate marked in osf
XPOLL_ERROR.FF = x ! Undefined exit status
POLLING_TIME = 10 ! Wait (seconds) before polling for next
! ENV values to be passed to the eng_poll.pl script
ENV.PIPELINE = FOF
ENV.DATA_ID = fof
ENV.PROC_EXT = .fof_proc
ENV.SET_STATUS = cw ! trigger FE stage only
ENV.DUPLICATE = d
ENV.POLL_STEP = FF
FGSENG Process Details
- FGSENG Process Description
This process resides in the (FE) stage of the FOF Pipeline. It is a 'fof'
class OSF-triggered process and its purpose is to extract FGS and/or Guide
Star Acquisition-related telemetry parameters from input FOF files in the
FOF_INPUT_DIR directory and package the telemetry
parameters into intermediary telemetry files in the
FOF_FGS_DIR directory. The
eng_telemetry database relation identifies
the telemetry parameters that are of interest to the FGSENG process. The
intermediary telemetry files are subsequently used as input to the EDPS FGS
Pipeline. The extraction and packaging of telemetry parameters of interest
is done only if observations or guide star acquisitions occur during the time
frame of the FOF file being processed. FGSENG queries the
product_eng_map database relation to
determine if observations do occur and thus determine if the telemetry
extraction/packaging is required. If it is not needed, FGSENG considers the
input FOF file to be of no use and considers itself to have completed its
processing successfully. When telemetry parameter extraction/packaging is
performed, various quality-related tolerances specified in the
FGSENG.RESOURCE resource file are used by FGSENG
to determine if an input FOF file is of sufficient quality. If it is not, the
intermediary telemetry files are still produced; however, "_bad" is appended
to the file name (and to the input FOF file name) to indicate that their
quality is questionable (see FOF_INPUT_DIR and
FOF_FGS_DIR for examples).
- FGSENG Process Trigger
Input Trigger:
FGSENG is triggered by the FOFPOL process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c w _ _ _ _ _ _ fof
Output Trigger:
FGSENG triggers the ASTENG process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c w _ _ _ _ _ fof
FGSENG Process I/O
Input to and Output from the FGSENG process is as follows:
INPUT:
FOF_INPUT_DIR_DIR:
tyyyydddhhmm.fof_done - input FOF file
OUTPUT:
FOF_FGS_DIR:
tyyyydddhhmm.fgs_good - intermediary telemetry file from good quality FOF file
tyyyydddhhmm.fgs_bad - intermediary telemetry file that has been renamed to indicate that it
came from a bad quality FOF file
FOF_INPUT_DIR:
tyyyydddhhmm.fof_done_bad - FOF file that has been renamed to indicate bad quality
FGSENG Process Modes
The FGSENG process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the FGSENG resource
file. The script in turn invokes a JAVA application that is reliant on
command line parameters provided by CORBA Servers. Therefore, FGSENG
must use a CORBA Blackboard and must be run only in a pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r fgseng (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the FGSENG process follows
opus_definitions_dir:your.path = path file to use
FGSENG Process Resource File
This example fgseng.resource file is current as of OPUS build 2011.4, expected
1st QTR 2012.
!-------------------------------------------------------------------------------
!
! fgseng.resource
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 10/24/01 44684 Goldst Created initial version
! 001 01/25/02 44684 jschultz various additions and modifications
! 002 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
! 003 03/13/03 47496 jschultz add new XPOLL_STATE and OSF_HOLD settings
! to use when need to hold FOF processing
! 004 03/25/03 47061 J.Baum Add two new dangles
! 005 08/12/03 49253 Goldst Added JCONN_JAR keyword
! 006 02/15/05 51570 Sontag Added missing XPOLL_ERROR line
! 007 03/18/05 51570 MSwam Sync with operational override
! 08/12/10 64432 MSwam use path-defined DB_TYPE (SYBASE or MSSQL)
! 10/31/2011 67820 LThompson add OSF_SUCCESS.AE to trigger next stage
!-------------------------------------------------------------------------------
!---------------------
! General Process info
!---------------------
PROCESS_NAME = fgseng
TASK = <xpoll -p $PATH_FILE -r fgseng>
COMMAND = fgseng.csh ^CL
DESCRIPTION = 'Produce FGS Subset Files from FOF Files'
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1
POLLING_TIME = 10 ! Wait (seconds) before polling again
!-------------
! OSF settings
!-------------
OSF_TRIGGER1.FE = w ! Process Trigger OSF
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.FE = p ! Process Processing OSF
OSF_SUCCESS.FE = c ! Success status: FE stage complete
OSF_SUCCESS.AE = w ! Success status: trigger AE stage
OSF_FAILURE.FE = f ! Process Failure OSF
OSF_HOLD.FE = h ! hold for MSC file processing
!-------------------
! Directories to use
!-------------------
ENV.INPATH = FOF_INPUT_DIR ! Contains input FOF Files
ENV.OUTPATH = FOF_FGS_DIR ! Contains output FGS Subset Files
!-----------------------
!message reporting level
!-----------------------
MSG_RPT_LVL = JMSG_REPORT_LEVEL ! message report level to use. Set
! MSG_RPT_LVL equal to JMSG_REPORT_LEVEL
! to cause your report level to be set
! to what JMSG_REPORT_LEVEL is set to in
! your path file. Otherwise set MSG_RPT_LVL
! to one of he following:
! ALL, NONE, DIAG, INFO, WARN, ERROR, FATAL
!--------------
! Database info
!--------------
DB_TYPE = DB_TYPE ! database type to use.
DB_SERVER = DSQUERY ! database server to use. Set DB_SERVER equal to
! DSQUERY. This will cause your server to be set
! to what DSQUERY is set to in your path file.
DB_NAME = OPUS_DB ! database to use. Set DB_NAME to OPUS_DB. This
! will cause your database to be set
! to what OPUS_DB is set to in your path file.
!-------------------------
! Process Control Switches
!-------------------------
DROP_LAST_VALUE_BEFORE_GAP = Y ! Indicates if to drop the last value of
! telemetry parameters that occur just prior
! to a gap.
! Set to "Y"es or "N"o
RETAIN_SUPERCOMMUTATED_DATA = N ! Indicates if supercommutated data is to be
! contained in the output product. In FOF
! data there is 1 packet per minor frame for
! non-supercommutated data and 2 packets per
! minor frame for supercommutated data with
! the supercommutated data contained in the
! second packet. For typical FGS related
! processing the supercommutated data is not
! required so this switch should be set to
! "N".
CHECK_FOF_TOLERANCES = Y ! Indicates if to see if any quality related
! tolerances were exceeded during fof
! conversion. If tolerances are not checked,
! fof file is assumed to be of good quality
! Set to "Y"es or "N"o
PRINT_FOF_CONVERSION_SUMMARY = N ! Indicates if to print out various counters
! that were maintained during fof conversion;
! for example, number of good elements,
! number of bad elements, number of discarded
! elements ... If a FOF Conversion Summary is
! selected to be printed, it will be printed
! to the fgseng process log file.
! Set to "Y"es or "N"o
PRINT_FOF_QUALITY_SUMMARY = Y ! Indicates if to print out various quality
! metrics that were computed during fof
! conversion; for example, percentage of
! elements discarded, percentage of packets
! discarded, ...
! If a FOF Quality Summary is selected to be
! printed and the CHECK_FOF_TOLERANCES switch
! is set to 'Y', the summary will be printed to
! the fgseng process log file.
! Set to "Y"es or "N"o.
PRINT_FOF_GAP_SUMMARY = Y ! Indicates if to print out information
! relating to gaps found in the FOF data
! during conversion; for example, the number
! of gaps, their start and end times, and their
! length. If a FOF GAP Summary is selected to
! be printed, it will be printed to the fgseng
! process log file.
! Set to "Y"es or "N"o
!-------------------------------
! FOF quality related Tolerances
!-------------------------------
BAD_ELEMENT_TOLERANCE = 10 ! Indicates the allowable percentage of
! bad quality telemetry elements that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the FE column and the file names of the FOF
! file and any FGS Subset file produced will
! be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DROPPED_ELEMENT_TOLERANCE = 10 ! Indicates the allowable percentage of
! telemetry elements that can be discarded
! during fof conversion because they contain
! the last values of the parameters prior to
! the start of a gap. If this percentage is
! exceeded, the OSF for the FOF File will be
! set to 'f' (fail) in the FE column and the
! file names of the FOF file and any FGS
! Subset file produced will be appended with
! the QUESTIONABLE_DATA_DANGLE value
! specified below in the Process Parameters
! section of this resource file
DISCARDED_ELEMENT_TOLERANCE = 15 ! Indicates the total allowable percentage of
! telemetry elements that can be discarded
! for any reason during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the FE column and the file names of the FOF
! file and any FGS Subset file produced will
! be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
BAD_PACKET_TOLERANCE = 10 ! Indicates the allowable percentage of
! bad quality telemetry packets that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the FE column and the file names of the FOF
! file and any FGS Subset file produced will
! be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DUPLICATE_PACKET_TOLERANCE = 5 ! Indicates the allowable percentage of
! duplicate telemetry packets that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the FE column and the file names of the FOF
! file and any FGS Subset file produced will
! be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
GAP_PACKET_TOLERANCE = 10 ! Indicates the allowable percentage of gap
! packets that can be discarded during fof
! conversion. If this percentage is exceeded,
! the OSF for the FOF File will be set to 'f'
! (fail) in the FE column and the file names
! of the FOF file and any FGS Subset file
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DISCARDED_PACKET_TOLERANCE = 15 ! Indicates the total allowable percentage of
! telemetry packets that can be discarded
! for any reason during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the FE column and the file names of the FOF
! file and any FGS Subset file produced will
! be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
!-------------------
! Process Parameters
!-------------------
TELEMETRY_TYPE = FGS ! Indicates that only FGS related
! telemetry parameters are of interest
! during fof conversion
PROCESS_DATA_DANGLE = _proc ! appended to names of FGS Subset data
! files at start of processing. The value
! will be replaced by GOOD_DATA_DANGLE or
! QUESTIONABLE_DATA_DANGLE at end of
! processing.
INDEX_DATA_DANGLE = _index ! appended to names of FGS Subset index
! files at start of processing. This
! is not changed after processing.
GOOD_DATA_DANGLE = _good ! appended to names of FGS Subset files
! when quality tolerances are checked and
! found to be within limits
QUESTIONABLE_DATA_DANGLE = _bad ! appended to names of FOF and FGS Subset
! files when quality tolerances are
! checked and found to be exceeded
MINBLOCKS = 50000 ! minimum disk space needed to run process
!--------------------
! Xpoll return states
!--------------------
XPOLL_STATE.00 = OSF_SUCCESS
XPOLL_STATE.01 = OSF_FAILURE
XPOLL_STATE.02 = OSF_HOLD
XPOLL_ERROR.FE = x ! Some entirely unexpected failure
!-------------------------------------------------------------------------------
! causes environment variables to be set in the sub process spawned by XPOLL
! to execute the scipt specified by the "COMMAND" in the "General Process info"
! section at the beginning of this resource file
!-------------------------------------------------------------------------------
ENV.OPUS_DB = OPUS_DB
ENV.DSQUERY = DSQUERY
ENV.OK_TO_UPDATE_DATABASE = OK_TO_UPDATE_DATABASE
ENV.JCONN_JAR = JCONN_PATH
FGSENG Process-accessed database relations
FGSENG accessed Relations
FGSENG Process database queries
FGSENG Process Queries
ASTENG Process Details
- ASTENG Process Description
This process resides in the (AE) stage of the FOF Pipeline. It is a 'fof'
class OSF-triggered process and its purpose is to extract Astrometry-related
telemetry parameters from input FOF files in the
FOF_INPUT_DIR directory and package the telemetry
parameters into intermediary telemetry files in the
FOF_AST_DIR directory. The
eng_telemetry database relation identifies
the telemetry parameters that are of interest to the ASTENG process. The
intermediary telemetry files are subsequently used as input to the EDPS
Astrometry Pipeline. The extraction and packaging of telemetry parameters of
interest is done only if astrometry observations occur during the time frame
of the FOF file being processed. ASTENG queries the
product_eng_map database relation to
determine if astrometry observations do occur and thus determine if the
telemetry extraction/packaging is required. If it is not needed, ASTENG
considers the input FOF file to be of no use and considers itself to have
completed its processing successfully. When telemetry parameter
extraction/packaging is performed, various quality-related tolerances
specified in the ASTENG.RESOURCE resource file
are used by ASTENG to determine if the input FOF file is of sufficient quality.
If it is not, the intermediary files are still produced; however, "_bad" is
appended to the file name (also to the input FOF file name) to indicate that
their quality is questionable (see FOF_INPUT_DIR
and FOF_AST_DIR for examples).
- ASTENG Process Trigger
Input Trigger:
ASTENG is triggered by the FGSENG process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c w _ _ _ _ _ fof
Output Trigger:
ASTENG triggers the SAVENG process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c w _ _ _ _ fof
ASTENG Process I/O
Input to and Output from the ASTENG process is as follows:
INPUT:
FOF_INPUT_DIR_DIR:
tyyyydddhhmm.fof_done - input FOF file
OUTPUT:
FOF_AST_DIR:
tyyyydddhhmm.ast_good - intermediary telemetry file from good quality FOF file
tyyyydddhhmm.ast_bad - intermediary telemetry file that has been renamed to indicate that it
came from a bad quality FOF file
FOF_INPUT_DIR:
tyyyydddhhmm.fof_done_bad - FOF file that has been renamed to indicate bad quality
ASTENG Process Modes
The ASTENG process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the ASTENG resource
file. The script in turn invokes a JAVA application that is reliant on
command line parameters provided by CORBA Servers. Therefore, ASTENG
must use a CORBA Blackboard and must be run only in a pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r asteng (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the ASTENG Process follows
opus_definitions_dir:your.path = path file to use
ASTENG Process Resource File
This example asteng.resource file is current as of OPUS build 2011.4, expected
1st QTR 2012.
!-------------------------------------------------------------------------------
!
! asteng.resource
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 10/24/01 44684 Goldst Created initial version
! 001 01/25/02 44684 jschultz various additions and modifications
! 002 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
! 003 03/13/03 47496 jschultz add new XPOLL_STATE and OSF_HOLD settings
! to use when need to hold FOF processing
! 004 03/25/03 47061 J.Baum Add two new dangles
! 005 08/12/03 49253 Goldst Added JCONN_JAR keyword
! 006 02/15/05 51570 Sontag Added missing XPOLL_ERROR line
! 08/12/10 64432 MSwam use path-defined DB_TYPE (SYBASE or MSSQL)
! 10/31/2011 67820 LThompson add OSF_SUCCESS.SE to trigger next stage
!-------------------------------------------------------------------------------
!---------------------
! General Process info
!---------------------
PROCESS_NAME = asteng
TASK = <xpoll -p $PATH_FILE -r asteng>
COMMAND = asteng.csh ^CL
DESCRIPTION = 'Produce Astrometry Subset Files from FOF Files'
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1
POLLING_TIME = 10 ! Wait (seconds) before polling again
!-------------
! OSF settings
!-------------
OSF_TRIGGER1.AE = w ! Process Trigger OSF
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.AE = p ! Process Processing OSF
OSF_SUCCESS.AE = c ! Success status: AE stage complete
OSF_SUCCESS.SE = w ! Success status: trigger SE stage
OSF_FAILURE.AE = f ! Process Failure OSF
OSF_HOLD.AE = h ! hold for MSC file processing
!-------------------
! Directories to use
!-------------------
ENV.INPATH = FOF_INPUT_DIR ! Contains input FOF Files
ENV.OUTPATH = FOF_AST_DIR ! Contains output Astrometry Subset Files
!-----------------------
!message reporting level
!-----------------------
MSG_RPT_LVL = JMSG_REPORT_LEVEL ! message report level to use. Set
! MSG_RPT_LVL equal to JMSG_REPORT_LEVEL
! to cause your report level to be set
! to what JMSG_REPORT_LEVEL is set to in
! your path file. Otherwise set MSG_RPT_LVL
! to one of he following:
! ALL, NONE, DIAG, INFO, WARN, ERROR, FATAL
!--------------
! Database info
!--------------
DB_TYPE = DB_TYPE ! database type to use.
DB_SERVER = DSQUERY ! database server to use. Set DB_SERVER equal to
! DSQUERY. This will cause your server to be set
! to what DSQUERY is set to in your path file.
DB_NAME = OPUS_DB ! database to use. Set DB_NAME to OPUS_DB. This
! will cause your database to be set
! to what OPUS_DB is set to in your path file.
!-------------------------
! Process Control Switches
!-------------------------
DROP_LAST_VALUE_BEFORE_GAP = Y ! Indicates if to drop the last value of
! telemetry parameters that occur just prior
! to a gap. Set to "Y"es or "N"o
RETAIN_SUPERCOMMUTATED_DATA = Y ! Indicates if supercommutated data is to be
! contained in the output product. In FOF
! data there is 1 packet per minor frame for
! non-supercommutated data and 2 packets per
! minor frame for supercommutated data with
! the supercommutated data contained in the
! second packet. For typical AST related
! processing the supercommutated data is
! required so this switch should be set to
! "Y".
CHECK_FOF_TOLERANCES = Y ! Indicates if to see if any quality related
! tolerances were exceeded during fof
! conversion. If tolerances are not checked,
! fof file is assumed to be of good quality
! Set to "Y"es or "N"o
PRINT_FOF_CONVERSION_SUMMARY = N ! Indicates if to print out various counters
! that were maintained during fof conversion;
! for example, number of good elements,
! number of bad elements, number of discarded
! elements ... If a FOF Conversion Summary is
! selected to be printed, it will be printed
! to the asteng process log file.
! Set to "Y"es or "N"o.
PRINT_FOF_QUALITY_SUMMARY = Y ! Indicates if to print out various quality
! metrics that were computed during fof
! conversion; for example, percentage of
! elements discarded, percentage of packets
! discarded, ...
! If a FOF Quality Summary is selected to be
! printed and the CHECK_FOF_TOLERANCES switch
! is set to 'Y', the summary will be printed to
! the asteng process log file.
! Set to "Y"es or "N"o.
PRINT_FOF_GAP_SUMMARY = Y ! Indicates if to print out information
! relating to gaps found in the FOF data
! during conversion; for example, the number
! of gaps, their start and end times, and their
! length. If a FOF GAP Summary is selected to
! be printed, it will be printed to the asteng
! process log file.
! Set to "Y"es or "N"o
!-------------------------------
! FOF quality related Tolerances
!-------------------------------
BAD_ELEMENT_TOLERANCE = 20 ! Indicates the allowable percentage of
! bad quality telemetry elements that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the AE column and the file names of the FOF
! file and any Astrometry Subset file
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DROPPED_ELEMENT_TOLERANCE = 20 ! Indicates the allowable percentage of
! telemetry elements that can be discarded
! during fof conversion because they contain
! the last values of the parameters prior to
! the start of a gap. If this percentage is
! exceeded, the OSF for the FOF File will be
! set to 'f' (fail) in the AE column and the
! file names of the FOF file and any
! Astrometry Subset file produced will be
! appended with the QUESTIONABLE_DATA_DANGLE value
! specified below in the Process Parameters
! section of this resource file
DISCARDED_ELEMENT_TOLERANCE = 25 ! Indicates the total allowable percentage of
! telemetry elements that can be discarded
! for any reason during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the AE column and the file names of the FOF
! file and any Astrometry Subset file
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
BAD_PACKET_TOLERANCE = 20 ! Indicates the allowable percentage of
! bad quality telemetry packets that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the AE column and the file names of the FOF
! file and any Astrometry Subset file
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DUPLICATE_PACKET_TOLERANCE = 20 ! Indicates the allowable percentage of
! duplicate telemetry packets that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the AE column and the file names of the FOF
! file and any Astrometry Subset file
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
GAP_PACKET_TOLERANCE = 20 ! Indicates the allowable percentage of gap
! packets that can be discarded during fof
! conversion. If this percentage is exceeded,
! the OSF for the FOF File will be set to 'f'
! (fail) in the AE column and the file names
! of the FOF file and any Astrometry Subset
! file produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DISCARDED_PACKET_TOLERANCE = 25 ! Indicates the total allowable percentage of
! telemetry packets that can be discarded
! for any reason during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the AE column and the file names of the FOF
! file and any Astrometry Subset file
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
!-------------------
! Process Parameters
!-------------------
TELEMETRY_TYPE = AST ! Indicates that only astrometry related
! telemetry parameters are of interest
! during fof conversion
PROCESS_DATA_DANGLE = _proc ! appended to names of FGS Subset data
! files at start of processing. The value
! will be replaced by GOOD_DATA_DANGLE or
! QUESTIONABLE_DATA_DANGLE at end of
! processing.
INDEX_DATA_DANGLE = _index ! appended to names of FGS Subset index
! files at start of processing. This
! is not changed after processing.
GOOD_DATA_DANGLE = _good ! appended to names of Astrometry Subset
! files when quality tolerances are
! checked and found to be within limits
QUESTIONABLE_DATA_DANGLE = _bad ! appended to names of FOF and Astrometry
! Subset files when quality tolerances
! are checked and found to be exceeded
MINBLOCKS = 50000 ! minimum disk space needed to run process
!--------------------
! Xpoll return states
!--------------------
XPOLL_STATE.00 = OSF_SUCCESS
XPOLL_STATE.01 = OSF_FAILURE
XPOLL_STATE.02 = OSF_HOLD
XPOLL_ERROR.AE = x ! Some entirely unexpected failure
!-------------------------------------------------------------------------------
! causes environment variables to be set in the sub process spawned by XPOLL
! to execute the scipt specified by the "COMMAND" in the "General Process info"
! section at the beginning of this resource file
!-------------------------------------------------------------------------------
ENV.OPUS_DB = OPUS_DB
ENV.DSQUERY = DSQUERY
ENV.OK_TO_UPDATE_DATABASE = OK_TO_UPDATE_DATABASE
ENV.JCONN_JAR = JCONN_PATH
ASTENG Process-accessed database relations
ASTENG accessed Relations
ASTENG Process database queries
ASTENG Process Queries
SAVENG Process Details
- SAVENG Process Description
This process resides in the (SE) stage of the FOF Pipeline. It is a 'fof'
class OSF-triggered process and its purpose is to extract Save file and
Breathing file-related telemetry parameters from input FOF files in the
FOF_INPUT_DIR directory. The
save_telemetry database relation identifies
the telemetry parameters that are of interest to the SAVENG process. The
telemetry parameters that are extracted are output to Save and/or Breathing
files in the FOF_SAV_DIR directory. The Save files
contain only parameters that were extracted from the input FOF files. The
Breathing files contain additional values that are computed from the extracted
telemetry parameters. SAVENG generation of Save and/or Breathing files is
optional as specified by switches in the
SAVENG.RESOURCE resource file. When telemetry
parameter extraction/packaging is performed, various quality-related
tolerances specified in the SAVENG.RESOURCE
resource file are used by SAVENG to determine if the input FOF file is of
sufficient quality. If it is not, the Save and Breathing files are still
produced; however, "_bad" is appended to the file names (also to the input
FOF file name) to indicate that their quality is questionable (see
FOF_INPUT_DIR and
FOF_SAV_DIR for examples).
- SAVENG Process Trigger
Input Trigger:
SAVENG is triggered by the ASTENG process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c w _ _ _ _ fof
Output Trigger:
SAVENG triggers the CHKFOF process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c w _ _ _ fof
SAVENG Process I/O
Input to and Output from the SAVENG process is as follows:
INPUT:
FOF_INPUT_DIR_DIR:
tyyyydddhhmm.fof_done - input FOF file
OUTPUT:
FOF_SAV_DIR:
eng_save_yydddhhmm.fof - Save file containing telemetry parameters from good quality FOF file
eng_save_yydddhhmm.fof_bad - Save file renamed to indicate its contents came from a bad
quality FOF file
breathing_yydddhhmm.fof - Breathing files containing telemetry parameters from good quality
FOF file
breathing_yydddhhmm.fof_bad - Breathing file renamed to indicate its contents came from a
bad quality FOF file
FOF_INPUT_DIR:
tyyyydddhhmm.fof_done_bad - FOF file that has been renamed to indicate bad quality
SAVENG Process Modes
The SAVENG process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the SAVENG resource
file. The script in turn invokes a JAVA application that is reliant on
command line parameters provided by CORBA Servers. Therefore, SAVENG
must use a CORBA Blackboard and must be run only in a pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r saveng (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the SAVENG Process follows
opus_definitions_dir:your.path = path file to use
SAVENG Process Resource File
This example saveng.resource file is current as of OPUS build 2011.4, expected
1st QTR 2012.
!-------------------------------------------------------------------------------
!
! saveng.resource
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 10/24/01 44684 Goldst Created initial version
! 001 01/25/02 44684 jschultz various additions and modifications
! 002 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
! 003 08/12/03 49253 Goldst Added JCONN_JAR keyword
! 004 02/15/05 51570 Sontag Added missing XPOLL_ERROR line
! 08/12/10 64432 MSwam use path-defined DB_TYPE (SYBASE or MSSQL)
! 10/31/2011 67820 LThompson add OSF_SUCCESS.CK to trigger next stage
!-------------------------------------------------------------------------------
!---------------------
! General Process info
!---------------------
PROCESS_NAME = saveng
TASK = <xpoll -p $PATH_FILE -r saveng>
COMMAND = saveng.csh ^CL
DESCRIPTION = 'Produce Save and Breathing Files from FOF Files'
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1
POLLING_TIME = 10 ! Wait (seconds) before polling again
!-------------
! OSF settings
!-------------
OSF_TRIGGER1.SE = w ! Process Trigger OSF
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.SE = p ! Process Processing OSF
OSF_SUCCESS.SE = c ! Success status: SE stage complete
OSF_SUCCESS.CK = w ! Success status: trigger CK stage
OSF_FAILURE.SE = f ! Process Failure OSF
!-------------------
! Directories to use
!-------------------
ENV.INPATH = FOF_INPUT_DIR ! Contains input FOF Files
ENV.OUTPATH = FOF_SAV_DIR ! Contains output Save and Breathing Files
!-----------------------
!message reporting level
!-----------------------
MSG_RPT_LVL = JMSG_REPORT_LEVEL ! message report level to use. Set
! MSG_RPT_LVL equal to JMSG_REPORT_LEVEL
! to cause your report level to be set
! to what JMSG_REPORT_LEVEL is set to in
! your path file. Otherwise set MSG_RPT_LVL
! to one of he following:
! ALL, NONE, DIAG, INFO, WARN, ERROR, FATAL
!--------------
! Database info
!--------------
DB_TYPE = DB_TYPE ! database type to use.
DB_SERVER = DSQUERY ! database server to use. Set DB_SERVER equal to
! DSQUERY. This will cause your server to be set
! to what DSQUERY is set to in your path file.
DB_NAME = OPUS_DB ! database to use. Set DB_NAME to OPUS_DB. This
! will cause your database to be set
! to what OPUS_DB is set to in your path file.
!-------------------------
! Process Control Switches
!-------------------------
GENERATE_SAVE_FILES = Y ! Indicates if Save Files are to be
! produced during fof conversion.
! Set to "Y"es or "N"o
GENERATE_BREATHING_FILES = Y ! Indicates if Breathing Files are to be
! produced during fof conversion.
! Set to "Y"es or "N"o
DROP_LAST_VALUE_BEFORE_GAP = Y ! Indicates if to drop the last value of
! telemetry parameters that occur just prior
! to a gap.
! Set to "Y"es or "N"o
CHECK_FOF_TOLERANCES = Y ! Indicates if to see if any quality related
! tolerances were exceeded during fof
! conversion. If tolerances are not checked,
! fof file is assumed to be of good quality
! Set to "Y"es or "N"o
PRINT_FOF_CONVERSION_SUMMARY = N ! Indicates if to print out various counters
! that were maintained during fof conversion;
! for example, number of good elements,
! number of bad elements, number of discarded
! elements ... If a FOF Conversion Summary is
! selected to be printed, it will be printed
! to the saveng process log file.
! Set to "Y"es or "N"o
PRINT_FOF_QUALITY_SUMMARY = Y ! Indicates if to print out various quality
! metrics that were computed during fof
! conversion; for example, percentage of
! elements discarded, percentage of packets
! discarded, ...
! If a FOF Quality Summary is selected to be
! printed and the CHECK_FOF_TOLERANCES switch
! is set to 'Y', the summary will be printed to
! the saveng process log file.
! Set to "Y"es or "N"o.
!-------------------------------
! FOF quality related Tolerances
!-------------------------------
BAD_ELEMENT_TOLERANCE = 10 ! Indicates the allowable percentage of
! bad quality telemetry elements that can
! be discarded during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the SE column and the file names of the FOF
! file and any Save and Breathing files
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
DROPPED_ELEMENT_TOLERANCE = 10 ! Indicates the allowable percentage of
! telemetry elements that can be discarded
! during fof conversion because they contain
! the last values of the parameters prior to
! the start of a gap. If this percentage is
! exceeded, the OSF for the FOF File will be
! set to 'f' (fail) in the SE column and the
! file names of the FOF file and any Save and
! Breathing files produced will be appended
! with the QUESTIONABLE_DATA_DANGLE value
! specified below in the Process Parameters
! section of this resource file
DISCARDED_ELEMENT_TOLERANCE = 15 ! Indicates the total allowable percentage of
! telemetry elements that can be discarded
! for any reason during fof conversion. If
! this percentage is exceeded, the OSF for
! the FOF File will be set to 'f' (fail) in
! the SE column and the file names of the FOF
! file and any Save and Breathing files
! produced will be appended with the
! QUESTIONABLE_DATA_DANGLE value specified
! below in the Process Parameters section of
! this resource file
!-------------------
! Process Parameters
!-------------------
QUESTIONABLE_DATA_DANGLE = _bad ! appended to names of FOF, Save, and
! Breathing files when quality tolerances
! are checked and found to be exceeded
MINBLOCKS = 50000 ! minimum disk space needed to run process
!--------------------
! Xpoll return states
!--------------------
XPOLL_STATE.00 = OSF_SUCCESS
XPOLL_STATE.01 = OSF_FAILURE
XPOLL_ERROR.SE = x ! Some entirely unexpected failure
!-------------------------------------------------------------------------------
! causes environment variables to be set in the sub process spawned by XPOLL
! to execute the scipt specified by the "COMMAND" in the "General Process info"
! section at the beginning of this resource file
!-------------------------------------------------------------------------------
ENV.OPUS_DB = OPUS_DB
ENV.DSQUERY = DSQUERY
ENV.OK_TO_UPDATE_DATABASE = OK_TO_UPDATE_DATABASE
ENV.JCONN_JAR = JCONN_PATH
SAVENG Process-accessed database relations
SAVENG accessed Relations
SAVENG Process database queries
SAVENG Process Queries
CHKFOF Process Details
- CHKFOF Process Description
This process resides in the (CK) stage of the FOF Pipeline. It is a 'fof'
class OSF-triggered process. Its purpose is to rename the intermediary
telemetry files produced by the FGSENG and ASTENG processes in the
FOF_FGS_DIR and
FOF_AST_DIR directories, respectively, so they can
be recognized and processed by the EDPS FGS and Astrometry pipelines. CHKFOF
is also indirectly a "gatekeeper" that protects the FGS and Astrometry
pipelines from using products that were produced from a FOF file of
questionable quality. CHKFOF is not triggered until the FGSENG, ASTENG, and
SAVENG processes each successfully complete for a given FOF file and determine
the FOF file to be of sufficient quality to produce output products.
- CHKFOF Process Trigger
Input Trigger:
CHKFOF is triggered by the SAVENG process (SAVENG sets the 'w' in CK, but
the CHKFOF trigger also requires a 'c' in each of the FE, AE, and SE stages)
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c w _ _ _ fof
Output Trigger:
CHKFOF triggers the FTPSAV process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c w _ _ fof
CHKFOF Process I/O
Input to and Output from the CHKFOF process is as follows:
INPUT:
FOF_FGS_DIR:
tyyyydddhhmm.fgs_good - intermediary FGS telemetry files produced from good quality FOF files
FOF_AST_DIR:
tyyyydddhhmm.ast_good - intermediary astrometry telemetry files produced from good quality FOF files
OUTPUT:
FOF_FGS_DIR:
tyyyydddhhmm.fgs - intermediary FGS telemetry files renamed to enable them to be recognized
and processed by the EDPS FGS pipeline
FOF_AST_DIR:
tyyyydddhhmm.ast - intermediary astrometry telemetry files renamed to enable them to be recognized
and processed by the EDPS Astrometry pipeline
CHKFOF Process Modes
The CHKFOF process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the CHKFOF resource
file. The script, and therefore the CHKFOF process executes in only the
pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r chkfof (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the CHKFOF process follows
opus_definitions_dir:your.path = path file to use
CHKFOF Process Resource File
This example chkfof.resource file is current as of OPUS build 2011.3,
December 2011.
!--------------------------------------------------------------------
!
! chkfof.resource
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! PR
! DATE NUMBER User Description
! -------- ------ --------- -------------------------------------
! 10/24/01 44684 Goldst Created initial version
! 01/15/02 45016 Goldst Corrected Triggers
! 03/05/02 45016 Goldst Add TRIGGER1.CK and DATA_ID
! 02/13/03 47570 J.Baum Change exit status values
! 06/24/04 51412 Sherbert For PR 50437 Remove DISPLAY_ORDER
!--------------------------------------------------------------------
PROCESS_NAME = chkfof
TASK = <xpoll -p $PATH_FILE -r chkfof>
DESCRIPTION = 'Process FOF engineering data'
COMMAND = check_fof.pl
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.FE = c ! Trigger
OSF_TRIGGER1.AE = c ! Trigger
OSF_TRIGGER1.SE = c ! Trigger
OSF_TRIGGER1.CK = w ! Trigger
OSF_TRIGGER1.DATA_ID = fof ! OSF data_id
OSF_PROCESSING.CK = p ! Processing
OSF_SUCCESS.CK = c ! Completion
OSF_SUCCESS.FS = w ! Completion
OSF_FAILURE.CK = f ! Failure
XPOLL_ERROR.CK = x ! Undefined exit status
XPOLL_STATE.07 = OSF_FAILURE
XPOLL_STATE.09 = OSF_SUCCESS
POLLING_TIME = 10 ! Wait (seconds) before polling for next
ENV.FGS_INPATH = FOF_FGS_DIR !
ENV.AST_INPATH = FOF_AST_DIR !
ENV.FGS_EXT_INPUT = .fgs_good !
ENV.AST_EXT_INPUT = .ast_good !
ENV.FGS_EXT_OUTPUT = .fgs !
ENV.AST_EXT_OUTPUT = .ast !
! forces values from path to be used
FTPSAV Process Details
- FTPSAV Process Description
This process resides in the (FS) stage of the FOF Pipeline. It is a 'fof'
class OSF-triggered process and its purpose is to copy the Save and Breathing
files produced by the SAVENG process in the FOF_SAV_DIR
directory to the STScI Engineering Team. FTPSAV accomplishes this task by
FTPing the files to a remote directory that can be accessed by the Engineering
Team. The remote directory is specified in the
FTPSAV.RESOURCE resource file by the "RMT_DIR"
mnemonic.
- FTPSAV Process Trigger
Input Trigger:
FTPSAV is triggered by the CHKFOF process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c w _ _ fof
Output Trigger:
FTPSAV triggers the DELSAV process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c c w _ fof
FTPSAV Process I/O
Input to and Output from the FTPSAV process is as follows:
INPUT:
FOF_SAV_DIR:
eng_save_yydddhhmm.fof - Save file to copy to Engineering Team
breathing_yydddhhmm.fof - Breathing file to copy to Engineering team
OUTPUT:
Save and Breathing files copied to the STScI Engineering Team
FTPSAV Process Modesi
The FTPSAV process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the FTPSAV resource
file. The script, and therefore the FTPSAV process executes in only the
pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r ftpsav (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the FTPSAV process follows
opus_definitions_dir:your.path = path file to use
FTPSAV Process Resource File
This example ftpsav.resource file is current as of OPUS build 2011.3,
December 2011.
!--------------------------------------------------------------------
!
! ftpsav.resource
!
! External poller using xpoll
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 10/24/01 44684 Goldst Created initial version
! 001 01/15/02 45016 Goldst Corrected Trigger
! 002 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
! 09/23/02 46357 Sherbe Set OSF_RANK=2,TIME_RANK=1 so process will go
! --- --- --- absent on incorrect password without failing OSFs
! 06/24/04 51412 Sherbert For PR 50437 Remove DISPLAY_ORDER
! 06/19/10 45189 MSwam changes for SFTP
!--------------------------------------------------------------------
PROCESS_NAME = ftpsav
TASK = <xpoll -p $PATH_FILE -r ftpsav>
DESCRIPTION = 'FTPs save files for engineering analysis'
COMMAND = save_copy.csh ^CL
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1
OSF_TRIGGER1.FS = w ! Trigger
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.FS = p ! Processing
OSF_SUCCESS.FS = c ! Completion
OSF_SUCCESS.CL = w ! Completion
OSF_FAILURE.FS = f ! Failure setting
XPOLL_ERROR.FS = x ! Undefined exit status
XPOLL_STATE.00 = OSF_SUCCESS
XPOLL_STATE.01 = OSF_FAILURE
ENV.INPATH = FOF_SAV_DIR
ENV.EXTENSION = .fof
ENV.RMT_MACHINE = ET_MACHINE ! Engineering machine
ENV.RMT_LOGIN = ET_LOGIN ! Engineering login
ENV.RMT_DIR = ET_REMOTE_DIR ! Engineering remote directory
POLLING_TIME = 10 ! Wait (seconds) before polling for next
! forces values from path to be used
DELSAV Process Details
- DELSAV Process Description
This process resides in the (CL) stage of the FOF Pipeline. It is a 'fof'
class OSF-triggered process and its purpose is to delete the Save and
Breathing files produced by the SAVENG process in the
FOF_SAV_DIR directory after the files have been
made available to the STScI Engineering Team by the FTPSAV process.
- DELSAV Process Trigger
Input Trigger:
DELSAV is triggered by the FTPSAV process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c c w _ fof
Output Trigger:
DELSAV triggers the DL_FOF process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c c c w fof
DELSAV Process I/O
Input to and Output from the DELSAV process is as follows:
INPUT:
FOF_SAV_DIR:
eng_save_yydddhhmm.fof - Save file to delete
breathing_yydddhhmm.fof - Breathing file to delete
OUTPUT:
N/A
DELSAV Process Modes
The DELSAV process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the DELSAV resource
file. The script, and therefore the DELSAV process executes in only the
pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r delsav (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the DELSAV process follows
opus_definitions_dir:your.path = path file to use
DELSAV Process Resource File
This example delsav.resource file is current as of OPUS build 2011.3,
December 2011.
!--------------------------------------------------------------------
!
! delsav.resource
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 10/24/01 44684 Goldst Created initial version
! 001 01/15/02 45016 Goldst Corrected Trigger
! 002 01/25/02 45016 Goldst ENV.OK_TO_UPDATE_DATABASE from path
! 003 03/06/02 45016 Goldst Change XPOLL states and ENV.
! 004 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
! 06/24/04 51412 Sherbert For PR 50437 Remove DISPLAY_ORDER
!--------------------------------------------------------------------
PROCESS_NAME = delsav
TASK = <xpoll -p $PATH_FILE -r delsav>
COMMAND = delete_save.csh
DESCRIPTION = 'Deletes save files '
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.CL = w ! Trigger
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.CL = p ! Processing
OSF_COMPLETE.CL = c ! Completion
OSF_COMPLETE.DL = w ! Completion - trigger for dl_fof
OSF_FAILURE.CL = f ! Failure setting
POLLING_TIME = 10 ! Wait (seconds) before polling for next
XPOLL_STATE.00 = OSF_COMPLETE ! Completion state
XPOLL_STATE.01 = OSF_FAILURE ! Failure state
XPOLL_ERROR.CL = x ! Undefined exit status
ENV.INPATH = FOF_SAV_DIR !
ENV.EXTENSION = .fof !
! forces values from path to be used
ENV.OPUS_DB = OPUS_DB
ENV.DSQUERY = DSQUERY
ENV.OK_TO_UPDATE_DATABASE = OK_TO_UPDATE_DATABASE ! Determined by path file
DL_FOF Process Details
- DL_FOF Process Description
This is the first of two processes which reside in the (DL) stage of the FOF
Pipeline. It is a 'fof' class OSF-triggered process and its purpose is to
delete FOF files from the FOF_INPUT_DIR directory
after the files have been successfully processed by the FGSENG, ASTENG, and
SAVENG processes.
- DL_FOF Process Trigger
Input Trigger:
DL_FOF is triggered by the DELSAV process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c c c w fof
Output Trigger:
DL_FOF triggers the FOFDEL process
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c c c c fof
DL_FOF Process I/O
Input to and Output from the DL_FOF process is as follows:
INPUT:
FOF_INPUT_DIR:
tyyyydddhhmm.fof_done - FOF file to be deleted
OUTPUT:
N/A
DL_FOF Process Modes
The DL_FOF process invokes a script to initiate its work. The script that
is invoked is indicated in the 'COMMAND' mnemonic in the DELSAV resource
file. The script, and therefore the DL_FOF process executes in only the
pipeline mode.
Pipeline Mode:
xpoll -p opus_definitions_dir:your.path -r dl_fof (in task line of resource file)
where: xpoll = External Poller Process used to invoke a script to do file polling
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the DL_FOF process follows
opus_definitions_dir:your.path = path file to use
DL_FOF Process Resource File
This example fofpol.resource file is current as of OPUS build 2011.3,
December 2011.
!--------------------------------------------------------------------
!
! dl_fof.resource
!
! External poller using xpoll
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 11/15/01 44684 Goldst Created initial version
! 001 03/06/02 45016 Goldst Change XPOLL states and ENV.
! 002 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
!--------------------------------------------------------------------
PROCESS_NAME = dl_fof
TASK = <xpoll -p $PATH_FILE -r dl_fof>
DESCRIPTION = 'Deletes telemetry files'
COMMAND = eng_delete.csh
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.DL = w ! Trigger
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.DL = p ! Processing
OSF_SUCCESS.DL = c ! Completion
OSF_FAILURE.DL = f ! Failure setting
XPOLL_ERROR.DL = x ! Undefined exit status
ENV.INPATH = FOF_INPUT_DIR !
ENV.EXTENSION = .fof_done !
POLLING_TIME = 10 ! Wait (seconds) before polling for next
XPOLL_STATE.00 = OSF_SUCCESS
XPOLL_STATE.01 = OSF_FAILURE
FOFDEL Process Details
- FOFDEL Process Description
This is the second of the two processes which reside in the (DL) stage of the
FOF Pipeline. It is a 'fof' class OSF-triggered process and its purpose is to
delete 'fof' class OSFs from the pipeline after their corresponding FOF files
have been successfully processed by the FGSENG, ASTENG, and SAVENG processes.
FOFDEL is triggered by a 'c' in the (DL) stage of an OSF. When triggered, it
looks in its resource file to determine which OSFs to delete. The following
resource file mnemonics are used in making the determination:
OSF_TRIGGER1.DL = c - value of (DL) stage that triggers FOFDEL
OSF_TRIGGER1.DATA_ID = fof - class of OSF to delete
FOFDEL Process Triggers
Input Trigger:
FOFDEL is triggered by the DL_FOF Process for 'fof' class OSFs
FF FE AE SE CK FS CL DL Class
-- -- -- -- -- -- -- -- -----
c c c c c c c c fof
Output Trigger:
N/A - last pipeline process
FOFDEL Process I/O
Input to and Output from the FOFDEL process is as follows:
INPUT:
N/A
OUTPUT:
N/A
FOFDEL Process Modes
The FOFDEL process invokes an OPUS pipeline generic executable to perform
its work, i.e., the 'osfdelete' executable. This executable can only be
executed in a pipeline mode.
Pipeline Mode:
osfdelete -p opus_definitions_dir:your.path -r fofdel (in task line of resource file)
where: osfdelete = generic process used to delete an OSF from the blackboard
-p = option indicating the path file specification follows
-r = option indicating the root name of the resource file for the FOFDEL process follows
opus_definitions_dir:your.path = path file to use
FOFDEL Process Resource File
This example fofdel.resource file is current as of OPUS build 2011.3,
December 2011.
!--------------------------------------------------------------------
!
! fofdel.resource
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! PR
! DATE NUMBER User Description
! -------- ------ --------- -------------------------------------
! 10/24/01 44684 Goldst Created initial version
! 01/15/02 45016 Goldst Corrected Trigger
! 03/11/02 45016 Goldst Added OSF_TRIGGER1.DATA_ID
! 06/24/04 51412 Sherbert For PR 50437 Remove DISPLAY_ORDER
!--------------------------------------------------------------------
PROCESS_NAME = fofdel
TASK = <osfdelete -p $PATH_FILE -r fofdel>
DESCRIPTION = 'Delete FOF class OSF's'
SYSTEM = FOF
CLASS = fof
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.DL = c ! Trigger
OSF_TRIGGER1.DATA_ID = fof ! Trigger class ID
OSF_PROCESSING.DL = p ! Processing
OSF_ERROR.DL = e ! Letter to be used when there is an error
POLLING_TIME = 5 ! Wait (seconds) before polling for next
! forces values from path to be used
FOF Pipeline Database usage
Database Relations Accessed
FGSENG Process-accessed relations
ASTENG Process-accessed relations
SAVENG Process-accessed relations
Database Queries Performed
Database Relations accessed
product_eng_map Relation
This relation is used to identify the FOF engineering telemetry files that need to be converted to provide
intermediary telemetry files containing telemetry parameters needed for FGS, GSA or AST product generation.
It is used to simplify creation and collection of telemetry for processing. When a product has no
eng_ready = N flags, then the OSF for the product can be created using the product rootname.
The FOF Pipeline FGSENG and ASTENG processes use this relation
to determine if they need to extract telemetry parameters of interest from a given FOF file.
Field name type/size description
---------- --------- -----------
product_rootname C14 IPPPSSOOT for jitter or astrometry products;
GYYYYDDDHHMMSS for GS acquisition data
product_type C3 FGS for jitter, AST for astrometry, or GSA for GS
acqusition data
eng_rootname C12 TYYYYDDDHHMM, rootname of the ENG telemetry file
eng_ready C1 (Y/N), 'N' indicates the telemetry file eng_rootname is not
yet recognized as ready for this product_type. 'Y' indicates
that the processing for this product type has recognized
the presence of eng_rootname. Having separate control for
each product type simplifies initiation of processing. "
eng_telemetry Relation
This relation is used to identify the engineering telemetry parameters received from CCS that must be
extracted from FOF files and packaged into intermediary telemetry files to be used by EDPS FGS and
Astrometry pipelines.
The FOF Pipeline FGSENG and ASTENG processes use this relation
to determine what telemetry parameters are of interest and thus should be extracted from a given FOF file.
Field name type/size description
---------- --------- -----------
eng_type C3 identifies which EDPS pipeline this parameter is of interest to;
"FGS" = FGS pipeline, "AST" = Astrometry pipeline
ccs_number I4 A positive value less than 64K defined by CCS for every unique
telemetry parameter in the engineering warehouse. It is the Numeric ID
described in the definition of FOF elements (see Appendix A of HST-ICD-T2)
eng_number I4 A small positive value used in the FGS or Astrometry intermediary
telemetry files to identify individual telemetry parameters. In the
intermediary files the eng_number value is used in place of the
ccs_number value
raw_flag C1 indicates if to use the raw or EU converted value of a telemetry
parameter; "Y = only use raw value, N = use engineering value."
change_only C1 indicates to use the value of a telemetry parameter only if the value
had changed since its last occurrence; "Y = change-only item,
N = all-points item."
save_telemetry Relation
This relation is used to identify the engineering telemetry parameters received from CCS that must be
extracted from FOF files and packaged into the Save and Breathing files that get copied to the STScI
Engineering Team.
The FOF Pipeline SAVENG process uses this relation to determine what telemetry
parameters are of interest and thus should be extracted from a given FOF file.
Field name type/size description
---------- --------- -----------
save_type C6 identifies which destination file this parameter is output to;
"ENG" = Save File, "BREATH" = Breathing file
ccs_number I4 A positive value less than 64K defined by CCS for every unique
telemetry parameter in the engineering warehouse. It is the Numeric ID
described in the definition of FOF elements (see Appendix A of HST-ICD-T2)
short_mnemonic C4 the short mnemonic for the telemetry parameter. It is always of the
form of an uppercase character and a three digit number. The first
character always matches the first character of the CCS mnemonic.
Usually thie short mnemonic is the same as that used in shuttle
software (called the MS_ID).
raw_flag C1 indicates if to use the raw or EU converted value of a telemetry
parameter; "Y" = only use raw value, "N" = use engineering value
breathing_coeff Relation
This relation is used to specify the components of the average temperatures output to Breathing files
produced by the EDPS FOF pipeline SAVENG process. The temperatures in the averages are weighted by a
coefficient supplied in this table. The telemetry items are identified by their CCS number. The long and
short names of these engineering telemetry temperatures can be found by joining to the save_telemetry
table with save_type set to BREATH and matching ccs_number.
The FOF Pipeline SAVENG process uses this relation to get temperature coefficients
that are used in computing values output to Breathing files.
Field name type/size description
---------- --------- -----------
temp_num I4 A numeric ID for the average temperature in a Breathing file.
ccs_number I4 A positive value less than 64K defined by CCS for every unique
telemetry parameter in the engineering warehouse. It is the Numeric ID
described in the definition of FOF elements (see Appendix A of HST-ICD-T2)
temp_coeff R8 The normalized coefficient used to weight the average for the telemetry
parameter identified by the ccs_number. The average is the accumulation
of the product of the telemetry temperature times the coefficient.
temp_name C17 The name of the temperature average
Database Queries Performed
FGSENG Process Queries
- The FGSENG process performs the following query on the product_eng_map
relation to determine if there are any observations or guide star acquisitions (GSA) that occur during the time
frame of a given FOF file. If so, FGS and GSA related telemetry parameters will be extracted from the FOF
file and stored into an intermediary telemetry file for input to the EDPS FGS pipeline.
SELECT *
FROM product_eng_map
WHERE eng_rootname = upperFofRootName (rootname of a particular FOF file)
AND product_type = "FGS"
OR product_type = "GSA"
The FGSENG process performs the following query on the eng_telemetry
relation to determine what telemetry parameters are of interest and thus need to be extracted from a given
input FOF file and packaged into intermediary telemetry files for input to the EDPS FGS pipeline.
SELECT *
FROM eng_telemetry
WHERE eng_type = "FGS"
ASTENG Process Queries
- The ASTENG process performs the following query on the product_eng_map
relation to determine if there are any astrometry observations that occur during the time frame of a given
FOF file. If so, astrometry related telemetry parameters will be extracted from the FOF file and stored into
an intermediary astrometry telemetry file for input to the EDPS Astrometry pipeline.
SELECT *
FROM product_eng_map
WHERE eng_rootname = upperFofRootName (rootname of a particular FOF file)
AND product_type = 'AST'
The ASTENG process performs the following query on the eng_telemetry
relation to determine what telemetry parameters are of interest and thus need to be extracted from a given
input FOF file and packaged into intermediary telemetry files for input to the EDPS Astrometry pipeline.
SELECT *
FROM eng_telemetry
WHERE eng_type = "AST"
SAVENG Process Queries
- The SAVENG process performs the following query on the save_telemetry
relation to determine which telemetry parameters are of interest and thus should be extracted into Save and
Breathing files that ultimately get copied to the STScI Engineering Team.
SELECT *
FROM save_telemetry
WHERE save_type like "[BE]%" (if are generating both Save and Breathing files)
"ENG" (if are only generating Save files)
"BREATH" (if are only generating Breathing files)
The SAVENG process performs the following query on the breathing_coeff
relation to get informtion on the telemetry parameters that are output to Breathing files. The telemetry
parameters in Breathing files constitute weighted average temperatures partitioned into 6 "temperature
groups".
for each of the 6 temperature groups:
SELECT *
FROM breathing_coeff
WHERE temp_num = (1 to 6)