FGS Pipeline Reference Manual
(Version 7 –
Nov 2002)
Table of
Contents
Pipeline
Overview
Intended
Audience and Scope
This Reference Manual is intended primarily to give OPUS Operations
Personnel information necessary to understand the operation of the FGS
Pipeline. This includes information on Input and Output Products, Directories
and Files used, Database Tables used, and the Processes involved. This document
will be updated as FGS Pipeline modifications are made. This is not the case
for other Requirements and Design documents as they are and will remain static.
Pipeline Components
There
are 12 stages to the FGS pipeline and 5 classes of processing. The ENG class
has the rootname of the FGS telemetry file and the processes of this class
create the OSF for the GSA, FGS and EPC class, and it controls the deletion of
this telemetry file. The unassociated jitter file is generated in the FGS class
having an OSF that has the same rootname as the jitter file. The GSA class is
used to update the database with Guide Star acquisition data and its OSF
rootname is based on the acquisition time, The FAS class is used to create FGS
associations of jitter files and its OSF has the rootname of the association
dataset. These stages operate in more than one class and they have
class-specific processes. Each named process has its own resource file that
specifies process parameters.
Class
|
Stage 1 FP
|
Stage 2 GA
|
Stage 3 RJ
|
Stage 4 RE
|
Stage 5 GN
|
Stage 6 DC
|
ENG
|
FGSPOL
|
GAHOLD
|
RJHOLD
|
REHOLD
|
--
|
--
|
GSA
|
--
|
GSACQD
|
--
|
--
|
--
|
--
|
FGS
|
--
|
--
|
RCHECK RAWJIT
|
--
|
JITGEN
|
DC_JIT
|
FAS
|
--
|
--
|
--
|
--
|
--
|
BLDASN
|
EPC
|
--
|
--
|
--
|
RAWEPC
|
EPCGEN
|
--
|
Class
|
Stage 7 AN
|
Stage 8 RQ
|
Stage 9 RS
|
Stage 10 CL
|
Stage 11 FN
|
Stage 12 DL
|
ENG
|
--
|
--
|
--
|
--
|
--
|
DL_FGS FGSDEL
|
GSA
|
--
|
--
|
--
|
--
|
GSAFIN
|
FGSDEL
|
FGS
|
--
|
FGSREQ
|
FGSRSP
|
FGSCLN
|
FGSFIN
|
FGSDEL
|
FAS
|
FGSASN
|
FGSREQ
|
FGSRSP
|
FASCLN
|
--
|
FGSDEL
|
EPC
|
--
|
FGSREQ
|
FGSRSP
|
EPCCLN
|
EPCFIN
|
FGSDEL
|
Some processes share the same command when the processing depends only on resource file parameters. Some processes use the external polling process XPOLL to invoke a shell script or procedure. Some processes share a command or procedure with another pipeline. The following table indicates the use of shared procedures in the FGS pipeline.
Command
|
FGS processes
|
Other pipelines
|
eng_poll.csh
|
FGSPOL
|
AST
|
eng_hold.pl
|
GAHOLD,
RJHOLD, REHOLD
|
AST
|
eng_delete.csh
|
DL_FGS
|
AST
|
eng_finish.pl
|
GSAFIN, FGSFIN, EPCFIN
|
AST
|
dc_collect.pl
|
DC_JIT
|
OPUS
|
genreq
|
FGSREQ
|
AST,
OPUS
|
ingrsp
|
FGSRSP
|
AST,
OPUS
|
cleandata
|
FGSCLN,
FASCLN, EPCCLN
|
Any
|
osfdelete.exe
|
FGSDEL
|
Any
|
Pipeline Nominal Operational Description
Except for the ENG class, the stages of the FGS pipeline are sequential. When FGSPOL creates the ENG class OSF it immediately triggers the start of the stages for GSHOLD, RJHOLD and REHOLD. These processes update the database about the arrival of the telemetry file, determine the creation of the FGS or GSA class OSF, and reserve the ENG class for FGS, GSA and EPC processing completion. Depending on the order of arrival of telemetry, the GS acquisition processing may occur before or after the arrival of all the telemetry for the observation, so the raw jitter generation process depends on two OSF stages to assure that both the GS acquisition data and the telemetry files are present. The following provides a very short description of the operational control of the pipeline.
Note: Capitalization is being used for emphasis only. All process names and status values are lower case values.
· ENG class processing:
1. The FGSPOL process uses an external file poll process to detect an FGS telemetry file in the input directory. The FGSPOL process creates the ENG class OSF with a W value in the GA, RJ and DL stages to trigger processes that determine telemetry dependencies.
2. The GAHOLD, RJHOLD and REHOLD processes share a common command. They search the control database tables to identify the rootnames of the acquisition, jitter or engineering data that depend on the telemetry. If none are found, it terminates with a C in the respective GA, RJ or RE stage of the ENG OSF. Otherwise, it creates or updates the GSA, FGS or EPC class OSF using the rootname obtained from the database. If all the telemetry is present, then it inserts a W either in the GA stage of the GSA OSF, in the RJ stage of the FGS OSF, or in the RE stage of the EPC OSF, else it inserts a V in that stage. If the telemetry was required, it terminates with an R in the GA, RJ or RE stage of the ENG class OSF. The value R is changed to a C by GSAFIN, FGSFIN or EPCFIN (see below).
3. The DL_FGS process uses a generic file deletion process that is triggered by a C status in the combined GA, RJ and RE stages and by a W in the DL stage. It deletes the files matched in the resource file. On completion it set a value of C in the DL stage.
4. The FGSDEL process deletes only the OSF for this class. It is triggered by a value of C in the DL stage.
· GSA class processing:
1. The GSACQD process is started with the W trigger in the GA stage of the GSA OSF. It stores the results of the GS acquisition analysis in the database and updates or creates the FGS class OSF with a C in the GA stage. It sets a W status in the FN stage of the GSA class OSF to trigger the GSAFIN process.
2. The GSAFIN process shares the same command as the FGSFIN process. It is triggered by a W is the FN stage. It updates the completion status based on the GSA rootname. It finds all the telemetry that depends on that rootname. For every telemetry file for which all GSA processing is completed, it updates the ENG class OSF with a C in the GA stage. This process does not delete any files. On completion it sets a value of C in the DL stage,
3. The FGSDEL process deletes only the OSF for this class. It is triggered by a value of C in the DL stage.
· FGS class processing:
1. The RCHECK process is started with the W trigger in the RJ stage of the FGS OSF. It checks that science timing data is available and determines if the observation depends on GS acquisition data. If timing data is missing it sets an H (for hold) in the RJ stage. The value H may be reset to the value W by the OPUS pipeline after it updates the science timing data. If acquisition data is not needed, then it sets an S (for shove) in the RJ stage. If acquisition data is needed and it is completed, then it sets a C in the GA stage and it sets a G (for GA dependent) in the RJ stage. If the acquisition data is needed but not yet completed it sets a G in the RJ stage and RAWJIT must wait for the GSACQD process to set the C in the GS stage.
2. The RAWJIT process is started with either of two triggers. If the RCHECK process has determined that no GSA data is needed, only a value S in the RJ stage is needed. If GSA data is needed, then the trigger requires both a value G in the RJ stage and a value C in the GA stage. RAWJIT reads the telemetry files and the database to generate a file of keywords and a file of raw jitter data. When RAWJIT is complete it sets the W trigger in the GN stage for the JITGEN process.
3. The JITGEN process corrects the raw jitter data, using orbit and ephemeris data taken from the database, calculates statistics from the corrected data, and writes FITS jitter files. When JITGEN completes, it only sets its DC stage to W to trigger the DC_JIT process.
4. The DC_JIT process is a data collector for jitter associations. It uses the same DC_COLLECT script as the OPUS pipeline. The DC_JIT resource file is used to determine the status codes for the stages. If the jitter file is unassociated, the AN stage is set to U (for unassociated) and the RQ stage is set to W to trigger the FGSREQ process. Otherwise, an FAS class OSF is created or updated. If this OSF exists, the DCF number is decremented. If the decremented DCF is zero, a value of W is set in the DC stage of the FAS class OSF, else a V is set in that stage. If this OSF does not exist, the database is queried to get the number of association members. This number is decremented and used for the DCF value of the new OSF and the DC stage is set to V. The W value in the DC stage of the FAS OSF triggers the BLDASN process. For association members, the AN stage of the FGS class OSF is set to M (for member).
5. The FGSREQ process uses the standard archive request process (genreq). It is started with the W trigger in the RQ stage of the FGS class OSF. It creates an archive request that identifies the FGS dataset to the Archive. On completion, it sets the C status in the RQ stage, and a W status in the RS stage, to indicate that FGSRSP is waiting for the archive response processing.
6. The FGSRSP process uses the standard archive response process (ingrsp). It is a time polling process. It reads the response files created by the ARCHIVE. It determines the OSF affected by the response and updates their OSFs. For each archived dataset, the RS stage is set to C and the CL stage is set to W.
7. The FGSCLN deletes datasets after the FGSRSP processes the response from the ARCHIVE. It uses the new standard (cleandata) process that relies on OUTPATH values in each process of the FGS pipeline. It is triggered when the FGSRSP pipeline sets the value W in the CL stage. On completion of the cleanup, it sets a value C in the CL stage and a value W in the FN stage of the FGS OSF.
8. The FGSFIN process shares the same command as the GSAFIN process (see above). It is triggered by a W is the FN stage. It updates the completion status based on the FGS rootname and product type. It finds all the telemetry that depends on that rootname and for every telemetry file for which all FGS processing is completed, it updates the ENG class OSF with a C in the RJ stage. This process does not delete any files. On completion is sets the DL stage of the FGS OSF to C.
9. The FGSDEL process deletes only the OSF for this class. It is triggered by a C status in the DL stage.
· The FAS class processing:
1. The BLDASN is triggered by a value W in the AN stage of the FAS OSF. The process checks whether asn_association.collect_date is blank. If it is blank, it logs a message and terminates with a value H in the AN stage. The value H indicates the processing must be held up until the OPUS pipeline updates the association tables and resets the AN stage of the FAS OSF to the value W. If the database data is ready, then BLDASN uses the asn_members status data to write the FITS ASN file. Upon completion it set a value A in the AN stage to start the FGSASN process.
2. The FGSASN process is trigger by an A in the AN stage, which occurs when the association file and all the jitter files are present. FGSASN constructs the associated jitter dataset. FGSASN updates the FGS class OSF for each member. If the member was inserted into an association, then the AN stage of the FGS OSF is changed from M to I, the RQ and RS stages are set to N, and the CL stage of the FGS OSF is set to V for wavecals that are still needed, or CL is set to W to trigger FGSCLN. Otherwise for an orphan member, the AN stage is set from M to O, the RQ and RS stages are set to N, and the CL stage is set to W. At the end of the FGS class OSF updates, FGSASN updates its FAS class OSF, by setting C in the AN stage and W in the RQ stage.
3. The FGSREQ process is a standard archive request process that is used for both FGS and FAS class datasets. It is started with the W trigger in the RQ stage of the FAS class OSF. It sends the dataset identified by the OSF rootname to the Archive. On completion, it sets the C status in the RQ stage, and a W status in the RS stage, to indicate that FASRSP is waiting for the ARCHIVE pipeline.
4. The FGSRSP process is a standard archive response process that is used for both FGS and FAS class datasets. It is a time polling process. It determines the OSF affected by the response and updates their OSFs. For each archived dataset the RS stage is set to C and the CL stage is set to W.
5. The FASCLN deletes datasets after the FASRSP processes the response from DADS. It is triggered when the FASRSP pipeline sets the value W in the CL stage. On completion of the cleanup, it sets a value C in the CL stage and DL stage.
6. The FGSDEL process deletes only the OSF for this class. It is triggered by a status C in the DL stage.
· EPC class processing:
1. The RAWEPC process is triggered by a W in the RE stage of the EPC OSF. RAWEPC reads the telemetry files and the database to generate a file of raw EPC data which is a binary file of times and telemetry values. When RAWEPC is complete it sets the W trigger in the GN stage for the EPCGEN process.
2. The EPCGEN process segregates the raw data into separate FITS tables that are placed in a separate extension, one extension per telemetry mnemonic. It computes statistics for extension keywords. When EPCGEN completes, it sets the RQ stage to W to trigger the FGSREQ process.
3. The FGSREQ process uses the standard archive request process (genreq). It is started with the W trigger in the RQ stage of the EPC class OSF. It creates an archive request that identifies the EPC dataset to the Archive. On completion, it sets the C status in the RQ stage, and a W status in the RS stage, to indicate that FGSRSP is waiting for the archive response processing.
4. The FGSRSP process uses the standard archive response process (ingrsp). It is a time polling process. It reads the response files created by the ARCHIVE. It determines the OSF affected by the response and updates the OSF. For each archived dataset, the RS stage is set to C and the CL stage is set to W.
5. The EPCCLN deletes datasets after the FGSRSP processes the response from the ARCHIVE. It uses the new standard (cleandata) process that relies on OUTPATH values in each process of the FGS pipeline. It is triggered when the FGSRSP pipeline sets the value W in the CL stage. On completion of the cleanup, it sets a value C in the CL stage and a value W in the FN stage of the EPC OSF.
6. The EPCFIN process shares the same command as the GSAFIN or FGSFIN process (see above). It is triggered by a W is the FN stage. It updates the completion status based on the EPC rootname and product type. It finds all the telemetry that depends on that rootname and for every telemetry file for which all EPC processing is completed, it updates the ENG class OSF with a C in the RE stage. This process does not delete any files. On completion is sets the DL stage of the EPC OSF to C.
7. The FGSDEL process deletes only the OSF for this class. It is triggered by a C status in the DL stage.
Pipeline Directories and Files
· OPUS_HOME_DIR
Contains Process Status Files (PSF) and Process Log Files
· OPUS_DEFINITIONS_DIR
Contains Path files, Stage Files, Process Resource Files, and OMG Data Files:
o FGS.PATH_TEMPLATE
- Defines directory mnemonics used by FGS Pipeline Processes when they are run in Pipeline mode. Also defines mnemonics such as OPUS_DB, DSQUERY, and OK_TO_UPDATE_DATABASE for pipeline mode database access. It defines the paths for all the directories described below (FGS_TELEM_DIR, FGS_RAW_DIR, FGS_OUTPUT_DIR, EPC_RAW_DIR, and EPC_OUTPUT_DIR).
o NULL_PATH.TEMPLATE
- Defines directory mnemonics used by FGS Pipeline Processes when they are run in Interactive mode. Also defines mnemonics such as OPUS_DB, DSQUERY, and OK_TO_UPDATE_DATABASE for interactive mode database access
o FGS_PIPELINE.STAGE
- Defines the stages in the FGS Pipeline and the possible processes for each stage.
§ FP - FGS telemetry poll (FGSPOL)
§ GA - GS Acquisition Data (GSACQD) or GS Acquisition Hold (GAHOLD)
§ RJ - raw jitter creation (RCHECK and RAWJIT) or hold ENG for raw jitter (RJHOLD)
§ RE – raw EPC creation (RAWEPC) or hold ENG for raw EPC (REHOLD)
§ GN - generation of FITS jitter (JITGEN) or EPC files (EPCGEN)
§ DC – data collection (DC_JIT) and association table generation (BLDASN)
§ AN – association of jitter files (FGSASN)
§ RQ - Archive request (FGSREQ)
§ RS - Archive response (FGSRSP)
§ CL - archived file cleanup (FGSCLN, FASCLN or EPCCLN)
§ FN - finished product – update database (GSAFIN, FGSFIN or EPCFIN)
§
DL - ENG telemetry cleanup (DL_FGS) and OSF
delete (FGSDEL) for all classes.
o Resource files
- Defines control parameters for each process
§ FGSPOL.RESOURCE
§ GAHOLD.RESOURCE
§ GSACQD.RESOURCE
§ RJHOLD.RESOURCE
§ RCHECK.RESOURCE
§ RAWJIT.RESOURCE
§ REHOLD.RESOURCE
§ RAWEPC.RESOURCE
§ JITGEN.RESOURCE
§ EPCGEN.RESOURCE
§ DC_JIT.RESOURCE
§ BLDASN.RESOURCE
§ FGSASN.RESOURCE
§ FGSREQ.RESOURCE
§ FGSRSP.RESOURCE
§ FGSCLN.RESOURCE
§ FASCLN.RESOURCE
§ EPCCLN.RESOURCE
§ DL_FGS.RESOURCE
§ GSAFIN.RESOURCE
§ FGSFIN.RESOURCE
§ EPCFIN.RESOURCE
§ FGSDEL.RESOURCE
· FGS_TELEM_DIR
Contains FGS-specific telemetry files that are input to the pipeline. The file name has the same date stamp as the FOF file used to create the file. The file extension (FGS) is used to distinguish this telemetry from CCS .MRG data or astrometry .AST data. After FGSPOL has detected the FGS file the extension is reset to FGS_DONE.
o t<yyyydddhhmm>.fgs_done - FGS engineering telemetry file
*** The file naming convention fields used in above notations are decoded as follows:
yyyy = 4 digit year ddd = 3 digit day of year (001 - 365) hh = 2 digit hour of day (00-23) mm = 2 digit minute of hour (00-59)
· FGS_RAW_DIR
Contains raw jitter and EPC files all having the rootname of the observation (with Type = J). The keyword and raw jitter files are output from RAWJIT.
o *.jkx - keywords for jitter
o *.jrx - raw jitter binary data
· FGS_OUTPUT_DIR
Contains intermediate and product files all having the rootname of the observation (with Type = J). The FITS jitter and associated jitter files output from the JITGEN or FGSASN process.
o *.jit - observation jitter table file
o *.jwt - observation jitter table file for wavcals
o *.jif - observation jitter image file
o *.jwf - observation jitter image file for wavecals
· EPC_RAW_DIR
Contains the raw EPC files having the rootname of the observation (with Type = J). This file is output from RAWEPC process.
o
*.erx - raw EPC binary data
· EPC_OUTPUT_DIR
Contains the EPC FITS file having the rootname of the observation (with Type = J). The keyword and raw jitter files are output by RAWJIT. The FITS jitter and associated jitter files output from the JITGEN or FGSASN process. The FITS EPC file is output from the EPCGEN process. Because this rootname is shared with the FGS jitter data, the EPC files must be placed in a different directory so DADS will not get confused with an archive request that relies on dataset rootnames.
o *.epc - engineering products for calibration.
Pipeline Data Flow
External Data Sources
There are four external sources providing many types of data for the FGS pipeline.
- The SPSS system supplies the SPSS_DB tables by the FGS class to generate jitter files.
- The PDR pipeline supplies a variety of database tables uses by most FGS classes. These PDR tables include the following: MSC data obtained from Mission Schedules; ASN data for associations, qolink_sms records containing status and timing data, and CONTRL tables used to control FGS pipeline processing.
- The FOF pipeline supplies the FGS-specific telemetry files derived from CCS FOF files.
- The OPUS pipeline provides status and
timing data in the qolink_sms table and association status is found in the
asn_members table.
External Data Destinations
There are two destinations for FGS pipeline data.
- The HDA receives the engineering products (EPC), unassociated jitter files, associated jitter files, association orphans, and the ASN table describing the status of each association member.
- The AST pipeline receives database
records in the ast_jitter_kw table containing keyword names and values for
jitter keywords needed by astrometry.
Data Flow Diagram between Classes
The following diagram describes data flow using solid
lines. The dotted lines with bold names indicate system commands or OSF
triggering. OSF triggers that have double ended arrows between two classes
indicate that both classes have processes that can trigger some process in the
other class. The details of what processes are affected by OSF triggers is
provided in other data flow diagrams.
The
ENG class provides FGS telemetry to the GSA, FGS and EPC class processes by
polling for these files and determining when all telemetry is available. The
ENG class deletes an FGS telemetry file when all FGS pipeline processing is
completed for that file. The GSA class provides GS acquisition data to the FGS
classes using CONTRL tables. The FGS class provides jitter files for the FAS
class to generate associated datasets. The EPC class generates Engineering
Products for Calibration using the same engineering file used by the FGS and
GSA classes.
Data Flow Diagram for ENG Class
The FGSPOL process creates the OSF for every FGS
telemetry file supplied by the FOF pipeline. The FGSPOL process sets the
trigger for the GAHOLD, RJHOLD and REHOLD processes, at the same time. GAHOLD,
RJHOLD and REHOLD uses the CONTRL tables to determine if the telemetry should
be held for GA acquisition processing, raw jitter processing, or raw EPC
processing, respectively. If GAHOLD determines that all telemetry is ready for
a particular acquisition, it creates the GSA class OSF to start acquisition
processing. If RJHOLD determines that all telemetry is ready for a particular
jitter file, it creates or updates the FGS OSF to indicate that readiness. If
RJHOLD determines that all telemetry is ready for a particular jitter file, it
creates or updates the FGS OSF to indicate that readiness. If REHOLD uses the
CONTRL tables to determine when all telemetry is ready for a particular EPC
file, and it creates or updates the EPC OSF to indicate that readiness.
When all GSA, FGS and EPC class processing have been
completed for a particular telemetry file, then the DL_FGS process is triggered
to the FGS telemetry and the FOF telemetry file. The final process FGSDEL is
used to delete the completed ENG class OSF.
Data Flow Diagram for GSA Class
The GSACQD process gets the acquisition time range
from the CONTRL tables, reads the FGS telemetry for that time range, and
updates the CONTRL tables with acquisition results. Using CONTRL data, this
process also creates or updates the FGS class OSF that depends on the
acquisition data to indicate that this processing is completed. The GSAFIN
process is triggered by GSACQD completion. It marks the acquisition completion
in the CONTRL tables and determines if any telemetry files are no longer
required for acquisition processing. For telemetry that is no longer needed,
the ENG class OSF is updated.
Data Flow Diagram for FGS Class
The ENG class RJHOLD process triggers the RCHECK process when all the telemetry is ready. RCHECK verifies the time range and status values in qolink_sms and determines if acquisition processing is needed. The GSA class GSACQD process and RCHECK process jointly trigger RAWJIT. The RAWJIT process uses telemetry and database data to generate raw jitter files before triggering the JITGEN process. The JITGEN process uses raw jitter data and database data to generate jitter datasets and occasionally updates the astrometry jitter keyword table, ast_jitter_kw. The JITGEN process can retrigger an AST pipeline OSF that is waiting for an ast_jitter_kw update. The DC_JIT process collects jitter files for associations and triggers FGSREQ for unassociated datasets. DC_JIT creates or updates FAS class OSF for associations. The FGSREQ process sends unassociated jitter datasets to the ARCHIVE pipeline.
The FGSFIN process is triggered by the completion of
FGSREQ or the completion of processing in the FAS pipeline. FGSFIN marks the
jitter completion in the CONTRL tables and determines if any telemetry files
are no longer required for jitter processing. For telemetry that is no longer
needed, the ENG class OSF is updated. The FGSDEL process deletes the FGS OSF
after the completion of the FGSFIN process.
Data Flow Diagram for FAS Class
The BLDASN process finds the FITS ASN files provided
by OPUS pipeline and creates or updates the FAS class OSF indicating the ASN
file is found. The BLDASN status and the status provided by the FGS class
DC_JIT process, together trigger the FGSASN process when both the ASN file and
the member jitter files are ready. FGSASN triggers the FGS class FGSFIN process
for all members. After creating the associated dataset using the jitter
datasets created by the FGS class, it triggers the FGSREQ process. The FGSREQ
process sends associated datasets, orphans, and the ASN file to the ARCHIVE
pipeline. FGSREQ triggers FASCLN to delete the FAS datasets. The final stage
FGSDEL deletes the OSF after the completion of FASCLN.
Data Flow Diagram for EPC Class
The ENG pipeline triggers the RAWEPC process when all
the engineering files are ready. The RAWEPC reads the engineering data within
the time range found in the qolink_sms table to create the raw EPC file. Then
it triggers the EPCGEN process. This process converts the raw EPC file into a
EPC dataset in FITS format. The FGSREQ process generates the archive request to
be sent to the ARCHIVE in the EPC archive class. The FGSRSP is a time polling
process that looks for the archive response file. The FGSRSP triggers EPCCLN to
delete the raw EPC file and the archived EPC FITS file. Then the EPCFIN process
is triggered to update the product_status table and update the RE stage of the
ENG class OSF when all the EPC products for that engineering file are complete.
The FGSDEL process runs in all the FGS pipelines. It deletes the completed EPC class OSF.
Pipeline Processes
Pipeline Processes Overview
As indicated in the Overview section, the FGS Pipeline is composed of 9 stages having the following 18 processes in 4 classes.
- FGSPOL - (ENG class) polls FGS telemetry and creates the ENG class OSF
- GAHOLD - (ENG class) updates telemetry status, creates GSA class OSF based on telemetry dependency and reserves telemetry until completion of all GSA dependencies.
- RJHOLD - (ENG class) updates telemetry status, creates FGS class OSF based on telemetry dependency, and reserves telemetry until completion of all FGS dependencies.
- REHOLD - (ENG class) updates telemetry status, creates EPC class OSF based on telemetry dependency, and reserves telemetry until completion of all EPC dependencies.
- DL_FGS - (ENG class) deletes telemetry files.
- GSACQD - (GSA class) process GS acquisition telemetry and updates database with acquisition data
- GSAFIN - (GSA class) mark GS acquisition finished and determine if telemetry still needed for GSA processing
- RCHECK - (FGS class) check for RAWJIT dependency on GSACQD processing.
- RAWJIT - (FGS class) generates keywords and raw jitter data from telemetry files and GSACQD data
- JITGEN - (FGS class) generates jitter table extension and image extension FITS files
- DC_JIT - (FGS class) checks for associations, updates or creates FAS class OSF.
- FGSREQ - (FGS class) request interface to HDA for unassociated jitter files
- FGSRSP – (FGS class) response interface to HAD for unassociated jitter files.
- FGSCLN - (FGS class) delete FGS files after archiving
- FGSFIN - (FGS class) mark FGS jitter finished and determine if telemetry still needed for FGS processing
- BLDASN - (FAS class) creates the FITS ASN table from the database
- FGSASN - (FAS class) creates jitter association datasets
- FASCLN - (FAS class) delete association files after archiving
- RAWEPC - (EPC class) generates raw engineering product data from telemetry files for times in qolink_sms
- EPCGEN - (EPC class) generates EPC FITS table extensions and formats EPC product files computing all keywords derived from telemetry values.
- EPCCLN - (EPC class) delete raw and EPC product file after archiving
- EPCFIN - (EPC class) mark EPC product finished and determine if telemetry still needed for EPC processing
- FGSDEL - (All classes) delete OSF in class that triggered the process.
Pipeline Processes Details
- FGSPOL Process Description
- FGSPOL Process Triggers
- FGSPOL Process I/O
- FGSPOL Process Error Handling
- FGSPOL Process Modes
- GAHOLD Process Description
- GAHOLD Process Triggers
- GAHOLD Process I/O
- GAHOLD Process Error Handling
- GAHOLD Process Modes
RJHOLD
- RJHOLD Process Description
- RJHOLD Process Triggers
- RJHOLD Process I/O
- RJHOLD Process Error Handling
- RJHOLD Process Modes
REHOLD
- REHOLD Process Description
- REHOLD Process Triggers
- REHOLD Process I/O
- REHOLD Process Error Handling
- REHOLD Process Modes
- RJHOLD Process Description
- RJHOLD Process Triggers
- RJHOLD Process I/O
- RJHOLD Process Error Handling
- RJHOLD Process Modes
- DL_FGS Process Description
- DL_FGS Process Triggers
- DL_FGS Process I/O
- DL_FGS Process Error Handling
- DL_FGS Process Modes
- GSACQD Process Description
- GSACQD Process Triggers
- GSACQD Process I/O
- GSACQD Process Error Handling
- GSACQD Process Modes
- GSAFIN Process Description
- GSAFIN Process Triggers
- GSAFIN Process I/O
- GSAFIN Process Error Handling
- GSAFIN Process Modes
- RCHECK Process Description
- RCHECK Process Triggers
- RCHECK Process I/O
- RCHECK Process Error Handling
- RCHECK Process Modes
- RAWJIT Process Description
- RAWJIT Process Triggers
- RAWJIT Process I/O
- RAWJIT Process Error Handling
- RAWJIT Process Modes
- JITGEN Process Description
- JITGEN Process Triggers
- JITGEN Process I/O
- JITGEN Process Error Handling
- JITGEN Process Modes
- DC_JIT Process Description
- DC_JIT Process Triggers
- DC_JIT Process I/O
- DC_JIT Process Error Handling
- DC_JIT Process Modes
- FGSREQ Process Description
- FGSREQ Process Triggers
- FGSREQ Process I/O
- FGSREQ Process Error Handling
- FGSREQ Process Modes
- FGSRSP Process Description
- FGSRSP Process Triggers
- FGSRSP Process I/O
- FGSRSP Process Error Handling
- FGSRSP Process Modes
FGSCLN
- FGSCLN Process Description
- FGSCLN Process Triggers
- FGSCLN Process I/O
- FGSCLN Process Error Handling
- FGSCLN Process Modes
- FGSFIN Process Description
- FGSFIN Process Triggers
- FGSFIN Process I/O
- FGSFIN Process Error Handling
- FGSFIN Process Modes
- BLDASN Process Description
- BLDASN Process Triggers
- BLDASN Process I/O
- BLDASN Process Error Handling
- BLDASN Process Modes
- FGSASN Process Description
- FGSASN Process Triggers
- FGSASN Process I/O
- FGSASN Process Error Handling
- FGSASN Process Modes
- FASCLN Process Description
- FASCLN Process Triggers
- FASCLN Process I/O
- FASCLN Process Error Handling
- FASCLN Process Modes
- RAWEPC Process Description
- RAWEPC Process Triggers
- RAWEPC Process I/O
- RAWEPC Process Error Handling
- RAWEPC Process Modes
- EPCGEN Process Description
- EPCGEN Process Triggers
- EPCGEN Process I/O
- EPCGEN Process Error Handling
- EPCGEN Process Modes
- EPCCLN Process Description
- EPCCLN Process Triggers
- EPCCLN Process I/O
- EPCCLN Process Error Handling
- EPCCLN Process Modes
- EPCFIN Process Description
- EPCFIN Process Triggers
- EPCFIN Process I/O
- EPCFIN Process Error Handling
- EPCFIN Process Modes
- FGSDEL Process Description
- FGSDEL Process Triggers
- FGSDEL Process I/O
- FGSDEL Process Error Handling
- FGSDEL Process Modes
FGSPOL (FP) Process
The script that executes this process, eng_poll, is shared with the AST pipeline. This is an external polling process that is triggered by a file. The resource file is used to distinguish the processing between the two pipelines. For FGSPOL processing, the resource keyword FILE_OBJECT1 will be used to find a telemetry file in the FILE_DIRECTORY1, where it will be renamed using the FILE_PROCESSING addition to the extension. The class of the resource file is used to qualify OSF processing.
1. If the OSF for this eng file exists, then its FP column is updated using the resource ENV variable OSF_DUPLICATE_STATUS, and the extension FILE_DUPLICATE is appended to the file, and a message about the renaming is logged.
2. Else, a new OSF is created for this eng file, with all status values supplied from the resource ENV variable OSF_SUCCESS, the extension FILE_SUCCESS is appended to the file, and a message about the renaming is logged.
3. If the file cannot be renamed, then an error message is logged, and the OSF is updated with a status value from the resource ENV variable OSF_FAILURE.
- Input Trigger
This is a file polling process that looks for FGS telemetry files.
- Output Trigger
It sets three parallel triggers when it creates the ENG class OSF.
(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
W |
W |
N |
W |
ENG |
An existing OSF indicates a duplicate. In this example there are outstanding jitter and product files from the previous telemetry file, but the FP column is set to D without changing the status of the other columns.
(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
D |
C |
R |
R |
N |
W |
ENG |
None.
·
FGSPOL Process Error
Handling
If
the file cannot be renamed a message is logged and the OSF is set to F.
The FGSPOL process is used only in pipeline mode.
- Pipeline Mode
xpoll -p $PATH_FILE -r fgspol command: eng_poll.cshwhere: xpoll = command name
-p = option for path $PATH_FILE = env for path file-r = option for resource
fgspol = name of resource file GAHOLD (GA) Process
This process is OSF-triggered by a W in the GA stage of the ENG class. The program that executes this process, eng_HOLD, is shared with the AST pipeline and with two FGS-pipeline processes. The resource file is used to distinguish the processing between the different stages. For GAHOLD processing, the following steps are performed for each ENG class OSF.
1. The resource keyword TARGET_CLASS is used to get the target process class name, that will be used in queries and OSF updates, below.
2. The target class and the OSF rootname are used to query OPUS_DB to determine the product root names of the not-yet-completed products that depend on this telemetry.
3. If no products are found, the ENG OSF is updated with the status OSF_NO_PRODUCTS, a message is logged, and the process completes normally. Otherwise continue.
4. For each product rootname queried in step 2, perform the following steps:
5. An OPUS_DB query sets the ENG_READY flag in the record for this class, product and telemetry file.
6. An OPUS_DB query determines if all the required telemetry for this product is ENG_READY.
7. If all the telemetry is ready, then the target-class OSF for the product rootname is updated using the resource keyword UPDATE_OSF_ENG_READY, else it is updated using the resource keyword UPDATE_OSF_ENG_WAITING.
8. A message is logged on the status of the product rootname.
- Input Trigger
A W must be present in the GA stage. The value in the RJ or RE stage is ignored.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
W |
W |
N |
W |
ENG |
- Output Trigger
For the case when no uncompleted acquisitions are found, then no new GSA class OSF is created.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
C |
W |
W |
N |
W |
ENG |
For newly created GSA-class OSF, when the telemetry is incomplete for GSACQD the following GSA OSF is created and the ENG GA status is R for reserved. (FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(FN) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
------ |
DL_FGS |
ENG |
C |
R |
W |
W |
N |
N |
W |
ENG |
------ |
GSACQD |
------ |
------ |
------ |
GSAFIN |
------ |
GSA |
N |
V |
- |
- |
- |
- |
- |
GSA |
For new or updated GSA-class OSF, when the telemetry is complete for GSACQD, then the following status is set to trigger GSA class GSACQD processing (FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(FN) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
------ |
DL_FGS |
ENG |
C |
R |
W |
W |
N |
N |
V |
ENG |
------ |
GSACQD |
------ |
------ |
------ |
GSAFIN |
------ |
GSA |
N |
W |
- |
- |
- |
- |
- |
GSA |
GAHOLD updates the product_eng_map table and queries the same table.
· GAHOLD Process Error Handling
If
GAHOLD fails to update the database or cannot create or update a GSA class OSF,
it logs the problem and sets the ENG class OSF to F.
The GAHOLD process can be used both in pipeline mode and in interactive mode.
o Interactive Mode
eng_hold eng_rootname -c GSA (typed into command line)where: eng_hold = command name
eng_rootname = Tyyyydddhhmm of the FGS telemetry file
-c = option for target class GSA = class for GAHOLD- Pipeline Mode
xpoll -p $PATH_FILE -r gahold command: eng_hold.pl where: eng_hold = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource gahold = name of resource file RJHOLD (RJ) Process
This
process is OSF-triggered by a W in the RJ stage of the ENG class. The program
that executes this process, eng_HOLD,
is shared with the AST pipeline and with two FGS-pipeline processes. The resource file is used to
distinguish the processing between the different stages. For RJHOLD processing,
the same steps are performed as described in GAHOLD Process Description.
- Input Trigger
A W must be present in the RJ stage. The value in the GA or RE stage is ignored.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
W |
W |
N |
W |
ENG |
- Output Trigger
For the case when no uncompleted jitter exposures are found, then no new FGS class OSF is created.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
C |
W |
N |
W |
ENG |
For newly created FGS-class OSF, when the telemetry is incomplete for RCHECK and the GS Acquisition processes is not yet done (because it would have created the OSF), then the following FGS OSF is created and the ENG RJ status is R for reserved. (FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
R |
W |
N |
W |
ENG |
------ |
------ |
RCHECK |
------ |
------ |
FGSDEL |
FGS |
N |
V |
V |
- |
- |
- |
FGS |
For new or updated FGS-class OSF, when the telemetry is complete for RCHECK, but the GS acquisition process is not yet done, then the following status is set for FGS class RCHECK processing.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
R |
W |
N |
W |
ENG |
------ |
------ |
RCHECK |
------ |
------ |
FGSDEL |
FGS |
N |
V |
W |
- |
- |
- |
FGS |
RJHOLD updates the product_eng_map table and queries the same table.
· RJHOLD Process Error Handling
If
RJHOLD fails to update the database or cannot create or update an FGS class
OSF, it logs the problem and sets the ENG class OSF to F.
The
RJHOLD process can be used both in pipeline mode and in interactive mode.
o Interactive Mode
eng_hold eng_rootname -c FGSwhere: eng_hold = command name
eng_rootname = Tyyyydddhhmm of the FGS telemetry file
-c = option for target classFGS = class for jitter
- Pipeline Mode
xpoll -p $PATH_FILE -r rjhold command: eng_hold.pl where: xpoll = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource rjhold = name of resource fileREHOLD (RE) Process
This process is OSF-triggered by a W in the RE stage of the ENG class. The program that executes this process, eng_HOLD, is shared with the AST pipeline and with three FGS-pipeline processes. The resource file is used to distinguish the processing between the different stages. For REHOLD processing, the same steps are performed as described in GAHOLD Process Description above.
- Input Trigger
A W must be present in the RE stage. The status of any other stage is ignored.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
W |
W |
N |
W |
ENG |
- Output Trigger
For the case when no uncompleted products are found, then no new EPC class OSF is created, and this stage is set to the complete status.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
W |
W |
C |
N |
W |
ENG |
For newly created EPC-class OSF, when the telemetry is incomplete for RAWEPC the following EPC OSF is created and the ENG RE status is R for reserved. (FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(FN) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
------ |
DL_FGS |
ENG |
C |
W |
W |
R |
N |
N |
W |
ENG |
------ |
------ |
----- |
RAWEPC |
----- |
EPCFIN |
------ |
EPC |
N |
N |
N |
V |
- |
- |
- |
EPC |
For new or updated EPC-class OSF, when the telemetry is complete for RAWEPC, then the following status is set to trigger EPC class RAWEPC processing (FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(FN) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
------ |
DL_FGS |
ENG |
C |
R |
W |
R |
N |
N |
W |
ENG |
------ |
------ |
----- |
RAWEPC |
----- |
EPCFIN |
------ |
EPC |
N |
N |
N |
W |
- |
- |
- |
EPC |
REHOLD updates the product_eng_map table and queries the same table.
· REHOLD Process Error Handling
If
REHOLD fails to update the database or cannot create or update an EPC class
OSF, it logs the problem and sets the ENG class OSF to F.
The REHOLD process can be used both in pipeline mode and in interactive mode.
o Interactive Mode
eng_hold eng_rootname -c epc (typed into command line)where: eng_hold = command name
eng_rootname = Tyyyydddhhmm of the FGS telemetry file
-c = option for target class epc = class for REHOLD
- Pipeline Mode
xpoll -p $PATH_FILE -r rehold command: eng_hold.pl where: eng_hold = command name
-p = option for path $PATH_FILE = env for path file-r = option for resource
rehold = name of resource file DL_FGS (DL) Process
This process is OSF-triggered by a C status in the GA, RJ and RE stages of the ENG class, plus the W status in the DL stage. The program that executes this process, eng_delete, is shared with the AST pipeline. The resource file is used to distinguish the processing between the different types of telemetry. For DL_FGS processing, the following steps are performed.
1. The
resource keywords are obtained to determine the input path and extension of the
telemetry to be deleted
2. The
file specification is generated using the path, the OSF rootname, and the
extension.
3. The
file is deleted and a log message generated on either the success or the
failure of the deletion of the named file.
- Input Trigger
A C must be present in all three “hold” stages as well the W in the DL stage. If the operator changes the value in the DL stage this process will not be triggered.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
C |
C |
C |
N |
W |
ENG |
- Output Trigger
The DL status is set to C for OSF deletion.(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
C |
C |
C |
N |
C |
ENG |
The DL_FGS process deletes files as specified in resource keywords.
· DL_FGS Process Error Handling
If
the file to be deleted does not exist a warning message is generated but the
process completes normally. If the file cannot be deleted an error message is
generated and the DL stage is set to F.
The
DL_FGS process is only used in pipeline mode. Manual deletion of files is
easily done in the shell.
- Pipeline Mode
xpoll -p $PATH_FILE -r dl_fgs command: eng_delete.cshwhere: xpoll = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource dl_fgs = name of resource file GSACQD (GA) Process
The GSACQD process is triggered by GAHOLD using the GA status of the GSA OSF, after GAHOLD finds all the telemetry needed for the generation of GS acquisition data. A GAHOLD status of R will prevent the telemetry from being deleted until successful completion of the GSACQD process. The results of GSACQD processing are inserted into a database table using a record indexed by the rootname of the GSA OSF. GSACQD uses the database to identify the telemetry items needed for its analysis and to obtain planned acquisition data. The acquisition data generated by this process reports any variance of the actual acquisition from the planned acquisition. The data is required for the RAWJIT processing in the FGS class.
- Input Trigger
The process checks only the W status in the GA stage. If the telemetry is incomplete the value will be V instead.
(FP) |
(GA) |
(Other) |
(FN) |
(DL) |
Class |
------ |
GSACQD |
----- |
GSAFIN |
FGSDEL |
GSA |
N |
W |
- |
- |
- |
GSA |
- Output Trigger
On successful completion the GSAFIN process is triggered.
(FP) |
(GA) |
(Other) |
(FN) |
(DL) |
Class |
------ |
GSACQD |
----- |
GSAFIN |
FGSDEL |
GSA |
N |
C |
N |
W |
- |
GSA |
After storing the GS acquisition data, the FGS OSF for each dependent jitter file is updated or created with a value C in the GA stage. If it is created, the RJ stage is left null to indicate that no raw jitter telemetry has been detected by RJHOLD. If the OSF is updated, then the RJ stage keeps its original status.
(FP) |
(GA) |
(RJ) |
(Other) |
Class |
------ |
------ |
RAWJIT |
------ |
FGS |
N |
C |
- |
- |
FGS |
N |
C |
V |
- |
FGS |
N |
C |
W |
- |
FGS |
The GSACQD process queries the gsa_data table to determine the time range of the telemetry to be processed. It reads the FGS engineering files to determine the acquisition status. It updates the gsa_data table with the results.
· GSACQD Process Error Handling
Telemetry
gaps will be logged but they do not cause an error condition. Failure to read a
telemetry file or update the database will cause an error message to be logged
and the GA status for the GSA class OSF will be set to F.
The GSACQD process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
gsacqd gsa_rootname (typed into command line)where: gsacqd = command name
gsa_rootname = Gyyyydddhhmmss of the gsa_data relation
- Pipeline Mode
gsacqd -p $PATH_FILE -r gsacqd where: gsacqd = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource gsacqd = name of resource file GSAFIN (DL) Process
This process is run in the FGS-pipeline Clean stage (DL) when it is triggered by a status value W. The program that executes this process, eng_finish, is shared with the AST pipeline and with two FGS-pipeline processes. The resource file is used to distinguish the processing between the different classes. For GSAFIN processing, the following steps are performed for each GSA class OSF.
1. The resource keyword CLASS is used to get the OSF class, which will be used in queries, below.
2. The class and the OSF rootname are used to query OPUS_DB to set the complete_flag toY.
3. A query is made to find all the telemetry-rootnames for this class that are required for this OSF rootname and have no records remaining with complete_flag set to N.
4. If no completed telemetry-rootnames are found, a message is logged, and the process completes normally. Otherwise continue.
5. From the resource file get the value of ENG_STEP to determine which stage to update.
6. For every telemetry-rootname queried in step 3, update the ENG_STEP of the ENG class OSF with the status C. Log a message for each update.
- Input Trigger
The process checks only the W status in the FN stage.
(FP) |
(GA) |
(Other) |
(FN) |
(DL) |
Class |
------ |
GSACQD |
----- |
GSAFIN |
FGSDEL |
GSA |
N |
C |
N |
W |
- |
GSA |
- Output Trigger
On successful completion the last stage of this class is marked for deletion by the FGSDEL process.
(FP) |
(GA) |
(Other) |
(FN) |
(DL) |
Class |
------ |
GSACQD |
----- |
GSAFIN |
FGSDEL |
GSA |
N |
C |
N |
C |
C |
GSA |
For each FGS telemetry file for which all GS acquisition processing is complete, the ENG OSF is updated as follows, changing the GA status from value R to C.
(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
C |
R |
R |
N |
W |
ENG |
This process uses database queries to product_status and log messages.
· GSAFIN Process Error Handling
A failure is reported as a log message and an F is set in the FN stage of the GSA OSF. The failure would involve some database problem or a failure to find or update an OSF.
The GSAFIN process can be executed only in the pipeline. Automatic cleanup is not needed in an interactive situation.
- Pipeline Mode
xpoll -p $PATH_FILE -r gsafin command: eng_finish.plwhere: xpoll = command name
-p = option for path $PATH_FILE = env for path file -r = option for resourcegsafin = name of resource file
RCHECK (RJ) Process
The RCHECK is an external OSF polling process that uses the command rawjit_check.pl. It is used to determine if RAWJIT should be run now, hold for OPUS, or wait for GSA data. If the database already has the acquisition data the GSACQD process may not be triggered and it would not update the GS stage of the FGS OSF. Some observations do not require GS acquisition data. Some observations have delayed or missing science data. RCHECK uses the OSF rootname and links to qolink_sms through the product_link table to get the science time_type and the science status. It also checks the jitter_evt_map and the product_status tables to determine if GS acquisition data is needed or complete. RCHECK sets OSF stage status values, as follows.
1. If the science status has the value N, the RCHECK procedure ends with a NO_SCIENCE condition that sets an N in the GA, RJ, RE, GN, AN and AR stage, and sets a W in the FN stage, to release the telemetry for this dataset.
2. If the RAWJIT processing does not need GS acquisition data, the RCHECK procedure ends with a NO_GSA _REQUIRED condition that sets an N in the GA stage and an S in the RJ stage.
3. If the science time_type is P for planned, the RCHECK procedure ends with a HOLD_FOR_SCIENCE condition that sets an H in the RJ stage, which will indicate to the OPUS pipeline that this OSF stage needs to be restarted after it completes its science processing.
4. If the RAWJIT processing needs the GS acquisition data but it is already completed, the procedure ends with a GSA_READY condition that sets a C in the GA stage and a G in the RJ stage.
5. If the RAWJIT processing needs the GS acquisition data but it is not yet marked completed, the procedure ends with a GSA_WAIT conditions that sets a G in the RJ stage.
- Input Trigger
The RCHECK process is triggered by the W in the RJ stage that indicates that RJHOLD has found all the required telemetry files.
(FP) |
(GA) |
(RJ) |
(Other) |
(FN) |
(DL) |
Class |
------ |
------ |
RCHECK |
------ |
FGSFIN |
FGSDEL |
FGS |
- |
- |
W |
- |
- |
- |
FGS |
- Output Trigger
The following possible conditions have been described above.
(FP) |
(GA) |
(RJ) |
(Other) |
(FN) |
(DL) |
Class |
------ |
------ |
RCHECK |
------ |
FGSFIN |
FGSDEL |
FGS |
N |
C |
G |
- |
- |
- |
FGS |
N |
N |
S |
- |
- |
- |
FGS |
N |
V |
G |
- |
- |
- |
FGS |
N |
N |
N |
N |
W |
- |
FGS |
If the science data processing was delayed so that the observation time range is not available, the RJ status will be set to H and the OPUS pipeline will restart the RCHECK process with a W status after updating the database. The operator can manually reset the RJ status to W to retrigger the RCHECK process so it checks the database again. Possible hold states for the FGS class OSF are shown in the following table.
(FP) |
(GA) |
(RJ) |
(Other) |
(FN) |
(DL) |
Class |
------ |
------ |
RCHECK |
------ |
FGSFIN |
FGSDEL |
FGS |
N |
C |
H |
- |
- |
- |
FGS |
N |
V |
H |
- |
- |
- |
FGS |
The RCHECK process reads various OPUS_DB tables. It logs informational records on the condition of the FGS processing.
· RCHECK Process Error Handling
If
required database records are missing, an error message is logged and the RJ
stage of the FGS OSF is set to F.
The RCHECK process is used for control flow in a pipeline. It is not used interactively.
- Pipeline Mode
xpoll -p $PATH_FILE -r rcheck command: rawjit_check.plwhere: xpoll = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource rcheck = name of resource file
RAWJIT (RJ) Process
The RAWJIT process is triggered when both the telemetry for the jitter dataset and the GS acquisition data are available or when the telemetry is available and the GS acquisition data is determined by RCHECK to be unneeded. The results of RAWJIT are written to two files: a keyword file and a binary file for raw jitter data having a record for every telemetry frame during the time range found in the database table qolink_sms.
- Input Trigger
The RAWJIT process has two triggers. If RCHECK has determined that no GS Acquisition data is needed, then RAWJIT is triggered by the value S in the RJ stage. Otherwise RCHECK puts a G is the RJ stage and RAWJIT is triggered by both the G status and a C status in the GA stage that indicates that the GS acquisition data has been stored in the database.
(FP) |
(GA) |
(RJ) |
(RE) |
(GN) |
(Other) |
Class |
------ |
------ |
RAWJIT |
------ |
JITGEN |
------ |
FGS |
N |
N |
S |
- |
- |
- |
FGS |
N |
C |
G |
- |
- |
- |
FGS |
- Output Trigger
On successful completion, the trigger for the JITGEN process is set with a value N in the RE stage and a value W in the GN stage.
(FP) |
(GA) |
(RJ) |
(RE) |
(GN) |
(Other) |
Class |
------ |
------ |
RAWJIT |
------ |
JITGEN |
------ |
FGS |
N |
N |
C |
N |
W |
- |
FGS |
N |
C |
C |
N |
W |
- |
FGS |
The RAWJIT process reads database tables and the FGS engineering telemetry. It writes log messages, the jitter keyword file, and the raw jitter file.
· RAWJIT Process Error Handling
Telemetry
gaps will be logged but they do not cause an error condition. Failure to read a
telemetry file or missing database records will cause an error message to be
logged and the RJ status for the FGS class OSF will be set to F.
The RAWJIT process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
rawjit rootname (typed into command line) where: rawjit = command namerootname = IPPPSSOOT of the rawjit dataset
EXAMPLE: To run RAWJIT with JKX and RGN files having the rootname U3450101M from the pipeline:rawjit u3450101m
- Pipeline Mode
rawjit -p $PATH_FILE -r rawjit where: rawjit = command name
-p = option for path $PATH_FILE = env for path file -r = option for resourcerawjit = name of resource file
JITGEN (GN) Process
JITGEN is an FGS-Class, OSF-triggered process that generates the JIF and JIT FITS files that are also called jitter files. Each OSF controls the generation of one dataset of jitter files. When an FGS Class OSF is set to W in the GN stage, the JITGEN process is triggered to perform the following actions:
- Construct the JKX and RGN file specification from the OSF rootname or from the interactive filename using the directory specified in the INPATH keyword.
- Reads all the keyword values in the JKX files and uses these values to process all the data in the raw jitter file (RGN).
- Using the rootname in the JKX to access related data in the database and for the time range of the raw jitter data extracts HST orbit parameters. It also reads the sun and moon ephemeris data. Position data is interpolated for each jitter time.
- Calculates velocity aberration corrections to raw jitter data and computes auxiliary pointing data for additional keywords or jitter table fields.
- Generate the FITS table and FITS image files either with extensions ".jit" and ".jif" or with extensions ".jwt" and ".jwf" for STIS wavecals identified by the TARGNAME keyword.
- Set the GN stage of the FGS Class OSF to the value V to trigger DC_JIT, or set the GN stage to the value F to indicate a processing failure has occurred.
- Input Trigger
Triggered by completion of RAWJIT processing:(GA) |
(RJ) |
(RE) |
(GN) |
(DC) |
(Other) |
Class |
------ |
RAWJIT |
------ |
JITGEN |
DC_JIT |
------ |
FGS |
N |
C |
N |
W |
- |
FGS |
|
C |
C |
N |
W |
- |
FGS |
- Output Trigger
The next step is a data collector DC_JIT for jitter associations. It is looking for a W in the DC status.(GA) |
(RJ) |
(RE) |
(GN) |
(DC) |
(Other) |
Class |
------ |
RAWJIT |
------ |
JITGEN |
DC_JIT |
------ |
FGS |
N |
C |
N |
C |
W |
- |
FGS |
C |
C |
N |
C |
W |
- |
FGS |
The JITGEN process reads the FITS keyword definitions from the database. It reads keyword data from the JKX file created by RAWJIT. It uses the JKX data to get the rootname of the RGN file that contains the raw jitter data to be processed into jitter FITS files. All errors while creating the FITS files are echoed to the JITGEN Process Log file.
· JITGEN Process Error Handling
Most errors that occur in JITGEN processing will cause the JITGEN (GN) stage in the FGS Class OSF to be set to "F". Error messages are also posted to the JITGEN Process Log File in the OPUS_HOME_DIR
The JITGEN process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
jitgen rootname (typed into command line) where: jitgen = command name rootname = IPPPSSOOT of the jitter dataset - Pipeline Mode
jitgen -p $PATH_FILE -r jitgen where: jitgen = command name
-p = option for path $PATH_FILE = env for path file -r = option for resourcejitgen = name of resource file
DC_JIT (DC) Process
The DC_JIT is an external OSF polling process. The resource file specifies an existing Perl procedure, dc_collect.pl, as the command to be used. The resource file sets the environmental variables ASN_CLASS to FAS, ASN_STEP to AN, MEM_VAIT to V and ASN_VAIT to W. It must set the environmental variable, OTFR_WORLD to TRUE, so that this procedure expects already collected associations, and so that it ignores members marked as orphans that are missing. It must set the environmental variable WRITE_TRAILER to FALSE because the FGS pipeline does not use a trailer file. It can return three completion values to XPOLL that are defined in the resource file. They are OSF_ERROR, OSF_MEMBER, and OSF_SINGLE. For the rootname of the FGS class OSF, the following steps are taken.
1. The process queries the database to find all the associations for which this rootname is a collected member.
2. If no associations are found it returns an OSF_SINGLE value so that the XPOLL sets a U for uncollected in the AN stage of the FGS OSF and sets W is the RQ stage. Otherwise it processes each association in the following steps.
3. If the association exists the DCF number is obtained.
4. If it does not exist, the database is queried to determine the number of collected members, and the ASN_CLASS OSF is created with the ASN_STEP set to MEM_VAIT and the DCF number set to the number of collected members.
5. The DCF number from the old or new OSF is decremented and tested for zero.
6. If it is zero the collection is complete, so the ASN_CLASS OSF is updated to set the ASN_STEP to ASN_VAIT and the DCF set to zero.
7. Otherwise the ASN_OSF is updated to set the DCF to the decremented value.
8. The value OSF_MEMBER is returned so that XPOLL sets the AN stage to M indicting a member status that will be updated by FGSASN.
- Input Trigger
This is an external OSF polling process that looks for a W is the AN stage of the FGS OSF.(GN) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
JITGEN |
DC_JIT |
------ |
FGSREQ |
------ |
FGS |
C |
W |
- |
- |
- |
FGS |
- Output Trigger
If the OSF_SINGLE status is returned, the unassociated FGS OSF gets a U, in the AN stage and RQ stage is set to W to send the jitter to the archive.
(GN) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
JITGEN |
DC_JIT |
------ |
FGSREQ |
------ |
FGS |
C |
C |
U |
W |
- |
FGS |
If the OSF_MEMBER status is returned, the member FGS OSF gets an M, in the AN stage. This AN status will be updated again by FGSASN to either a status value I for included or O for orphan.
(GN) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
JITGEN |
DC_JIT |
------ |
FGSREQ |
------ |
FGS |
C |
C |
M |
- |
- |
FGS |
When it finds a member, it updates or creates the FAS OSF to indicate the number of remaining members to be collected. If the collection is incomplete, then the AN stage has a value V. Otherwise, the FAS class DC stage is set to W to trigger the BLDASN process. The first three stages are not used in the FAS class and their status values are always set to N.
(Other) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
------ |
BLDASN |
FGSASN |
FGSREQ |
------ |
FAS |
N |
V |
- |
- |
- |
FAS |
N |
W |
- |
- |
- |
FAS |
The DC_JIT process queries the asn_members table to determine the initial count for the association members. After the FAS class OSF is created, it only tests and updates that OSF.
· DC_JIT Process Error Handling
The DC_JIT process logs a message and set an F status in the FGS class OSF if it cannot test, create or update the FAS class OSF.
The DC_JIT process can be run only in the pipeline.
- Pipeline Mode
xpoll -p $PATH_FILE -r dc_jit command: dc_collect.plwhere: xpoll = external poller
-p = option for path $PATH_FILE = env for path file -r = option for resource dc_jit = name of resource file
FGSREQ (RQ) Process
· FGSREQ Process Description
The FGSREQ process, using the genreq command, submits a dataset to the ARCHIVE pipeline. The resource file is used to specify parameters particular to the FGS pipeline, including FGS class, FAS class and EPC class requests. The ARCHIVE will read the request file and write a response file containing the archive status.
·
FGSREQ Process Triggers
- Input Trigger
The FGSREQ process is an OSF polling process that is triggered for FGS pipeline datasets to be sent to the archive. A status of W must be set in the RQ stage of the OSF.
(GN) |
(DC) |
(AN) |
(RQ) |
(RS) |
(Other) |
Class |
JITGEN |
DC_JIT |
----- |
FGSREQ |
FGSRSP |
------ |
FGS |
C |
C |
U |
W |
- |
- |
FGS |
C |
C |
O |
W |
- |
- |
FGS |
- Output Trigger
After successfully writing the request file and updating the database, the RQ status is set to C, and the RS stage is set to W, while waiting for the ARCHIVE response file.
(GN) |
(DC) |
(AN) |
(RQ) |
(RS) |
(Other) |
Class |
JITGEN |
DC_JIT |
----- |
FGSREQ |
FGSRSP |
------ |
FGS |
C |
C |
U |
C |
W |
- |
FGS |
C |
C |
O |
C |
W |
- |
FGS |
- FGSREQ Process I/O
The FGSREQ process writes a request file into the directory specified in the resource file to be used for archive requests.
· FGSREQ Process Error Handling
The FGSREQ only reports errors concerning validity of the dataset to be sent to the ARCHIVE. Any failure by FGSREQ will generate an F in the RQ stage for the FGS class OSF.
·
FGSREQ Process Modes
The FGSREQ process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
genreq rootname -p path_name -r fgsreq (typed into command line)where: genreq = command name
rootname = jitter IPPPSSOOT
-p = option for path $PATH_FILE = env for path file -r = option for resource fgsreq = name of resource file - Pipeline Mode
genreq -p $PATH_FILE -r fgsreq where: genreq = command name
-p = option for path $PATH_FILE = env for path file
-r = option for resource fgsreq = name of resource fileFGSRSP (RS) Process
The FGSRSP process, using the ingrsp command, processes a response from the ARCHIVE. The resource file is used to specify parameters particular to the FGS pipeline, including FGS class, FAS class and EPC class responses. We will show the example of FGS class processing.
- Input Trigger
The FGSRSP process is a time polling process that uses the response data to update the OSF of the archived dataset. The FGSREQ process sets a W in the RS stage but this is not a trigger for this process.
(DC) |
(AN) |
(RQ) |
(RS) |
(CL) |
(Other) |
Class |
DC_JIT |
------ |
FGSREQ |
FGSRSP |
FGSCLN |
------ |
FGS |
C |
U |
C |
W |
- |
- |
FGS |
C |
O |
C |
W |
- |
- |
FGS |
- Output Trigger
After successfully processing the response data the RS status is set to C, and the CL stage is set to W, which triggers the FGSCLN process.
(DC) |
(AN) |
(RQ) |
(RS) |
(CL) |
(Other) |
Class |
DC_JIT |
------ |
FGSREQ |
FGSRSP |
FGSCLN |
------ |
FGS |
C |
U |
C |
C |
W |
- |
FGS |
C |
O |
C |
C |
W |
- |
FGS |
The FGSRSP process reads a response file from the directory specified in the resource file to be used for archive responses.
· FGSRSP Process Error Handling
The FGSRSP reports errors concerning the HDA response for this dataset.
The FGSREQ process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
ingrsp rootname -p path_name -r fgsrsp (typed into command line)where: ingrsp = command name
rootname = jitter IPPPSSOOT
-p = option for path $PATH_FILE = env for path file -r = option for resource fgsrsp = name of resource file - Pipeline Mode
ingrsp -p $PATH_FILE -r fgsrsp where: ingrsp = command name
-p = option for path $PATH_FILE = env for path file
-r = option for resource fgsrsp = name of resource file
FGSCLN (CL)
Process
The FGSCLN process, using the cleandata command, uses the resource file to delete OSF related files from any directory they may be found. This is a standard process used in OPUS. This process is triggered using the AR stage value set by the ARCHIVE pipeline.
- Input Trigger
The FGSCLN process is an OSF polling process that is triggered for any uncollected or associated jitter files that have been sent to the archive. A status of W must be set in the CL stage of the FGS class OSF by the Archive pipeline.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
FGSREQ |
FGSRSP |
FGSCLN |
FGSFIN |
FGSDEL |
FGS |
U |
C |
C |
W |
- |
- |
FGS |
- Output Trigger
After successfully deleting the archived files, the next stage is triggered by setting a W in the FN stage so that the telemetry data may be released for this uncollected jitter dataset.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
FGSREQ |
FGSRSP |
FGSCLN |
FGSFIN |
FGSDEL |
FGS |
U |
C |
C |
C |
W |
- |
FGS |
The FGSCLN process reads resource files from the definitions directory.
· FGSCLN Process Error Handling
Any failure by FGSCLN will generate an F in the CL stage for the FGS class OSF.
The FGSCLN process can be executed only in pipeline mode as follows:
- Pipeline Mode
cleandata -p $PATH_FILE -r fgscln
where: cleandata = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource fgscln = name of resource fileFGSFIN (DL) Process
This process is run in the FGS-pipeline Clean stage (DL) when it is triggered by a status value W. The program that executes this process, eng_finish, is shared with the AST pipeline and with two FGS-pipeline processes. The resource file is used to distinguish the processing between the different classes. For FGSFIN processing, the same steps are performed as in GSAFIN Process Description.
- Input Trigger
The FGSFIN process is triggered after the jitter dataset is cleaned up by FGSCLN. The trigger is a value W in the FN column of the FGS class OSF.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
FGSREQ |
FRSRSP |
FGSCLN |
FGSFIN |
FGSDEL |
FGS |
U |
C |
C |
C |
W |
- |
FGS |
I |
N |
N |
C |
W |
- |
FGS |
O |
N |
N |
C |
W |
- |
FGS |
- Output Trigger
The completion of the entire class is indicated by a C status in the DL stage set by the FGSFIN process.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
FGSREQ |
FGSRSP |
FGSCLN |
FGSFIN |
FGSDEL |
FGS |
U |
C |
C |
C |
C |
C |
FGS |
I |
N |
N |
C |
C |
C |
FGS |
O |
N |
N |
C |
C |
C |
FGS |
For each FGS telemetry file for which all raw-jitter processing is complete, the ENG OSF is updated as follows, changing the RJ status from value R to C.
(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
R |
C |
R |
N |
W |
ENG |
This process uses database queries and log messages.
· FGSFIN Process Error Handling
A failure is reported as a log message and an F is set in the FN stage of the FGS OSF. The failure would involve some database problem or a failure to find or update an OSF.
The FGSFIN process can be executed only in the pipeline. Automatic cleanup is not needed in an interactive situation.
- Pipeline Mode
xpoll -p $PATH_FILE -r FGSFIN command: eng_finish.plwhere: xpoll = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource FGSFIN = name of resource fileBLDASN (DC) Process
The BLDASN is process is an internal OSF polling process. It creates a FITS association table from the data in the OPUS_DB database.
1. The process checks if the complete_date in the asn_association table is non-blank.
2. If not, it terminates with an OSF_HOLD status.
3. It reads the database to get the root names of all the exposure and product members of the association. It also gets the member status. The data from the query is stored in a list.
4. The FITS primary header is written. The FITS table extension keywords are written. The list of members is used to write the FITS table.
- Input Trigger
The BLDASN process is triggered by a status W in the DC stage of the FAS OSF. This is set either by the DC_JIT process after it completes the collect of jitter files, or by the OPUS pipeline after it has updated the asn_association table. (Other) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
------ |
BLDASN |
FGSASN |
FGSREQ |
------ |
FAS |
- |
W |
- |
- |
- |
FAS |
- Output Trigger
It updates the FAS OSF to indicate the generation of the association file by setting a value W in the AN stage that is the trigger for the FGSASN process. (Other) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
------ |
BLDASN |
FGSASN |
FGSREQ |
------ |
FAS |
- |
C |
W |
- |
- |
FAS |
If the OPUS pipeline has not yet updated the asn_association table, it updates the FAS OSF to indicate that it is holding for external updating by setting a value H in the DC stage. The H is reset to W by the OPUS pipeline. (Other) |
(DC) |
(AN) |
(RQ) |
(Other) |
Class |
------ |
BLDASN |
FGSASN |
FGSREQ |
------ |
FAS |
- |
H |
- |
- |
- |
FAS |
BLDASN queries database tables to get the member status and the product names. It uses the Farris FITSPP package to write the FITS association file.
· BLDASN Process Error Handling
If a processing error occurs a log message is written and the GN stage of the FAS OSF is set to F.
The BLDASN process can be run interactively or in the pipeline.
o Interactive Mode
asn_table rootname -p path_name -r bldasn (typed into command line)where: asn_table = command name
-p = option for path
path_name = name of path file
rootname = association IPPPSSAA0
-r = option for resource bldasn = name of resource
- Pipeline Mode
asn_table -p $PATH_FILE -r bldasnwhere: asn_table = command name
-p = option for path $PATH_FILE = env for path file -r = option for resourcebldasn = name of resource file
FGSASN (AN) Process
This process is run in the AN stage of the FAS class OSF when both the association file and the member jitter files are known to be present. It performs the following tasks.
1. Reads the entire association file identifying products and members.2. Verifies the member jitter files are present.3. Includes jitter members into FITS extensions of the associated FITS products.4. Computes association keywords for the primary headers of the association products from keywords in member extensions.5. Updates association keywords for the primary headers of orphan members that are not included in association products.6. For included members of the association, the FGS class OSF has the AN stage set to I for included, and it has the CL stage set to W to trigger FGSCLN.7. For orphan members, the member class OSF has the AN stage set to O for orphan, and depending on the association type, it either has the RQ stage set to W to wait for the ARCHIVE, or it has the CL stage set to W to trigger FGSCLN.8. The AN stage of the FAS OSF is set to C at completion and the RQ stage is set to W to send the FITS association file, any association products, and any orphan members to the ARCHIVE.
- Input Trigger
Triggered by completion the BLDASN process, which sets a value W in the AN stage after it completes.
(DC) |
(AN) |
(RQ) |
(RS) |
(CL) |
(Other) |
Class |
BLDASN |
FGSASN |
FASREQ |
FASRSP |
FASCLN |
------ |
FAS |
C |
W |
- |
- |
- |
- |
FAS |
- Output Trigger
The FAS archive request process is triggered by a status W in the RQ stage.
(DC) |
(AN) |
(RQ) |
(RS) |
(CL) |
(Other) |
Class |
BLDASN |
FGSASN |
FASREQ |
FASRSP |
FASCLN |
------ |
FAS |
C |
C |
W |
- |
- |
- |
FAS |
The FGS class OSF for each member is set according to its condition in the association table. For an included member, the AN stage status is set to the value I, and the RQ stage is set to the value N since the FASREQ does the work, and either the CL stage is set to V for wavecal jitter files that are still needed, or it is set to W to clean up the member jitter files. For an orphan member, the stage status is set to O, and the RQ and RS stages are set to the value N, and the CL stage is set to W to wait for completion of processing by the ARCHIVE. Orphans that are sent separately to the ARCHIVE have the RQ stage status set to W.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
FGSREQ |
FGSRSP |
FGSCLN |
FGSFIN |
FGSDEL |
FGS |
I |
N |
N |
V |
- |
- |
FGS |
I |
N |
N |
W |
- |
- |
FGS |
O |
N |
N |
W |
- |
- |
FGS |
O |
W |
- |
- |
- |
- |
FGS |
The FGSASN process uses FITSPP package to read and write FITS tables. It reads the FITS association table, updates any orphan jitter image file, and creates the associated FITS jitter table and jitter image files.
· FGSASN Process Error Handling
The FGSASN process set the value F in the GN stage of the FAS OSF when it has a fatal error. The log file describes the error.
The FGSASN process can be executed interactively or in pipeline mode as follows:
o
Interactive Mode
fgsasn rootname (typed into command line)where: fgsasn = command name
rootname = association name
- Pipeline Mode
fgsasn -p $PATH_FILE -r fgsasn where: fgsasn = command name
-p = option for path $PATH_FILE = env for path file -r = option for resourcefgsasn = name of resource file
FASCLN (CL) Process
The FASCLN process, using the cleandata command, uses the resource file to delete OSF related files from any directory in which they may be found. This is a standard process used in OPUS. This process is triggered using the CL stage value set by the FASRSP process.
- Input Trigger
The FASCLN process is an OSF polling process that is triggered for any associated jitter files that have been sent to the archive. A status of W must be set in the CL stage of the FAS class OSF, by the FASRSP process.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
FGSASN |
FASREQ |
FASRSP |
FASCLN |
------ |
FGSDEL |
FAS |
C |
C |
C |
W |
- |
- |
FAS |
- Output Trigger
After successfully deleting the archived files, the last stage is triggered by setting a C in the DL stage so that the OSF may be deleted.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
FGSASN |
FASREQ |
FASRSP |
FASCLN |
------ |
FGSDEL |
FAS |
C |
C |
C |
C |
N |
C |
FAS |
The FASCLN process reads resource files from the definitions directory.
· FASCLN Process Error Handling
Any failure by FASCLN will generate an F in the AR stage for the FAS class OSF.
The FASCLN process can be executed only in pipeline mode as follows:
- Pipeline Mode
cleandata -p $PATH_FILE -r fascln
where: cleandata = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource fascln = name of resource file RAWEPC (RE) Process
The RAWEPC process is triggered when the telemetry for the engineering product is available. The results of RAWEPC are written to a single file: a binary file for raw engineering data having a record for every telemetry item during the time range found in the database table qolink_sms as extended by the time pads in eng_dataset_pads table.
- Input Trigger
The RAWEPC process is triggered by a W status in the RE stage.
(RE) |
(GN) |
(AN) |
RQ |
(Other) |
Class |
RAWEPC |
EPCGEN |
------ |
EPCREQ |
------ |
EPC |
W |
- |
- |
- |
- |
EPC |
- Output Trigger
On successful completion, the trigger for the EPCGEN process is set with a value W in the GN stage.
(RE) |
(GN) |
(AN) |
RQ |
(Other) |
Class |
RAWEPC |
EPCGEN |
------ |
EPCREQ |
------ |
EPC |
C |
W |
- |
- |
- |
EPC |
The RAWEPC process reads database tables and the FGS engineering telemetry. It writes log messages and the raw product file.
· RAWEPC Process Error Handling
Telemetry
gaps do not cause an error condition. Failure to read a telemetry file or
missing database records will cause an error message to be logged and the GN
status for the EPC class OSF will be set to F.
The RAWEPC process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
rawepc rootname (typed into command line) where: rawepc = command name rootname = IPPPSSOOT of the product dataset EXAMPLE: To run RAWEPC with product files having the rootname U3450101M from the pipeline:rawepc u3450101m
- Pipeline Mode
rawepc -p $PATH_FILE -r rawepc where: rawepc = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource rawepc = name of resource file EPCGEN is an EPC-Class, OSF-triggered process that generates the EPC FITS file that is also called the product file. Each OSF controls the generation of one product file. When an EPC Class OSF is set to W in the GN stage, the EPCGEN process is triggered to perform the following actions:
- Construct the EPC raw file specification from the OSF rootname or from the interactive filename using the directory specified in the INPATH keyword.
- Calculates mean and standard deviation keywords for each telemetry table extension while reading and segregating data from all the records in the raw data file.
- Generate the FITS tables for each telemetry item as specified in the eng_products database table.
- Set the RQ stage of the EPC Class OSF to the value to trigger EPCREQ, or set the GN stage to the value F to indicate a processing failure has occurred.
- Input Trigger
Triggered by completion of RAWEPC processing:(RE) |
(GN) |
(AN) |
(RQ) |
(Other) |
Class |
RAWEPC |
EPCGEN |
------ |
EPCREQ |
------ |
EPC |
C |
W |
- |
- |
- |
EPC |
- Output Trigger
The next step is the archive request EPCREQ for products. It is looking for a W in the RQ status.(RE) |
(GN) |
(AN) |
(RQ) |
(Other) |
Class |
RAWEPC |
EPCGEN |
------ |
EPCREQ |
------ |
EPC |
C |
C |
N |
W |
- |
EPC |
The EPCGEN process reads the FITS keyword definitions from the database. It reads keyword data from the JKX file created by RAWEPC. It uses the JKX data to get the rootname of the ERX file that contains the raw EPC data to be processed into an EPC FITS file. All errors while generating the FITS file are echoed to the EPCGEN Process Log file.
· EPCGEN Process Error Handling
Most errors that occur in EPCGEN processing will cause the EPCGEN (GN) stage in the EPC Class OSF to be set to "F". Error messages are also posted to the EPCGEN Process Log File in the OPUS_HOME_DIR
The EPCGEN process can be executed interactively or in pipeline mode as follows:
o Interactive Mode
epcgen rootname (typed into command line) where: epcgen = command name rootname = IPPPSSOOT of the jitter dataset - Pipeline Mode
epcgen -p $PATH_FILE -r epcgen where: epcgen = command name
-p = option for path $PATH_FILE = env for path file-r = option for resource
epcgen = name of resource file
EPCCLN (CL)
Process
The EPCCLN process, using the cleandata command, uses the resource file to delete OSF related files from any directory they may be found. This is a standard process used in OPUS. This process is triggered using the AR stage value set by the ARCHIVE pipeline.
- Input Trigger
The EPCCLN process is an OSF polling process that is triggered for any uncollected or associated jitter files that have been sent to the archive. A status of W must be set in the CL stage of the EPC class OSF by the Archive pipeline.
(Other) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
EPCRSP |
EPCCLN |
EPCFIN |
EPCDEL |
EPC |
- |
C |
W |
- |
- |
EPC |
- Output Trigger
After successfully deleting the archived files, the next stage is triggered by setting a W in the FN stage so that the telemetry data may be released for this uncollected jitter dataset.
(Other) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
EPCRSP |
EPCCLN |
EPCFIN |
EPCDEL |
EPC |
- |
C |
C |
W |
- |
EPC |
The EPCCLN process reads resource files from the definitions directory.
· EPCCLN Process Error Handling
Any failure by EPCCLN will generate an F in the CL stage for the EPC class OSF.
The EPCCLN process can be executed only in pipeline mode as follows:
- Pipeline Mode
cleandata -p $PATH_FILE -r epccln
where: cleandata = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource epccln = name of resource fileEPCFIN (DL) Process
This process is run in the EPC-pipeline Clean stage (DL) when it is triggered by a status value W. The program that executes this process, eng_finish, is shared with the AST pipeline and with two EPC-pipeline processes. The resource file is used to distinguish the processing between the different classes. For EPCFIN processing, the same steps are performed as in GSAFIN Process Description.
- Input Trigger
The EPCFIN process is triggered after the product dataset is cleaned up by EPCCLN. The trigger is a value W in the FN column of the EPC class OSF.
(Other) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
EPCRSP |
EPCCLN |
EPCFIN |
EPCDEL |
EPC |
- |
C |
C |
W |
- |
EPC |
- Output Trigger
The completion of the entire class is indicated by a C status in the DL stage set by the EPCFIN process.
(Other) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
EPCRSP |
EPCCLN |
EPCFIN |
EPCDEL |
EPC |
- |
C |
C |
C |
C |
EPC |
For each FGS telemetry file for which all raw-jitter processing is complete, the ENG OSF is updated as follows, changing the RE status from value R to C.
(FP) |
(GA) |
(RJ) |
(RE) |
(Other) |
(DL) |
Class |
FGSPOL |
GAHOLD |
RJHOLD |
REHOLD |
------ |
DL_FGS |
ENG |
C |
R |
R |
C |
N |
W |
ENG |
This process uses database queries and log messages.
· EPCFIN Process Error Handling
A failure is reported as a log message and an F is set in the FN stage of the EPC OSF. The failure would involve some database problem or a failure to find or update an OSF.
The EPCFIN process can be executed only in the pipeline. Automatic cleanup is not needed in an interactive situation.
- Pipeline Mode
xpoll -p $PATH_FILE -r EPCFIN command: eng_finish.plwhere: xpoll = command name
-p = option for path $PATH_FILE = env for path file -r = option for resource FGSFIN = name of resource fileFGSDEL (DL) Process
The FGSDEL process is used to delete the OSF for any class in the FGS pipeline.
- Input Trigger
The FGSDEL process is an OSF polling process that is triggered only by a status designated in the FGSDEL resource file. Since the completion status in this stage is C, if the trigger used is C, the OSF will be deleted immediately after the DL stage has been set to C, if the FGSDEL process is running.
(AN) |
(RQ) |
(RS) |
(CL) |
(FN) |
(DL) |
Class |
------ |
------ |
------ |
------ |
------ |
FGSDEL |
ENG |
N |
N |
N |
N |
N |
C |
ENG |
------ |
------ |
------ |
------ |
GSAFIN |
FGSDEL |
GSA |
N |
N |
N |
N |
C |
C |
GSA |
DC_JIT |
FGSREQ |
FGSRSP |
FGSCLN |
FGSFIN |
FGSDEL |
FGS |
U |
C |
C |
C |
C |
C |
FGS |
I |
N |
N |
C |
C |
C |
FGS |
O |
C |
C |
C |
C |
C |
FGS |
FGSASN |
FGSREQ |
FGSRSP |
FASCLN |
------ |
FGSDEL |
FAS |
C |
C |
C |
C |
N |
C |
FAS |
------ |
FGSREQ |
FGSRSP |
EPCCLN |
EPCFIN |
FGSDEL |
EPC |
N |
C |
C |
C |
C |
C |
EPC |
- Output Trigger
None. The OSF is deleted.
None.
·
FGSDEL Process Error
Handling
If
the OSF cannot be deleted a message is logged and an attempt is made to set the
DL stage to E. It is highly unlikely that the OSF update will succeed.
The FGSDEL process can be executed only in pipeline mode as follows:
- Pipeline Mode
osfdelete -p $PATH_FILE -r fgsdel where: osfdelete = command name
-p = option for path $PATH_FILE = env for path file -r = option for resourcefgsdel = name of resource file