---

Modularized OMS Pipeline Reference Manual

---

Table of Contents

• Modularized OMS Pipeline Overview
• Modularized OMS Pipeline Reference Manual Caveats
• Intended Audience and Scope
• The need for a Modularized OMS Pipeline
• Pictorial View Of Modularized OMS Pipeline
• Modularized OMS Pipeline Nominal Operational Description
• Modularized OMS Pipeline Directories and Files
• Modularized OMS Pipeline Processes
• Modularized OMS Pipeline Processes Overview
• Modularized OMS Pipeline Processes Details
• Modularized OMS Pipeline Process Resource Files
• Modularized OMS Pipeline Database Usage
• Database Relations Accessed
• Database Queries

---

Pipeline Overview

Pipeline Caveats

• Caveat 1 - It should be understood that in the context of this document, the phrase "Modularized OMS Pipeline" is not used to refer to the entire OMS Pipeline; it is meant to specifically address 4 new processes that are replacing and assuming the responsibility of the previously singular OMSANAL Process.
• Caveat 2 - This document has been updated for the second and final implementation of JITTER and XTRACT. Some examples of configured software such as resource files have not been altered and the user should not expect all the values to be the same as the phase-2 delivery.

Intended Audience and Scope

This Reference Manual is intended primarily to give OPUS Operations Personnel information necessary to understand the operation of the Modularized OMS Pipeline. This includes information on Input and Output Products, Directories and Files used, Database Tables used, and the Processes involved. Although the document is partially a consolidation of information contained in Modularized OMS Pipeline Requirements and Design Documents, additional information not previously presented is provided. Also, it is envisioned that the document is to be a living document that will be updated as Modularized OMS Pipeline modifications are made. This is not the case for the Requirements and Design documents as they are and will remain static.

The need for a Modularized OMS Pipeline

In its original form, the OMSANAL process was very monolithic and hard to maintain. It performed a lot of processing sequentially where much of the processing could be performed in parallel. OMSANAL was also very inflexible in its production of output products; all output products are always produced even if only a small subset of the products are actually desired. Therefore, in the interest of increasing the flexibility, maintainability, and efficiency of OMS Processing, the original OMSANAL process was partitioned into 4 distinct, concurrent processes; i.e., the XTRACT, JITTER, COLLCT, and SAVGEN Processes. Each of these processes will provide a specific portion of the functionality previously provided by the singular OMSANAL process. The 4 new processes also provide additional capabilities not provided by OMSANAL.

Pictorial View of Pipeline

This section is provided to illustrate how the original OMSANAL process was partitioned into the 4 processes constituting the Modularized OMS Pipeline. Partial stage diagrams of the pipelines are as follows:

 
 
 Old
 OMS Pipeline ... OMSCHD  OMSANAL  SAVCPY ...
 ------------- (SD)  (AL)  (SC)
  /\
 / \
 / \
  / \
 / \
 / \
 / \
 / \
  / \
 / \
 / \
 / \
 /  \
 / \
 / \ 
 Modularized  / \ 
 OMS Pipeline ... OMSCHD  XTRACT JITTER COLLCT SAVGEN SAVCPY ...
 ------------ (SD)  (XT) (JT) (DC) (SV)  (SC)

Pipeline Nominal Operational Description

The Modularized OMS Pipeline is invoked when its XTRACT process is notified that there are Telemetry Subset Files ready for processing. This notification, in the form of an OSF "W" trigger in the pipeline "XT" Stage Column, is generated by the OMS Pipeline OMSCHD process after a number of activities prior to the Modularized Pipeline have occurred. These activities, ending with the actual notification, include the following:

1. Weekly Science Mission Schedule (SMS) is parsed by the Non-Science Pipeline YECMSC process to create a Weekly Event File (*.EVT file)

2. Non-Science Pipeline OMSEVT process is run to load events from the Weekly Event File into a database table (OMS_SCHED_TABLE) used by the OMS Pipeline OMSCHD process to create Scheduling Unit Files

3. Telemetry Subsets are received by the OMS Pipeline; Error Files are generated for the received subsets; and the Pipeline OMSTLM process inserts Telemetry Events into the OMS_SCHED_TABLE to indicate contiguous portions of telemetry within the received subset files

4. OMS Pipeline OMSCHD process examines SMS and Telemetry Events in OMS_SCHED_TABLE and creates Scheduling Unit Files indicating which portions of which telemetry subsets are ready for processing by the OMS Modularized Pipeline

5. OMSCHD process creates Scheduling Class (SCD) OSF and sets the OSF to "W" in the Modularized Pipeline "XT" stage column to trigger the XTRACT Process

Upon being triggered, the Modularized OMS Pipeline examines various switch settings in its process' resource files to determine what work it is to perform and what output products it is to produce; see XTRACT Switches, COLLCT Switches, and SAVGEN Switches. Once it knows what is being requested of it, the Pipeline proceeds to read the input Scheduling Unit Files that indicate which portions of which telemetry subsets are ready for processing. Then, as dictated by the requested output products and in accordance with mission schedule events in the .EVT file, appropriate telemetry data (jitter data and/or save file data and/or astrometry data) is extracted from the subset files and packaged into the requested output products. As this processing occurs, records are written to various Database Relations to maintain a history of the processing that was performed.

---

Pipeline Directories and Files

· OPUS_HOME_DIR

Contains Process Status Files (PSFs) and Process Log Files

· OPUS_HOME_DIR_LOCK

Contains transient Lock Files used to avoid PSF collisions

· OPUS_OBSERVATIONS_DIR

Contains Observation Status Files (OSFs)

· OPUS_OBSERVATIONS_DIR_LOCK

Contains transient Lock Files used to avoid OSF collisions

· OPUS_DEFINITIONS_DIR

Contains Path files, Stage Files, Process Resource Files, and OMG Data Files:

o OMS.PATH_TEMPLATE

- Defines directory mnemonics used by OMS Processes when they are run in Pipeline mode. Also defines mnemonics such as OPUS_DB, DSQUERY, and OK_TO_UPDATE_DATABASE for pipeline mode database access

o NULL_PATH.TEMPLATE

- Defines directory mnemonics used by OMS Processes when they are run in Interactive mode. Also defines mnemonics such as OPUS_DB, DSQUERY, and OK_TO_UPDATE_DATABASE for interactive mode database access

o OMS_PIPELINE.STAGE

- Defines the stages in the OMS Pipeline and the possible status values for each stage

§ XT - extract telemetry (XTRACT) Stage

§ JT - produce obslogs (JITTER) Stage

§ DC - collect obslogs data (COLLCT) stage

§ SV - generate save files (SAVGEN) Stage

§ XTRACT.RESOURCE

- Defines characteristics of the XTRACT Process

§ JITTER.RESOURCE

- Defines characteristics of the JITTER Process

§ COLLCT.RESOURCE

- Defines characteristics of the COLLCT Process

§ SAVGEN.RESOURCE

- Defines characteristics of the SAVGEN Process

§ OBSFIL_COMPLETE.DAT

- Used by the OMG to determine what OSFs are complete and can be removed from the OMG

§ OBSFIL_DELETE.DAT

- Used by the OMG to delete a given OSF and the specified files associated with the OSF

· OMS_INIT_DIR

Contains initialization files input to the pipeline

o delay_time.dat - limits messages output delay specification file

o limits.dat - telemetry parameter limits specification file

o keywords.oms - obslogs header keyword specification file

o testbed.ver - software version file

o si_mode.dat - science instrument mode definitions file

o wemjpl.epm - JPL ephemeris file

· OMS_EVT_DIR

Contains mission schedule event files input to the pipeline

o msc_yydddhhmm_yydddhhmm_*.evt - mission schedule event file

· OMS_DATA_DIR

Contains scheduling unit files and telemetry subset and error files input to the pipeline

o yydddhhmmss_yydddhhmmss.lis - scheduling unit file

o Symdhhmm.vc*_done - telemetry subset file

o Symdhhmm.er*_done - telemetry error file

· OMS_OUTPUT_DIR

Contains various files output by the pipeline

o acq_timing*.oms - save file

o breathing*.oms - save file

o costar_save*.oms - save file

o earth_avd*.oms - save file

o fgs_flags*.oms - save file

o fgsa_servo*.oms - save file

o fgsg_servo*.oms - save file

o fgs_save*.oms - save file

o foc_save*.oms - save file

o fos_save*.oms - save file

o gen_save*.oms - save file

o gs_acg*.oms - save file

o hrs_save*.oms - save file

o nic_save*.oms - save file

o oms_errors*.oms - save file

o oms_summary*.oms - save file

o pmt_dark_cnt*.oms - save file

o power*.oms - save file

o stis_save*.oms - save file

o tel_errors*.oms - save file

o thermal*.oms - save file

o theta_p*.oms - save file

o walkdown*.oms - save file

o wfii_save*.oms - save file

o gs_jit*.oms - guide star jitter file

o combined_yydddhhmmss.sav - combined save file produced by the XTRACT process and used by the SAVGEN process to generate individual save files

o header*.oms - telemetry subset header file

o oms_errors*.oms - telemetry errors file

o jitter*.oms - high-resolution jitter data for JITTER process

o lo_res*.oms - low-resolution jitter data for JITTER process

o *.jkx - jitter keywords for each observation for JITTER process

· OMS_OBSLOGS_DIR

Contains obslogs files output by the JITTER process (.jit, .jwt, .jif and .jwf Files)

o *.jit - observation jitter table file

o *.jwt - observation jitter table file for wavcals

o *.jif - observation jitter image file

o *.jwf - observation jitter image file for wavecals

· OMS_ASTDATA_DIR

Contains astrometry header and data files output by the pipeline

o *.akx - astrometry data header file

o *.arx - astrometry data file

· OPUS_HOME_DIR

Contains XTRACT Process Status Files (PSFs) and Process Log Files

· OPUS_OBSERVATIONS_DIR

Contains Observation Status Files (OSFs) created by the XTRACT process

 
*** The file naming convention fields used in above notations are decoded as follows:
 yyyy = 4 digit year
 yy = 2 digit year
 ddd = 3 digit day of year (001 - 365)
 hh = 2 digit hour of day (00-23)
 mm = 2 digit minute of hour (00-59)
 ss = 2 digit second of minute (00-59)
 
 y = year since 1980 (1-Z)
 m = month of year (1-C)
 d = day of month (1-V)

---

Pipeline Processes

Pipeline Processes Overview

As indicated in the Overview section, the Modularized OMS Pipeline is composed of the following 4 processes:

1. XTRACT - extracts telemetry data from subset files corresponding to events in mission schedule event files
2. JITTER - generates obslogs and sets observation OSFs to trigger the COLLCT process
3. COLLCT - ensures that all jitter files produced under a given scheduling unit are completed before deleting no-longer-needed files and triggering the SAVGEN process
4. SAVGEN - produces specific individual save files from a master combined save file

Pipeline Processes Details

XTRACT

• XTRACT Process Description
• XTRACT Process Triggers
• XTRACT Process I/O
• XTRACT Process Error Handling
• XTRACT Process Modes
• XTRACT Process Queries
• XTRACT Process Resource File

JITTER

• JITTER Process Description
• JITTER Process Triggers
• JITTER Process I/O
• JITTER Process Error Handling
• JITTER Process Modes
• JITTER Process Resource File

COLLCT

• COLLCT Process Description
• COLLCT Process Triggers
• COLLCT Process I/O
• COLLCT Process Error Handling
• COLLCT Process Modes
• Collect Process Queries
• COLLCT Process Resource File

SAVGEN

• SAVGEN Process Description
• SAVGEN Process Triggers
• SAVGEN Process I/O
• SAVGEN Process Error Handling
• SAVGEN Process Modes
• SAVGEN Process Resource File

=========================================================================================== XTRACT (XT) Process

• XTRACT Process Description

This process is the 1st of the 4 processes in the Modularized OMS Pipeline. It is an SCD Class OSF triggered process that performs most of the tasks that were performed by the original OMSANAL process. In fact, much of the existing OMSANAL code is reused in this process. The primary purpose of this process is to extract telemetry data from subset files corresponding to events in mission schedule event files. The extracted telemetry data is composed of jitter data, save file data, and astrometry data. A major feature of the XTRACT process is the capability to select, via (Y)es/(N)o switches in the process resource file, which output products are generated. Output products generated by XTRACT may be temporary/intermediate products solely for use by downstream processes or actual end products. The XTRACT "output product" switches are as follows:

• generate_save_data - for generation of intermediate combined save files utilized by the SAVGEN process to generate individual save files. If this flag is set to "no", a combined save file will still be produced however the file will be empty.
• generate_errors - for generation of OMS errors files; one for each scheduling unit (SU) file processed
• generate_header - for generation of telemetry subset header files; one for each telemetyry subset file processed
• generate_gs_jit - for generation of guide star jitter files
• generate_ast - for generation of intermediate astrometry data files used by the Astrometry Pipeline
• generate_jitter - for generation of keyword files and high- and low-resolution jitter data for use by JITTER process.

When a Scheduling Unit (SCD Class) OSF is set to "W" in the XT stage column, the XTRACT process is triggered to perform the following actions:

• Check the settings of the "output product" switches to determine what output products to generate
• Update information in the OMS_SU_TABLE database table to indicate each portion of each telemetry subset processed under the current scheduling unit
• Extract telemetry from subset files according to events in mission schedule event file for the output products selected to be generated
• Insert information into the SCD_OMS_MAP database table to indicate each observation processed under the current scheduling unit and the date the observation was processed
• Insert information into the SCD_SAVE_DATE database table to indicate what save files were processed under the current scheduling unit
• Create observation (OMS class) OSFs for each observation processed under current scheduling unit and set the OSF to trigger the JITTER process
• Set the SCD class OSF that triggered the process to "C" in the XTRACT (XT) column
• Set the SCD class OSF that triggered the process to "V" in the SAVGEN (SV) column. The "V" indicates to the SAVGEN process that it should wait until the COLLCT process has finished collecting all of the observations processed under a given scheduling unit before generating save files associated with the scheduling unit.

· XTRACT Process Triggers

• Input Trigger

 
 Triggered by OMSCHD process:
 
 OMSCHD XTRACT JITTER COLLCT SAVGEN SAVCPY OSF 
 (SD) (XT) (JT) (DC) (SV) (SC) CLASS
 ----- ---- ---- ---- ---- ---- -----
 C W _ _ _ _ SCD
 

• Output Triggers

 
 Trigger for JITTER process when obslogs are produced:
 
 JITTER COLLCT SAVGEN OSF
 (JT) (DC) (SV) CLASS
 ----- ---- ---- -----
  W _ _ OMS
 
 Trigger for SAVGEN process when obslogs are produced
 
 OMSCHD XTRACT JITTER COLLCT SAVGEN SAVCPY OSF 
 (SD) (XT) (JT) (DC) (SV) (SC) CLASS
 ----- ---- ---- ---- ---- ---- -----
 C C _ _ V _ SCD
 

• XTRACT Process I/O

 
 INPUT:
 OMS_EVT_DIR - contains mission schedule event files
 msc_yydddhhmm_yydddhhmm_*.evt - event file
 
 OMS_DATA_DIR - contains scheduling unit files and telemetry subset/error files 
 yydddhhmmss_yydddhhmmss.lis - scheduling unit file
 Symdhhmm.vc*_done - telemetry subset file
 Symdhhmm.er*_done - telemetry error file
 
 Information input from the WIEPHEMERIS Relation
 
 "output_product" switch settings from process resource file
 
 OUTPUT:
 All of the output files that are specified in the
 Modularized OMS Pipeline Directories and Files section
 earlier in this document are produced by the XTRACT process according to the
 "output product" switches specified in the XTRACT process resource file
 
 Information output to the OMS_SU_TABLE, SCD_OMS_MAP, and SCD_SAVE_DATE Database Relations
 
 Setting of OMS and SCD Class OSFs to trigger the JITTER and SAVGEN Processes, respectively
 

• XTRACT Process Error Handling

Most errors that occur in XTRACT processing cause the SCD Class OSF that triggered the process to be set to "F" in the XTRACT (XT) column. Database errors however prohibit setting the OSF to "F" as these type errors cause the process to terminate before it can set the OSF. This situation results in OSF statuses becoming "frozen" in the state that they were in prior to the database error. Error messages are also posted to the XTRACT Process Log File in the OPUS_HOME_DIR

· XTRACT Process Modes

o Interactive Mode

 
  XTRACT i -f input_file_name -options (typed into command line)
 
 where: XTRACT = command name
 i = denotes interactive processing
 -f = denotes that the next argument is the input File
 input_file_name = denotes input file name
 Input files can be a scheduling unit (SU.LIS) file or a
 single telemetry subset file
 -options = denotes desired options
 -s = denotes input file is a scheduling unit (SU) file
 
 EXAMPLE:
 To run XTRACT using a scheduling unit file:
 
 XTRACT i -f 98240233730_98241012231.LIS -s
 
 
 To run XTRACT using a single telemetry subset file:
 
  XTRACT i -f SH331214W.VC0_DONE
 

o Pipeline Mode

 
 XTRACT -p opus_definitions_dir:your.path (in task line of resource file)
 
 where: XTRACT = command name
 -p = denotes path file specification follows
 
  opus_definitions_dir:your.path = path file to use

========================================================================================== JITTER (JT) Process

· JITTER Process Description

This process is the 2nd of the 4 processes in the Modularized OMS Pipeline. It is an OMS Class OSF triggered process that generates the JIT and JIT FITS files that are also referred to as obslogs. Each OSF controls the generation of one set of obslogs. When an Observation (OMS Class) OSF is set to "W" in the JT stage column, the JITTER process is triggered to perform the following actions:

• Construct the JKX file specification from the OSF rootname or from the interactive filename using the directory specified in the INPATH keyword.
• Open the jitter files specified in the JKX data and process the jitter data within the time range specified in the JKX file.
• Generate the FITS table and FITS image files with extensions ".jit" and ".jif". or ".jwt" and ".jwf" depending on the JKX keyword WAVEFLAG.
• Set the OMS Class OSF that triggered it to "C" in the JT column and "W" in the DC column to trigger the COLLCT (DC) process.

· JITTER Process Triggers

• Input Trigger

 
 Triggered by XTRACT process when generate_jitter = Y:
 
  XTRACT JITTER COLLCT SAVGEN OSF
 (XT) (JT) (DC) (SV) CLASS
 ----- ---- ---- ---- -----
 C W _ _ OMS
 
 

• Output Trigger

 
  Trigger for COLLCT process on successful completion:
 
 XTRACT JITTER COLLCT SAVGEN OSF
 (XT) (JT) (DC) (SV) CLASS
 ----- ---- ---- ----  -----
 C C W _ OMS 
 

• JITTER Process I/O

The JITTER process reads the FITS keyword definitions from the database. It reads keyword data from the JKX file created by XTRACT. It uses the JKX data to find the file and time range of jitter*.oms and lo_res*.oms file that contain the raw data to be processed into obslogs. The FITS files are written using the Farris FITSPP package. All errors from the FITSPP package are echoed to the JITTER Process Log file.

· JITTER Process Error Handling

Most errors that occur in JITTER processing will cause the JITTER (JT) column in the OMS Class OSF to be set to "F". Error messages are also posted to the JITTER Process Log File in the OPUS_HOME_DIR

· JITTER Process Modes

The JITTER process can be executed interactively or in pipeline mode as follows:

o Interactive Mode

 
 JITTER i -f input_file_name (typed into command line)
 
 where: JITTER = command name
 i = denotes interactive processing
  -f = denotes that the next argument is the input File
 input_file_name = denotes input JKX file name
 
 
 EXAMPLE:
 To run JITTER with a JKX file from the pipeline:
 
 JITTER i -f U3SR2601J.JKX_98240233730_98241012231

• Pipeline Mode

 
 JITTER -p opus_definitions_dir:your.path (in task line of resource file)
 
 where: JITTER = command name
 -p = denotes path file specification follows
 opus_definitions_dir:your.path = path file to use
 

========================================================================================= COLLCT (DC) Process

• COLLCT Process Description

This process is the 3rd of the 4 processes in the Modularized OMS Pipeline. It is an Observation (OMS Class) OSF triggered process that performs data collecting and data clean-up tasks. The primary purpose of this process is to ensure that all obslogs produced under a given scheduling unit have been completed before further processing associated with the scheduling unit can proceed; i.e., generation of save files for observations processed under the given scheduling unit. COLLCT examines obslog and scheduling unit information in the SCD_OMS_MAP database table. This process also examines "clean-up" (y)es/(n)o switches in its process resource file to determine when to delete "no-longer-needed" intermediate files produced by the XTRACT process during its obslog processing. These "clean-up" switches, applicable only when the XTRACT process "output-product" switch generate_jitter is set to (y)es in phase 2, are as follows:

• delete_jkx - delete all the .JKX files produced under a given scheduling unit
• delete_jit - delete all hi and lo resolution jitter files produced under a given scheduling unit

When an Observation (OMS Class) OSF is set to "W" in the DC stage column, the COLLCT process is triggered to perform the following actions:

• Perform a query on the SCD_OMS_MAP database table to obtain the scheduling unit name (su_name) that the observation (rootname) was processed under and the date (extract_date) that the XTRACT process processed the observation.
• Perform a 2nd query, using information obtained from the first query, to set the (complete_date) field of the table for the observation to the current system time. Also set the Observation (OMS Class) OSF to "C" for the current observation.
• Perform a 3rd query on the table to check if all observations processed under a given scheduling unit (su_name) and extract date (extract_date) have been completed; i.e., have a value in their complete date (complete_date) field. If this is the case, set the Scheduling Unit (SCD Class) OSF for the given scheduling unit to "W" in the SV column to trigger the SAVGEN process
• Check the settings of the "clean-up" switches in the process resource file to determine if intermediate files produced by the XTRACT process during its jitter processing can be deleted; if the switches are set to (Y)es, delete the appropriate files.

· COLLCT Process Triggers

• Input Trigger

 
 Triggered by JITTER process:
 
 XTRACT JITTER COLLCT SAVGEN OSF
 (XT) (JT) (DC) (SV) CLASS
 ----- ---- ---- ---- -----
 C C W _ OMS
 

• Output Trigger

 
 Trigger for SAVGEN process when obslogs are produced:
 
 OMSCHD XTRACT JITTER COLLCT SAVGEN SAVCPY OSF 
 (SD) (XT) (JT) (DC) (SV) (SC) CLASS
 ----- ---- ---- ---- ---- ---- -----
 C C _ _ W _ SCD
 

• COLLCT Process I/O

 
 INPUT:
 su_name and extract_date values from the SCD_OMS_MAP database table
 
 delete_jit and delete_jkx switch settings from the process resource file.
 
 OUTPUT:
 setting of complete_date field in the SCD_OMS_MAP database table and setting of OMS Class
 and SCD Class OSfs.
 
 setting of SCD Class OSF to trigger the SAVGEN Process
 

• COLLCT Process Error Handling

Most errors that occur in COLLCT processing cause the OMS Class OSF that triggered the process to be set to "F" in the COLLCT (DC) column. Database errors however prohibit setting the OSF to "F" as these type errors cause the process to terminate before it can set the OSF. This situation results in OSF statuses becoming "frozen" in the state that they were in prior to the database error. Error messages are also posted to the COLLCT Process Log File in the OPUS_HOME_DIR

· COLLCT Process Modes

The COLLCT Process can be executed in a pipeline mode only

o Pipeline Mode

 
 COLLCT -p opus_definitions_dir:your.path (in task line of resource file)
 
 where: COLLCT = command name
 -p = denotes path file specification follows
 
 opus_definitions_dir:your.path = path file to use

========================================================================================== SAVGEN (SV) Process

· SAVGEN Process Description

This process is the 4th of the 4 processes in the Modularized OMS Pipeline. It is a Scheduling Unit (SCD Class) OSF triggered process that generates individual save files from the "master" combined save file generated by the XTRACT process. A list of "save file" (y)es/(n)o switches in the process resource file indicate what specific save files should be generated and if the "master" combined save file should be deleted after the specific save files are generated. The switches are as follows:

• extract_acq - for generation of the acq_timing*.oms save file
• extract_acs - for generation of the acs_save*.oms save file
• extract_bre - for generation of the breathing*.oms save file
• extract_cos - for generation of the cos_save*.oms save file
• extract_cst - for generation of the costar*.oms save file
• extract_ear - for generation of the earth_avd*.oms save file
• extract_fga - for generation of the fgsa_servo*.oms save file
• extract_fgg - for generation of the fgsg_servo*.oms save file
• extract_fgs - for generation of the fgs_save*.oms save file
• extract_flg - for generation of the fgs_flags*.oms save file
• extract_foc - for generation of the foc_save*.oms save file
• extract_gen - for generation of the gen_save*.oms save file
• extract_gsr - for generation of the gs_req*.oms save file
• extract_ni1 - for generation of the nic1_save*.oms save file
• extract_ni2 - for generation of the nic2_save*.oms save file
• extract_ni3 - for generation of the nic3_save*.oms save file
• extract_pmt - for generation of the pmt_dark_cnt*.oms save file
• extract_pow - for generation of the power*.oms save file
• extract_sti - for generation of the stis_save*.oms save file
• extract_sum - for generation of the oms_summary*.oms save file
• extract_thl - for generation of the thermal*.oms save file
• extract_thp - for generation of the theta_p*.oms save file
• extract_wal - for generation of the walkdown*.oms save file
• extract_wf2 - for generation of the wfii_save*.oms save file
• extract_wf3 - for generation of the wf3_save*.oms save file
• delete_combined - for deletion of the master combined save file

When a Scheduling Unit (SCD Class) OSF is set to "W" in the SV stage column, the SAVGEN process is triggered to perform the following actions:

• Examine the "save file" switches in the process resource file to determine what individual save file should be generated from the "master" combined save file.
• Generate the individual save files as dictated by the "save file" switch settings
• Delete the "master" combined save file if the "delete_combined" switch is set to yes in the process resource file.
• Set the SCD Class OSF that triggerred it to "C" in the SV column and "W" in the SC column to trigger the SAVCPY (SC) process.

· SAVGEN Process Triggers

• Input Trigger

 
 
 Triggered by COLLCT process when obslogs are produced
 or triggered by XTRACT process when obslogs are not produced:
 
 OMSCHD XTRACT JITTER COLLCT SAVGEN SAVCPY OSF 
 (SD) (XT) (JT) (DC) (SV) (SC) CLASS
 ----- ---- ---- ---- ---- ---- -----
 C C _ _ W  _ SCD
 

• Output Trigger

 
 Trigger for SAVCPY process:
 
 OMSCHD XTRACT JITTER COLLCT SAVGEN SAVCPY OSF 
 (SD) (XT) (JT) (DC) (SV) (SC) CLASS
 ----- ---- ----  ---- ---- ---- -----
 C C _ _ C W SCD
 

• SAVGEN Process I/O

 
 INPUT:
 OMS_OUTPUT_DIR - "master" combined save file produced by XTRACT Process
 combined_yydddhhmmss.oms - master save file 
 
 "save file" switches from the process resource file
 
 OUTPUT:
 OMS_OUTPUT_DIR - individual save files 
 *.OMS - individual save files
 
 deletion of "master" combined save file
 
  setting of SCD Class OSf to trigger the SAVCPY Process
 

• SAVGEN Process Error Handling

Most errors that occur in SAVGEN processing cause the SCD Class OSF that triggered the process to be set to "F" in the SAVGEN (SV) column. Error messages are also posted to the SAVGEN Process Log File in the OPUS_HOME_DIR

· SAVGEN Process Modes

o Pipeline Mode

 
 SAVGEN -p opus_definitions_dir:your.path (in task line of resource file)
 
 where: COLLCT = command name
  -p = denotes path file specification follows
 
 opus_definitions_dir:your.path = path file to use
 

o Interactive Mode

 
 SAVGEN i -f name_of_combined_save_file (typed into command line)
 
 where: SAVGEN = command name
 i = denotes interactive processing
 -f = denotes data file specification follows
 
 
 EXAMPLE: 
 
 SAVGEN i -f combined_99125091545.oms 
 

---

Pipeline Process Resource Files

• XTRACT Process Resource File
• JITTER Process Resource File
• COLLCT Process Resource File
• SAVGEN Process Resource File

==========================================================================================

XTRACT Resource File

 
!--------------------------------------------------------------------
!
!  XTRACT RESOURCE FILE
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 06/29/99 38816 Goldst Created
!--------------------------------------------------------------------
 
PROCESS_NAME = XTRACT ! Short name for process id
TASK = XTRACT -P 'PATH_FILE -s
CLASS = SCD
DESCRIPTION = 'Extract telemetry and save data'
SYSTEM = OMS
DISPLAY_ORDER = 1
OK_TO_UPDATE_DATABASE = BLANK ! Determined by PATH
INTERNAL_POLLING_PROCESS = TRUE
 
OSF_RANK = 1 ! First Trigger
 
OSF_TRIGGER1.XT = W ! Need a 'wait' flag for XTRACT.
 
OSF_PROCESSING.XT = P ! Set the processing flag to 'Processing'
 
OMS_CREATE.JT = W ! The XTRACT process must create an OMS
 ! class OSF for each obslog to be
 ! processed by the JITTER process and set
  ! the JT column of the related OSFs to "W".
 
SCD_COMPLETE1.XT = C ! Normal: GENERATE_JITTER = Y
 ! GENERATE_SAVE_DATA = Y
SCD_COMPLETE1.SV = V ! SAVGEN is set to "vait" state
 
SCD_COMPLETE2.XT = C ! Reprocessing: GENERATE_JITTER = N
 ! GENERATE_SAVE_DATA = Y
SCD_COMPLETE2.SV = W ! Trigger creation of individual save files
 
SCD_FAIL.XT = F ! Error: Set the trouble flag
 
POLLING_TIME = 10 ! Wait (seconds) before polling for next
 
INPATH = OMS_DATA_DIR ! Directory of input files.
OMS_INIT_PATH = OMS_INIT_DIR ! Directory of init files.
OMS_PDB_PATH = OMS_PDB_DIR ! Directory of derived pdb files.
OMS_EVT_PATH = OMS_EVT_DIR ! Directory of event files.
OMS_OBSLOGS_PATH = OMS_OBSLOGS_DIR ! Directory of obslogs files. PHASE I only
OMS_DBASE_PATH = OMS_DBASE_DIR ! Directory of dbase files.
OUTPATH  = OMS_OUTPUT_DIR ! Directory of combined files.
ASTDATA = OMS_ASTDATA_DIR ! AST Directory of AKX and ARX files
MINBLOCKS = 50000 ! blocks required on output disk
 
GENERATE_SAVE_DATA = Y ! (Y/N) set to Y to get data for SAVGEN process
GENERATE_ERRORS = Y ! (Y/N) set to Y to get oms_errors*.oms files
GENERATE_HEADER = Y ! (Y/N) set to Y to get oms_header*.oms files
GENERATE_GS_JIT = Y ! (Y/N) set to Y to get gs_jit*.oms files
GENERATE_AST = Y ! (Y/N) set to Y to get data for OMSAST process
GENERATE_JITTER = Y ! (Y/N) set to Y to get data for JITTER process
 
! forces values from path to be used
 
 ENV.OPUS_DB = OPUS_DB
 ENV.DSQUERY = DSQUERY

==========================================================================================

JITTER Resource File

 
!--------------------------------------------------------------------
!
! JITTER RESOURCE FILE
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 06/15/99 38816 Goldst Created
!--------------------------------------------------------------------
 
PROCESS_NAME = JITTER ! Short name for process id
TASK = JITTER -P 'PATH_FILE
CLASS = OMS
DESCRIPTION = 'Produce obslogs'
SYSTEM = OMS
DISPLAY_ORDER = 1
OK_TO_UPDATE_DATABASE = BLANK ! Determined by PATH
INTERNAL_POLLING_PROCESS = TRUE
 
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.JT = W ! Need a 'wait' flag for production of obslogs
 
OSF_PROCESSING.JT = P ! Set the processing flag to 'Processing'
 
OSF_COMPLETE.JT = C ! Normal: Obslogs extracted
OSF_COMPLETE.DC = W ! Trigger for the Data Collector (COLLCT)
 
OSF_FAIL.JT = F ! Error: Set the trouble flag
 
POLLING_TIME = 10 ! Wait (seconds) before polling for next
 
INPATH = OMS_OUTPUT_DIR ! Directory of input files.
OUTPATH = OMS_OBSLOGS_DIR ! Directory of obslogs files.
 
MINBLOCKS = 50000 ! blocks required on output disk
 
! forces values from path to be used
 
 ENV.OPUS_DB = OPUS_DB
 ENV.DSQUERY = DSQUERY

==========================================================================================

COLLCT Resource File

 
!--------------------------------------------------------------------
!
! COLLCT RESOURCE FILE
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 06/29/99 38816 Goldst Created
!--------------------------------------------------------------------
 
PROCESS_NAME = COLLCT ! Short name for process id
TASK = COLLCT -P 'PATH_FILE
CLASS = OMS
DESCRIPTION = 'Collect all obslogs for an su'
SYSTEM = OMS
DISPLAY_ORDER = 1
OK_TO_UPDATE_DATABASE = BLANK ! Determined by PATH
INTERNAL_POLLING_PROCESS = TRUE
 
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.JT = C ! Need a 'complete' flag
OSF_TRIGGER1.DC = W ! Need a 'wait' flag
 
OSF_PROCESSING.DC = P ! Set the processing flag to 'Processing'
 
OMS_COMPLETE.DC = C ! Normal: Completion of the COLLCT process
 ! Set DC to a "C" status for each OMS
 ! class OSF until all obslogs are created
 ! per their related SCD class OSF.
 
OSF_FAIL.DC = F ! Error: Set the trouble flag
 
POLLING_TIME = 10 ! Wait (seconds) before polling for next
 
 
DELETE_JIT = Y  ! (Y or N) if DELETE_JIT is "Y" then delete
 ! both the hi-res and low-res jitter file
DELETE_JKX = Y ! (Y or N) if DELETE_JKX is "Y" then delete
 ! all of the *.JKX files for an su.
OUTPATH = OMS_OUTPUT_DIR ! Directory for output files.
 
! forces values from path to be used
 
 ENV.OPUS_DB = OPUS_DB
 ENV.DSQUERY = DSQUERY

==========================================================================================

SAVGEN Resource File

 
!--------------------------------------------------------------------
!
! SAVGEN RESOURCE FILE
!
!
! This file is used to construct the trigger, error, and success
! status fields in the observation status file.
!
!
!--------------------------------------------------------------------
! REVISION HISTORY
!--------------------------------------------------------------------
! MOD PR
! LEVEL DATE NUMBER User Description
! ----- -------- ------ ------ -------------------------------------
! 000 06/29/99 38816 Goldst Created
! 001 07/20/99 38816_01 JSCHULTZ changed mnemonic from OSF_COMPLETE1
! to OSF_COMPLETE
! 002 07/20/99 38816_02 JSCHULTZ change column for OSF_FAIL mnemonic
! from .DC to .SV
!--------------------------------------------------------------------
 
PROCESS_NAME = SAVGEN ! Short name for process id
TASK = SAVGEN -P 'PATH_FILE
CLASS = SCD
DESCRIPTION = 'Produce save files'
SYSTEM = OMS
DISPLAY_ORDER = 1
OK_TO_UPDATE_DATABASE = BLANK ! Determined by PATH
INTERNAL_POLLING_PROCESS = TRUE
 
OSF_RANK = 1 ! First Trigger
OSF_TRIGGER1.XT = C !
OSF_TRIGGER1.SV = W ! Need a 'wait' flag
 
OSF_PROCESSING.SV = P ! Set the processing flag to 'Processing'
 
OSF_COMPLETE.SV = C ! Normal:
OSF_COMPLETE.SC = W ! Trigger for copying of save files
 
OSF_FAIL.SV = F  ! Error: Set the trouble flag
 
POLLING_TIME = 10 ! Wait (seconds) before polling for next
 
INPATH = OMS_OUTPUT_DIR ! Directory of input files.
OUTPATH = OMS_OUTPUT_DIR ! Directory of SAVE files.
MINBLOCKS = 50000  ! blocks required on output disk
 
DELETE_COMBINED = Y ! (Y/N) set to Y to delete combined*.sav
EXTRACT_ACQ = Y ! (Y/N) set to Y to get acq_timing*.oms
EXTRACT_ACS = Y ! (Y/N) set to Y to get acs_save*.oms
EXTRACT_BRE = Y ! (Y/N) set to Y to get breathing*.oms
EXTRACT_COS = Y ! (Y/N) set to Y to get cos_save*.oms
EXTRACT_CST = Y ! (Y/N) set to Y to get costar_save*.oms
EXTRACT_EAR = Y ! (Y/N) set to Y to get earth_avd*.oms
EXTRACT_FGA = Y ! (Y/N) set to Y to get fgsa_servo*.oms
EXTRACT_FGG = Y ! (Y/N) set to Y to get fgsg_servo*.oms
EXTRACT_FGS = Y ! (Y/N) set to Y to get fgs_save*.oms
EXTRACT_FLG = Y ! (Y/N) set to Y to get fgs_flags*.oms
EXTRACT_FOC = Y ! (Y/N) set to Y to get foc_save*.oms
EXTRACT_GEN = Y ! (Y/N) set to Y to get gen_save*.oms
EXTRACT_GSA = Y ! (Y/N) set to Y to get gs_acq*.oms
EXTRACT_NIC = Y ! (Y/N) set to Y to get nic1_save*.oms
EXTRACT_PMT = Y ! (Y/N) set to Y to get pmt_dark_cnt*.oms
EXTRACT_POW = Y ! (Y/N) set to Y to get power*.oms
EXTRACT_STI = Y ! (Y/N) set to Y to get stis_save*.oms
EXTRACT_SUM = Y ! (Y/N) set to Y to get oms_summary*.oms
EXTRACT_THL = Y ! (Y/N) set to Y to get thermal*.oms
EXTRACT_THP = Y ! (Y/N) set to Y to get theta_p*.oms
EXTRACT_WD1 = Y ! (Y/N) set to Y to get walkdown1*.oms
EXTRACT_WD2 = Y ! (Y/N) set to Y to get walkdown2.oms
EXTRACT_WD3 = Y ! (Y/N) set to Y to get walkdown3.oms
EXTRACT_WF2 = Y ! (Y/N) set to Y to get wfii_save*.oms
EXTRACT_WF3 = Y ! (Y/N) set to Y to get wf3_save*.oms
 
! forces values from path to be used
 
 ENV.OPUS_DB = OPUS_DB
 ENV.DSQUERY = DSQUERY

---

Pipeline Database Usage

• Database Relations Accessed
• WIEPHEMERIS Relation
• OMS_SU_TABLE Relation
• SCD_OMS_MAP Relation
• SCD_SAVE_DATE Relation

· Database Queries Performed

o XTRACT Process Queries

o COLLCT Process Queries

WIEPHEMERIS Relation

This relation is used to obtain ephemeris information used by the XTRACT process to perform calculations. (See XTRACT Query)

 
 Field name type size  description 
 ---------- ---- ---- -----------
  epoch_time float 8
 sd_mean_anom float 8
 cir_vel float 8
 cos_orb_incl float 8
 eccentricity float 8
 two_x_ecc  float 8
 ecc_cbd_x_ft float 8
 ecc_sqd_x_fh float 8
 ecc_cbd_x_t float 8
 fd_mean_anom float 8
 rt_node_rad float 8
 rt_node_rev float 8
 arg_perigee float 8
 mean_anomoly float 8
 rt_chng_peri float 8
 rt_asc_node float 8 
 sin_orb_incl  float 8
 semilat_rect float 8
 time_effect float 8
 sdma3sq float 8
 hstorb float 8 

OMS_SU_TABLE Relation

This relation is used to keep an account of the telemetry format of processed subset files and to indicate what subset files were processed under what scheduling unit (SCD-class OSF). The XTRACT process updates the "tlmfmt" and "status" fields in this relation. (See XTRACT Query 1) and (See XTRACT Query 2)

 
 Field name type size  description 
 ----------  ---- ---- -----------
 suname char 30 name of telemetry subset file
 
 sustart char 56 calender start time of scheduling unit; 
 format is YYYY.DDD:HH:MM:SS.sss
 
 sustop char 56 calender stop time of scheduling unit; 
 format is YYYY.DDD:HH:MM:SS.sss
 
 tlmfmt char 5 format of telemetry subset
 
 status char 4 status of subset; S=scheduled,
 C=completed processing by the XTRACT
 process 
  sutime SYBASE_DATETIME current system time in SYBASE_DATETIME
 format; MMM DD YYYY HH:MM(AM/PM)
 
 Example field values are as follows:
 
 suname - SH8S2306W.VC0_DONE
  sustart - 1997.240:28:48:31.182
 sustop - 1997.240:23:06:24.807
 tlmfmt - HN
 status - S
 sutime - Aug 26 1997 11:20AM

SCD_OMS_MAP Relation

This relation is used to map individual observations (OMS-class OSFs) to the Scheduling Unit (SCD-class OSF) the observations were processed under. It also maps observations to a specific invocation of the XTRACT process in which the OMS-class OSFs for the observations were created. This information tracks the origin of exposure obslog files which are identified by rootname using the time of processing of a scheduling unit (SU). This extraction time is the same for every obslog processed under an SU. The name of the telemetry subset file used as the source of the obslog is also present and well as the completion time of that obslog. The XTRACT process inserts records into this relation but leaves the "complete_date" field blank. The COLLCT process uses this relation to determine if all the observations processed under a given scheduling unit have been completed; if so, the COLLCT process updates the "complete_date" field. (See XTRACT Query) or (See COLLCT Querries)

 
 Field name type size  description 
 ---------- ---- ---- -----------
 su_name char  23 name of scheduling unit (SCD-Class OSF)
 that an observation was processed under.
 The format of this field is
  YYDDDHHMMSS_YYDDDHHMMSS
 
 extract_date char 17 time the XTRACT Process began processing a
 scheduling unit (SCD-Class OSF). This 
  field distinguishes between multiple
 XTRACT processing runs of the same
 scheduling unit. The format for this field
  is YYYY.DDD:HH:MM:SS
 
 rootname char 9 observation rootname and OMS-class OSF name
 of the form IPPPSSOOT, where T is always
  'J' and IPPPSSOO matches the science
 rootname of the observation (OMS-class OSF)
 
 subset_name char 13 name of telemetry subset file providing
  data for an observation
 
 multiple char 1 (S/M) Indicates whether a (S)ingle or
 (M)ultiple subset files provided data to
  produce the observation
 
 complete_date char 17 time of completion of jitter processing for
 the individual observation. The date
  format is YYYY.DDD:HH:MM:SS
 
 
 Example field values are as follows:
 
 su_name - 97063042016_97063060327
 extract_date - 1999.214:13:48:28
 rootname - X3SF0109J
 subset_name - SH340347W.VC0
 multiple - S
 complete_date - 1999.214:13:55:25

SCD_SAVE_DATE Relation

This relation is used to map save files to the Scheduling Unit (SCD-class OSF) they were processed under. The XTRACT process inserts records into this relation. (See XTRACT Query)

 
 Field name type size  description 
 ---------- ---- ---- -----------
 su_name  char 23 name of scheduling unit (SCD-Class OSF)
 that an observation was processed under.
 The format of this field is
  YYDDDHHMMSS_YYDDDHHMMSS
 
 save_date char 9 time string used in the file name of SAVE
 files. SAVE files have the format
  AAAAAAAAAAAAAYYDDDHHMM.OMS_, where
 AAAAAAAAAAAAA is a fixed-size 13-character
 field, padded with trailing underscores.
  The save_date is the the value YYDDDHHMM in
 the save file naming format. This YYDDHHMM
 matches either the first 9-characters of
 the su_name, the next day boundary 
 (YYDDD0000) if the scheduling unit crosses
 a day boundry, or the time of the first
 telemetry received after gaps having
 durations longer than an hour 
 
 
 Example field values are as follows:
 
 su_name - 97063042016_97063060327
 save_date - 970630420

============================================================================================

XTRACT Process Queries

• Given a specific request time (reqtime), the XTRACT process performs the following query on the WIEPHEMERIS Relation to read ephemeris information used in performing calculations

 
 SELECT epoch_time, sd_mean_anom, cir_vel, cos_orb_incl, eccentricity, two_x_ecc,
 ecc_cbd_x_ft, ecc_sqd_x_fh, ecc_cbd_x_t, fd_mean_anom, rt_node_rad, rt_node_rev,
 arg_perigee, mean_anomaly, rt_chng_peri, rt_asc_node, sin_orb_incl, semilat_rect,
 time_effect, sdma3sq, hsthorb
 FROM wiephemeris
 WHERE epoch_time = (SELECT MAX(epoch_time) FROM wiephemeris 
 WHERE time_effect < @reqtime and epoch_time < @reqtime)
 

• The XTRACT process performs the following query on the OMS_SU_TABLE Relation to update the "tlmfmt" field for every portion of every subset file listed in a scheduling unit

 
 UPDATE oms_su_table 
 SET tlmfmt = @tlm_format
 WHERE suname = @subset_name AND
 sustart = @start_time AND
 sustop = @stop_time
 

• The XTRACT process performs the following query on the OMS_SU_TABLE Relation to update the "status" field for every portion of every subset file listed in a scheduling unit

 
 UPDATE oms_su_table 
 SET status = @status_value
 WHERE suname = @subset_name AND
 sustart = @start_time AND
 sustop = @stop_time
 

• The XTRACT Process performs the following query on the SCD_OMS_MAP Relation to insert records which indicate what individual observations were processed under what scheduling unit and when the observations were processed.

·  
·  INSERT INTO scd_oms_map
·  su_name, extract_date, rootname, subset_name, multiple, complete_date
·  VALUES (@su_name, @extract_date, @rootname, @subset_name, @multiple, @complete_date)
 

• The XTRACT process performs the following query on the SCD_SAVE_DATE Relation to insert records which indicate which save files were produced under which scheduling units

 
 INSERT INTO scd_save_date
 su_name, save_date
 VALUES (@su_name, @save_date)
 

COLLCT Process Querries

• The COLLCT process performs the following query on the SCD_OMS_MAP Relation to obtain the scheduling unit name (su_name) and extract date (extract_date) for the most recent (by extract_date) record matching the given observation rootname (rootname). This information is then used in a seperate query to update the "complete_date" field of the scd_oms_map table for the observation

 
 SELECT su_name, extract_date
 FROM scd_oms_map
 WHERE rootname = @rootname
 AND extract_date = (SELECT MAX(extract_date) FROM scd_oms_map
 WHERE rootname = @rootname)
 

• The COLLCT process performs the following query on the SCD_OMS_MAP Relation to update the "complete_date" field (complete_date) field for the record matching the given observation root name (rootname); scheduling unit (su_name); and extraction date (extract_date).

·  
·  UPDATE scd_oms_map
·  SET complete_date = @complete_date
·  WHERE su_name = @su_name AND rootname = @rootname AND extract_date = @extract_date
 

• The COLLCT process performs the following query on the SCD_OMS_MAP Relation to obtain information used to determine if all observations processed under a given scheduling unit have been completed.

 
 SELECT su_name, extract_date
 FROM scd_oms_map
 WHERE rootname = @rootname
 ORDER BY extract_date DESC
 

• The COLLCT process performs the following query on the SCD_OMS_MAP Relation to determine if all observations processed under a given scheduling unit have been completed (have their complete_date fields populated). When this is the case, the COLLCT process can trigger the SAVGEN process to produce the save files associated with the given scheduling unit

 
 SELECT su_name
 FROM scd_oms_map
 WHERE su_name = @su_name
 AND complete_date <= @blank"
 AND extract_date = @extract_date
 

---