OMS Data Receipt (ODR) Pipeline Reference Manual

Contents

ODR Pipeline Overview

Intended Audience and Scope

This Reference Manual is intended primarily to give OPUS Operations Personnel information needed to understand the operation of the ODR Pipeline. This includes information on the ODR Pipeline Input and Output products, the directories and files used, and the processes involved. Although this document is partially a consolidation of information contained in the ODR Pipeline Requirements Document and the Design Documents for each of the respective ODR Pipeline processes, additional information not previously presented is provided. Also, it is envisioned that this document is to be a living document that will be updated as ODR Pipeline modifications are made. This in not the case for the Requirements and Design documents; they are and will remain static.

The need for an ODR Pipeline

In producing its output products, the OMS system requires use of Change-Only Engineering Telemetry Subset Files. These files have historically been generated by the by the AEDP system at GSFC and transferred to the OMS system at STScI via an AEDP Subset transfer mechanism. In early 1999, the AEDP system at GSFC is to be retired and a new system, i.e., the Vision 2000 Control Center System (CCS) is to come on line. This new system does not provide Change-only Engineering Telemetry Subset Files directly however it does provide Merged-All-Points Engineering Telemetry Data Files from which Change-only Engineering Telemetry Subset Files can be generated. In the CCS era, obtaining All-Points Engineering Telemetry Data Files from CCS, and conversion of the data files into Change-Only Engineering Telemetry Subset Files is the responsibility of the OMS system itself. Hence, the OMS system ODR pipeline was designed and developed to fulfill these responsibilities.

ODR Pipeline Nominal Operational Description

When the CCS system has Merged-All-Points Engineering Telemetry Data Files available for processing, it notifies the ODR Pipeline of such by sending Product Notification Message (PNM) files to the pipeline. These PNM files are text files specifying what Telemetry Data Files are available and where the Telemetry Data Files reside within the CCS system. The available Telemetry Data Files are binary files in a CCS Front End Process Output Format (FOF). Each FOF file is in a GZIP compressed state and contains an hours worth of telemetry; about 200Mbytes. Multiple FOF files may be specified in each PNM file.

Upon receiving a PNM file, the ODR Pipeline creates and executes FTP scripts to transfer the PNM specified FOF files from the CCS system into specific ODR Pipeline directories. Once transferred, the FOF files are de-compressed and the telemetry data within the files is used to produce the Change-Only Engineering Telemetry Subset Files and Telemetry Error Files used by the OMS Pipeline to produce its output products.

As a final step, after all of the FOF files for a given PNM have been successfully processed, i.e., Engineering Telemetry Subset Files and Telemetry Error Files have been produced from each FOF file in a PNM, the ODR Pipeline performs a "clean-up" procedure to determine if no longer needed PNM Files, FTP Scripts, and FOF files can be deleted.

ODR Pipeline Directories and Files

            *** The file naming convention fields indicated above are decoded as follows:
                yyyy = 4 digit year
                 ddd = 3 digit day of year (001 - 365)
                  hh = 2 digit hour of day (00-23)
                  mm = 2 digit minute of hour (00-59)
                  ss = 2 digit second of minute (00-59)

                   y = year since 1980 (1-Z)
                   m = month of year (1-C)
                   d = day of month (1-V)

ODR Pipeline Processes

ODR Pipeline Processes Overview

As alluded to in the previous section, the ODR Pipeline is composed of the following 5 processes:

  1. PNMFOF - processes PNM files sent to the ODR Pipeline from the CCS System
  2. FTPFOF - uses FTP scripts to transfer FOF Files and Error Log Files from the CCS system to the ODR Pipeline
  3. ZIPFOF - decompresses (GZIP) FOF Files that were transferred from the CCS system to the ODR Pipeline
  4. CVTFOF - converts FOF Files transferred from the CCS system to the ODR Pipeline into Engineering Telemetry Subsets Files and and Telemetry Error Files
  5. CLNFOF - determines when to delete no longer needed PNM Files, PNM Log Scripts, FOF Scripts, and FOF Files
These processes are designed and implemented to execute on DEC/ALPHA/OPENVMS, DEC/ALPHA/DIGITALUNIX, or SUN/SPARC/SOLARIS platforms; DEC/VAX/VMS platforms are not supported. ODR Pipeline Processes executing on DEC/ALPHA/OPENVMS or DEC/ALPHA/DIGITALUNIX platforms have a different byte-ordering than processes executing in the CCS system. The CCS is a Silicon Graphics UNIX based system having a Big Endian byte order whereas DEC/ALPHA/OPENVMS and DEC/ALPHA/DIGITALUNIX platforms have a Little Endian byte order. Therefore, in scenarios where byte ordering between the CCS system and ODR Pipeline processes is different, the ODR Pipeline processes are responsible for performing any necessary byte swapping. In regards to Floating Point Number Representation, both the CCS system and the ODR Pipeline processes use the IEEE standard so there is no problem in this area.

To enable the ODR Pipeline to feed the Engineering Telemetry Subset Files it produces into the OMS Pipeline, a new OMS Pipeline processes was developed; i.e., the SUBPOL Process. This process constitutes the interface between the ODR and OMS Pipelines. Detailed information on the SUBPOL process is provided in Appendix C. of the ODR Pipeline Requirements Document however a short description follows:

ODR Pipeline Processes Details

ODR Pipeline Database usage

Database Relations Accessed

	pnmfof_table Relation
	=====================
        This database relation, populated by the PNMFOF and FTPFOF processes, 
        is used to keep track of PNM receivals, FOF file transfer dates, and
        what FOF files are listed in what PNMs.  Knowing what FOF files are 
        listed in what PNMs is information that is used by the CLNFOF process
        to determine when it can delete PNM files which list FOF files ready
        for processing.  These type PNM files can only be deleted when all the
        FOF files listed in the PNM have been successfully processed.
                                                 
	Field name               type    size	 description  
        ----------               ----    ----    -----------
	pnm_name		 char	  17     pnm file name
	pnm_notification_date 	 char     20     pnm notification time stamp
	pnm_receipt_date	 char     20	 date received time stamp
	fof_name		 char     14	 FOF file name
	fof_transfer_date  	 char     20     FOF transfer time stamp
	fof_complete_date  	 char     20     FOF complete time stamp


        Example field values are as follows:

                        pnm_name  -  PNM_1998100081530 
           pnm_notification_date  -  1998:100:08:15:30 
	        pnm_receipt_date  -  1998.100:08:15:30.000 
                        fof_name  -  t1998095050000      
               fof_transfer_date  -  1999.100:08:17:00.000
               fof_complete_date  -  1999.100:08:27:00.000



	fofsub_table Relation
	=====================
        This database relation, populated by the CVTFOF process, is used to keep
        track of what FOF files are listed in what PNM files and what Subset 
        Files are generated from what FOF files and when.  Knowing what FOF 
        files are listed in what PNM files is information that is used by the 
        CLNFOF process to determine when it can delete PNM files which list FOF
        files ready for processing.  These type PNM files can only be deleted 
        when all the FOF files listed in the PNM have been successfully 
        processed.
                                                 
        A join of this relation with the OMS OMS_SU_TABLE relation can be used
        to provide a mapping between Subset files and the OMS Scheduling Units
        they were processed under.

	Field name               type    size	 description  
        ----------               ----    ----    -----------
	pnm_name		 char	  17     pnm file name
        fof_name                 char     14     name of FOF file
        sub_name                 char     13     name of subset file
        sub_start_time           char     17     start time of subset file 
        sub_stop_time            char     17     stop time of subset file 
        sub_create_date          char     17     time subset file was created


        Example field values are as follows:

                  pnm_name  -  PNM_1998101090030 
                  fof_name  -  t1998095050000   
                  sub_name  -  shba0500w.vc0      
            sub_start_time  -  1997.314:05:00:00
             sub_stop_time  -  1997.314:05:59:59 
           sub_create_date  -  1997.317:09:00:00

Database Querries Performed

ODR Pipeline Stage Failure/Re-Processing Issues

Pictorial view of ODR Pipeline 

	A pictorial view of the ODR Pipeline and a portion of the OMS Pipeline 
        is provided below.  The vertical lines (|) indicate what processes 
        trigger subsequent processes in the respective pipelines either 
        directly via an OSF or indirectly by providing files that are being 
	polled for. 

	ODR Pipeline
	------------
	PNMFOF(PN)
	     |                                
	     FTPFOF(FT)                                  
	          |
        	  ZIPFOF(ZI) 
	               |
        	       CVTFOF(CV) 
	               |    |
	               |    CLNFOF(CL)
	               |
	               |
		       |
	---------------|-------------------------------------------------
	OMS Pipeline   |     
	------------   |
	               |
	               SUBPOL(SP)
	                    |
	                    OMSTLM(TM)
	                         |
	                         OMSCHD(SD)
	                              |
	                              OMSANAL(AL)
	                                    |
	                                    OMSUPD(UD)
	                                         |
	                                         GENREQ(RQ)
	                                              |
	                                              INGRSP(RS)
	                                                   |
	                                                   ARCCLN(CL)