EDPS Engineering Data Receipt (EDR) Pipeline Reference Manual

Contents

EDR Pipeline Overview

The need for an EDR Pipeline

The EDR Pipeline is the Pipeline within the EDPS that is responsible for receiving and decompressing compressed FOF Engineering Telemetry files sent from the HST Control Center System (CCS). This decompression is performed to get the FOF files into a format that enables them to be processed by the EDPS FOF Conversion pipeline.

Intended Audience and Scope

This Reference Manual is intended primarily to give OPUS Operations Personnel information needed to understand the operation of the EDPS EDR Pipeline. This includes information on the EDR Pipeline Input and Output products, the directories and files used, and the processes involved. Hopefully, as envisioned, this manual will be a dynamic document that is updated when EDR Pipeline changes are made.

EDR Pipeline Dependencies

The EDR pipeline is one of two EDPS pipelines that is not dependent upon other EDPS pipelines to function correctly; the EDPS PASS Data Receipt (PDR) Pipeline being the other one. The EDR and PDR pipelines however are dependent upon systems outside of the EDPS. The EDR pipeline is dependent upon the HST Control Center System (CCS) to provide notification that HST Engineering Telemetry data is available for processing and to provide the Engineering telemetry itself. Also, the EDR Pipeline and the OPUS EDPS in general for that matter are highly dependent upon database interaction; see EDR Pipeline Database Usage. In order for the EDR pipeline database interaction to occur correctly, the processes in the pipeline should be executed in a pipeline mode as opposed to an interactive mode. Typically, as specified by the OK_TO_UPDATE_DATABASE mnemonic in process resource files, processes that run in interactive mode are prohibited from performing database updates. Therefore, to ensure correct operation of the pipeline, !!! ALL EDR PIPELINE PROCESSES SHOULD BE RUN ONLY IN THE PIPELINE MODE !!!.

EDR Pipeline Nominal Operational Description

The EDR pipeline remains idle until it receives Product Notification Message (PNM) files from the CCS. The PNM files contain lists of all the FOF files currently ready to be processed by the EDPS. To insure that the EDR pipeline does not try to access a FOF file before it is available on disk, the CCS transfers all the FOF files listed in a PNM file prior to sending the PNM file to the EDR pipeline. When a PNM file is received, the EDR pipeline performs database queries to determine if the PNM file is a duplicate; i.e., if the PNM file had been previously received or if it listed FOF files that had been previously received. If the PNM file is a duplicate, it will be renamed as such and a 'pnm' class OSF will be created that aslo indicates that the PNM file is a duplicate. Duplicate PNM files will not be processed by the EDR pipeline without manual intervention from an operator. This manual intervention is described in the PNMFOF Process Description section later in this document. When a PNM file is not a duplicate, the EDR pipeline creates a 'pnm' class OSF for the PNM file and 'fof' class OSFs for each FOF file listed in the PNM file. The pipeline also inserts information into the database to indicate what PNM files have been received and what FOF files are listed in each PNM file. The EDR pipeline then verifies that each FOF file listed in a given PNM file exists on disk. After it verifications succeed, the EDR pipeline decompresses each FOF file to get it into a form useable by the EDPS FOF Conversion pipeline. As it is decompressing the FOF files, the EDR pipeline updates the database to keep track of which FOF files have been decompressed. Once all the FOF files listed in a PNM file have been decompressed, the PNM file and its corresponding OSF as well as the OSFs for each decompressed FOF file are deleted as they are no longer needed. When the EDR pipeline if finished processing a given PNM file, the decompressed versions of each FOF file listed in the PNM file will be on disk and be ready to be processed by the EDPS FOF Conversion pipeline.

EDR Pipeline Directories and Files

EDR Pipeline Processes

EDR Pipeline Processes Overview

The EDR Pipeline is composed of the following 4 processes:

  1. PNMFOF - polls the EDR_RECEIPT_DIR directory for Product Notification Message (PNM) files from the CCS system and creates 'pnm' class OSfs for each PNM file and 'fof' class OSFs for each FOF file listed in each PNM file.
  2. ZIPFOF - unzips, i.e., decompresses the compressed FOF files in the EDR_FOF_DIR directory using the 'gzip' command.
  3. CLNFOF - deletes PNM files that have had all of their associated FOF file decompressed. Also, deletes PNM files determined to be duplicates as well as deleting all of the FOF file associated with the duplicate PNM files.
  4. EDRDEL - deletes 'pnm' class and 'fof' class OSFs after EDR pipeline processing has successfully completed.

EDR Pipeline Processes Details

PNMFOF Process details

ZIPFOF Process details

CLNFOF Process details

  • EDRDEL Process Description
    • - This process resides in the (DL) stage of the EDR Pipeline. It is an OSF triggered process and its purpose is to delete 'pnm' and 'fof' class OSFs from the pipeline after their associated processing has completed. EDRDEL is triggerred by a 'c' in the (DL) stage of an OSF. When EDRDEL is triggerred, it looks in its resource file to determine which OSFs to delete. The following resource file mnemonics are used in making the determination: 
        OSF_TRIGGER1.DL      = c        - value of (DL) stage that triggers EDRDEL
    OSF_TRIGGER1.DATA_ID = pnm      - class of OSF to delete
    OSF_TRIGGER2.DL      = c        - value of (DL) stage that triggers EDRDEL
    OSF_TRIGGER2.DATA_ID = fof      - class of OSF to delete

  • EDRDEL Process Triggers
    •      Input Triggers:

        EDRDEL is triggerred by the the CLNFOF Process for 'pnm' class OSFs

        PN   ZI   CL   DL       Class
        --   --   --   --       -----
         c    n    c    c       pnm

        EDRDEL is triggerred by the the CLNFOF Process for 'fof' class OSFs

        PN   ZI   CL   DL       Class
        --   --   --   --       -----
         c    c    c    c       fof

  • EDRDEL Process I/O - Input to and Output from the EDRDEL process is as follows:
    •        INPUT:
  	OPUS_OBSERVATIONS_DIR:
	  'pnm' and 'fof' class OSFs to be deleted

       OUTPUT:
          N/A

  • EDRDEL Process Modes - The EDRDEL process invokes an OPUS pipeline generic executable to perform its work; i.e., the 'osfdelete' executable. This executable can only be executed in a pipeline mode.
    •     Pipeline Mode:

       osfdelete -p opus_definitions_dir:your.path -r edrdel (in task line of resource file)

          where:
                 -p = denotes path file specification follows
	         -r = denotes resource file for the EDRDEL Process
                 opus_definitions_dir:your.path = path file to use

  • EDRDEL Process Resource File
    • 
    !--------------------------------------------------------------------
    !
    ! edrdel.resource
    !
    ! Purpose: This file is used to construct the trigger, error, and
    !          success status fields in the observation status file.
    !
    !       This resource file uses an OSF trigger.
    !
    !--------------------------------------------------------------------
    !                          REVISION HISTORY
    !--------------------------------------------------------------------
    !           PR
    ! DATE     NUMBER  User   Description
    ! ------   ------ ------- ------------------------------
    !  MOD             PR
    ! LEVEL   DATE   NUMBER  User   Description
    ! ----- -------- ------  ------ -------------------------------------
    !  000  10/24/01 44684   Goldst Created initial version
    !  001  03/12/02 45016   Goldst Added TRIGGER2 and DATA_ID
    !--------------------------------------------------------------------

    PROCESS_NAME = edrdel
    TASK         = <osfdelete -p $PATH_FILE -r edrdel>
    DESCRIPTION  = 'Delete OSFs from the BB'
    SYSTEM       = EDR
    CLASS        = all

    DISPLAY_ORDER = 1

    !---------------------------------------------------------------------------
    ! EVNT resource.
    !---------------------------------------------------------------------------

    OSF_RANK = 1                    ! OSF event ordering.

    OSF_TRIGGER1.DL = c             ! Archive completed will trigger OSF deletion
    OSF_TRIGGER1.DATA_ID = pnm      ! Trigger1 class ID

    OSF_TRIGGER2.DL = c             ! Archive completed will trigger OSF deletion
    OSF_TRIGGER2.DATA_ID = fof      ! Trigger2 class ID

    POLLING_TIME = 5                ! Response time of the application

    !---------------------------------------------------------------------------
    ! Application Specific resource
    !---------------------------------------------------------------------------

    OSF_PROCESSING.DL = p           ! letter to be used when an OSF is processed.
    OSF_ERROR.DL      = e           ! letter to be used when there is an error.

    EDR Pipeline Database usage

    Database Relations Accessed

    Database Queries Performed

    Database Relations accessed

    pnmfof_table Relation

         This relation is used to track the transfer and processing of FOF files designated by a
     PNM (Product Notification Message) from CCS.  It is used to indicate what FOF files are
     contained within what PNM file; to determine if a given PNM file or FOF file had been
     previously received; i.e., is a duplicate; and to determine when all of the FOF files
     listed in a PNM file have been decompressed.

     The EDR Pipeline PNMFOF process is the process that inserts new records
     into this relation.  PNMFOF provides values for all of the relations fields except the
     fof_complete_date field as it is the EDR Pipeline CLNFOF process that
     provides values for this field.


       	Field name               type/size    description
        ----------               ---------    -----------
        pnm_name                 C17          PNM file name

        pnm_notification_date    C22          PNM File notification Date in yyyy.ddd:hh:mm:ss.sss

        pnm_receipt_date         C22          PNM File receipt Date in yyyy.ddd:hh:mm:ss.sss

        fof_name                 C15          FOF file name

        fof_transfer_date        C22          FOF File transfer Date in yyyy.ddd:hh:mm:ss.sss

        fof_complete_date        C22          FOF File competion Date in yyyy.ddd:hh:mm:ss.sss

    product_eng_map Relation

         This relation is used to identify the FOF engineering telemetry files that need to be converted to provide
     intermediary telemetry files containing telemetry parameters needed for FGS, GSA or AST product generation.
     It is used to simplify creation and collection of telemetry for processing.  When a product has no
     eng_ready = N flags, then the OSF for the product can be created using the product rootname.

     The EDR Pipeline PNMFOF process uses this relation to indicate which FOF telemetry
     files have not yet been decompressed by the ZIPFOF process and thus are not ready to be processed by the
     EDPS FOF Conversion pipeline.


       	Field name           type/size	 description
        ----------           ---------   -----------
        product_rootname     C14         IPPPSSOOT for jitter or astrometry products;
                                         GYYYYDDDHHMMSS for GS acquisition data

        product_type         C3          FGS for jitter, AST for astrometry, or GSA for GS
                                         acqusition data

        eng_rootname         C12         TYYYYDDDHHMM, rootname of the ENG telemetry file


        eng_ready            C1          (Y/N), 'N' indicates the telemetry file eng_rootname is not
                                         yet recognized as ready for this product_type. 'Y' indicates
                                         that the processing for this product type has recognized
                                         the presence of eng_rootname. Having separate control for
                                         each product type simplifies initiation of processing. "

    Database Queries Performed

    PNMFOF Process Queries

    CLNFOF Process Queries