To obtain FOF data from CCS and to produce Subsets from the data, a new OMS pipeline will be used; i.e., The OMS Data Receipt Pipeline. This pipeline is composed of five processes; the details of which comprise the majority of this document. The processes are described in subsequent sections however short descriptions are provided below:
These processes will be designed and implemented such that they are capable of executing on DEC/ALPHA/OPENVMS, DEC/ALPHA/DIGITALUNIX, or SUN/SPARC/SOLARIS platforms; DEC/VAX/VMS platforms will not be supported. The ZIPFOF process however will most likely execute exclusivsly on the SUN/SPARC/SOLARIS platform. Pipeline processes executing on DEC/ALPHA/OPENVMS or DEC/ALPHA/DIGITALUNIX platforms will have a different byte-ordering than processes executing in the CCS system; CCS is a Silicon Graphics UNIX based system having a Big Endian byte order. DEC/ALPHA/OPENVMS and DEC/ALPHA/DIGITALUNIX platforms have a Little Endian byte order. Therefore, in scenarios where byte ordering between involved platforms is different, the OMS Data Receipt Pipeline processes will be responsible for performing any necessary byte-swapping when processing data transferred from CCS. Another area of system incompatibility, i.e., Floating Point Number Representation, is not an issue as both the CCS platform and all of the proposed data receipt pipeline platforms use the IEEE standard.
In conjunction with the new OMS Data Receipt Pipeline, a new OMS pipeline process (SUBPOL) will be added to serve as the interface between the two pipelines. SUBPOL will also be designed and implemented to be capable of executing on DEC/ALPHA/OPENVMS, DEC/ALPHA/DIGITALUNIX, or SUN/SPARC/SOLARIS platforms. A short description of SUBPOL follows:
PNMFOF (Product Notification Message)
=================================
i. Algorithms
This process is responsible for processing PNM files delivered to the
institute via FTP from CCS. The PNM(s) are FTP to an operationally
designated location as per the ICD, and can be any one of the following
transfer types:
TEST - PNM files with a "TEST" transfer type will trigger the
==== following:
o Write test information to process log file
o Delete PNM file.
PRODUCT,FAILURE - PNM files with a "PRODUCT,FAILURE" transfer
=============== type will trigger the following:
o Generate a PNM script of the name PNM_yyyydddhhmmss.sh to
be executed by FTPFOF.
o Generate a PNM OSF of the name PNM_yyyydddhhmmss
PN FT ZI CV CL CLASS
== == == == == =====
C W _ _ _ PNM
note: N/A for standing order deliveries.
PRODUCT,SUCCESS - PNM files with a "PRODUCT,SUCCESS" transfer
=============== type will trigger the following:
o Generate a PNM OSF of the name PNM_yyyydddhhmmss
PN FT ZI CV CL CLASS
== == == == == =====
P _ _ _ _ PNM
o Create a list containing all the FOF files
to be transferred contained in the PNM.
o Generate all the FOF scripts contained in the list with
the convention Tyyyydddhhmmss.sh for FTPFOf to execute.
o Insert a fofpnms record, one for each FOF file, populate
the following fields:
pnm_name
pnm_notification_date
pnm_receipt_date
fof_name
Reference APPENDIX A for a description of the fofpnms
database relation.
o Generate all the FOF OSF contained in the list of the
name Tyyyydddhhmmss
PN FT ZI CV CL CLASS
== == == == == =====
C W _ _ _ FOF
o Update the PNM OSF of the name PNM_yyyydddhhmmss
PN FT ZI CV CL CLASS
== == == == == =====
C N N N _ PNM
Reference APPENDIX B for sections 3.1 and 3.1.1 of the HST-ICD-T2
pertaining to Product Notification Message.
ii. Error Handling
Should a I/O or database error occur during the processing of a
PNM OSF, the PNM OSF will be set to "E".
iii. Mode
This process will allow both pipeline and interactive processing.
iv. Interfaces
1) interactive
eg.
PNMFOF i -f PNM_yyyydddhhmmss.TXT
(this command would be typed into the command line)
PNMFOF: command name
i : denotes interactive processing
-f : file_specification followed by the PNM to be
processed.
pipeline
eg.
PNMFOF -p opus_definitions_dir:your.path
(this command is contained in the process resource file)
PNMFOF: command name
-p : denotes path file will be used to direct the process
opus_definitions_dir:your.path: path file to use
2) input/output
input : PNM_yyyydddhhmmss.TXT PNM file name from CCS.
output: PNM_yyyydddhhmmss.sh log script.
output: Tyyyydddhhmmss.sh fof scripts.
log scripts are executed by FTPFOF to transfer error log files
generated by CCS in response to a incomplete product request.
fof scripts are executed by FTPFOf to transfer FOf data files
generated by CCS in response to a standing product order for
current telemetry.
To examine details of the PNMFOF Process Design, reference the
PNMFOF Process Design Document
FTPFOF (File Transport Protocol)
============================
i. Algorithms
This process is responsible for transferring FOF data, and CCS error
logs via FTP using the script generated by PNMFOF. OFS's marked "W"
for FTPFOF will trigger the execution of the FTP scripts. FTPFOF
will use the "C" SYSTEM function to execute the scripts, transfer the
files, and update the fofpnms record with the following field(FOF
script only).
FOF_TRANSFER_DATE
Reference APPENDIX A for a description of the fofpnms
database relation.
PRODUCT,FAILURE - Transfers the CCS LOG error files generate by
CCS in response to the errors generated while processing the
product request notification.
o Mark the ZIPFOF, and CVTFOF columns with an "N", and
the CLNFOF column with a "W" when FTPFOF completes.
PN FT ZI CV CL CLASS
== == == == == =====
C C N N W PNM
PRODUCT,SUCCESS - Transfers the CCS FOF data files generate by
CCS in response to the standing order.
PN FT ZI CV CL CLASS
== == == == == =====
C N N N _ PNM
C C W _ _ FOF
ii. Error Handling
Should a I/O or database error occur during the processing of a
PMN OSF, the PMN OSF will be set to "F".
Should a I/O or database error occur during the processing of a
FOF OSF, the FOF OSF will be set to "F".
iii. Mode
This process will allow both pipeline and interactive processing.
iv. Interfaces
1) interactive
eg.
FTPFOF i -f PNM_yyyydddhhmmss.sh
(this command would be typed into the command line)
FTPFOF i -f tyyyydddhhmmss.sh
(this command would be typed into the command line)
FTPFOF: command name
i : denotes interactive processing
-f : file_specification followed by the script to be
processed.
pipeline
eg.
FTPFOF -p opus_definitions_dir:your.path
(this command is contained in the process resource file)
FTPFOF: command name
-p : denotes path file will be used to direct the process
opus_definitions_dir:your.path: path file to use
2) input/output
input : PNM_yyyydddhhmmss.sh PNM script.
input : Tyyyydddhhmmss.sh FOf script.
output: CCS_ERROR_FILE.LOG CCS errror log file.
output: Tyyyydddhhmmss.MRG-GZ fof data file.
CCS error logs descibe the CCS errors encountered when processing
a request for product notification.
fof data files are the CCS products which contain the GZIP full
resolution telemetry data.
To examine details of the FTPFOF Process Design, reference the
FTPFOF Process Design Document
=======================
i. Algorithms
This process is responsible for decompressing FOF data transferred
to the institute. OFS's marked "W" for ZIPFOF will trigger the
decompression of the FOF data. ZIPFOF will use the ZIP -d command
to perform this decompression of the FOF data. As stated in the
Overview section, this process will most likely execute exclusivsly
on the SUN/SPARC/SOLARIS platform.
When successfully processed, the OSF will be marked with a "C" and the
CVTFOF process will be marked with a "W".
PN FT ZI CV CL CLASS
== == == == == =====
C N N N _ PNM
C C C W _ FOF
ii. Error Handling
Should a I/O or database error occur during the processing of a
FOF OSF, the FOF OSF will be set to "F".
iii. Mode
This process will allow both pipeline and interactive processing,
and will be restricted to running on unix work stations.
iv. Interfaces
1) interactive
eg.
ZIPFOF i -f Tyyyydddhhmmss.MRG-GZ
(this command would be typed into the command line)
ZIPFOF: command name
i : denotes interactive processing
-f : file_specification followed by the FOf file to be
unziped.
pipeline
eg.
ZIPFOF -p opus_definitions_dir:your.path
(this command is contained in the process resource file)
ZIPFOF: command name
-p : denotes path file will be used to direct the process
opus_definitions_dir:your.path: path file to use
2) input/output
input : Tyyyydddhhmmss.MRG-GZ zipped FOF data file.
output: Tyyyydddhhmmss.MRG unzipped FOF data file.
zipped fof data files are the CCS products which contain the GZIP
full resolution telemetry data.
unzipped fof data files are the CCS products which contain the unGZIP
full resolution telemetry data.
To examine details of the ZIPFOF Process Design, reference the
ZIPFOF Process Design Document
CVTFOF (Convert all-points FOF data to change-only Subsets)
===================================================
i. Algorithms
As stated in the Overview Section, this process is responsible for
converting all-points FOF Engineering Telemetry Data into change-only
Engineering Telemetry Subsets. For the most part, the subsets
generated by this process and the subsets currently generated by AEDP
will contain the same information and be in the same format. The major
difference however is that CVTFOF generated subsets will not contain
S/C Clock to UT conversion data. This difference in content will be
accounted for by processes reading the Subset files; the processes will
be modified to be capable of reading and processing both CVTFOF and
AEDP generated subsets.
A second responsibility of the CVTFOF process is to produce the
Telemetry Error Files that are currently produced by the OMS PREPROC
process. When the OMS Data Receipt Pipeline comes on-line, the PREPROC
process will become inactive; it will however be retained to provide
for re-processing capability of old AEDP subsets should the need arise.
Like PREPROC, CVTFOF will produce a Telemetry Error File for every
Engineering Subset file produced. CVTFOF produced error files will be
in the same format and provide the same information as the PREPROC
produced error files.
To produce Engineering Telemetry Subsets and Error Files, the CVTFOF
process will reside in the OMS Data Receipt Pipeline between the ZIPFOF
and CLNFOF processes (a pictorial view of the pipeline is provided in
appendix D). CVTFOF will be an OSF Poller that is triggered when the
ZIPFOF process inserts a "W" into the CVTFOF (CV) pipeline stage column
as shown below:
PN FT ZI CV CL CLASS
== == == == == =====
C C C W _ FOF
Once triggered, CVTFOF will generate Subset files and corresponding
Error files for the FOF data file specified in the OSF. Multiple
Subset and Error files may result from each FOF file as new files are
created whenever telemetry format changes occur within a FOF file. As
Subsets files are generated, their existence is noted in a database
table; i.e., the "fofsub" table. The fields of this table are described
in APPENDIX A. The Subset and Error files will be generated in a directory
that is an input directory to the OMS Pipeline; this will trigger
processing by the OMS Pipeline. As mentioned in the overview, the OMS
Pipeline SUBPOL process will poll the input directory for pairs of
Subset and Error files. To ensure that SUBPOL does not prematurely
find the existence of the files, the file names will be appended with
"_NEW" until the files are completely constructed, at which time the
"_NEW" will be replaced with "_FOF". Upon finding a corresponding
Subset/Error File pair, the SUBPOL process will create an OSF for the
Subset and trigger the OMS Pipeline OMSTLM process to start processing
of the files. From OMSTLM forward, the process stages of the OMS
Pipeline will remain the same. In addition to triggering the OMS
Pipeline, when CVTFOF processing is complete, a "C" and a "W"
respectively will be inserted into the CVTFOF (CV) and CLNFOF (CL)
pipeline stage columns to trigger the CLNFOF process as shown below:
PN FT ZI CV CL CLASS
== == == == == =====
C C C C W FOF
ii. Error Handling
As CVTFOF processing progresses, any errors encountered will be posted
to the CVTFOF process log file. Any errors causing incomplete
processing of a FOF file, incomplete generation of Subset/Error Files,
or incomplete updating of database tables will result in the CVTFOF
pipeline stage column being set to "F" as shown below:
PN FT ZI CV CL CLASS
== == == == == =====
C C C F _ FOF
iii. Mode
CVTFOF accommodates both pipeline and interactive mode processing.
iv. Interfaces
During normal operation in the OMS Data Receipt Pipeline, CVTFOF is
invoked and executed automatically upon completion of and triggering
by the ZIPFOF process. CVTFOF however can also be invoked
interactively from the command line. Example invocations are as
follows:
1) interactive
eg.
CVTFOF i -f Tyyyydddhhmmss.MRG
(this command would be typed into the command line)
CVTFOF: command name
i : denotes interactive processing
-f : file_specification followed by the FOf file to be
converted to Subset files
pipeline
eg.
CVTFOF -p opus_definitions_dir:your.path
(this command is contained in the process resource file)
CVTFOF: command name
-p : denotes path file will be used to direct the process
opus_definitions_dir:your.path: path file to use
2) input/output
input : Tyyyydddhhmmss.MRG decompressed FOF data file
output: Symdhhmm.vc0_fof Telemetry Subset file(s)
Symdhhmm.er0_fof Telemetry Error File(s)
CVTFOF (CV) Pipeline stage column being set to "C"
CLNFOF (CL) Pipeline stage column being set to "W"
records written to the fofsub database table which is
described in APPENDIX A.
To examine details of the CVTFOF Process Design, reference the
CVTFOF Process Design Document
==============
i. Algorithms
For PNM class OSF:
==================
The PNM file is always deleted no matter what type of transfer.
If the FTPFOF status is not set to N, then an FTP script exists for
the transfer of error log files. In this case, this script is also
deleted. The PNM class OSF that is marked with a "W" will be set to
"C".
At the completion of "PRODUCT,FAILURE" processing, the PNM class OSF
will have the following status.
PN FT ZI CV CL CLASS
== == == == == =====
C C N N C PNM
At the completion of "PRODUCT,SUCCESS" processing, the PNM class OSF
will have the following status.
PN FT ZI CV CL CLASS
== == == == == =====
C N N N C PNM
For FOF class OSF:
==================
This process deletes the FOF transfer script and FOF data files
(.MRG) after they have been succesfully processed by CVTFOF. After the
deletions, it must be determined whether the PNM class can be cleaned.
This involves a query of the fofpnms table to obtain the pnm_name for
this FOF. For any FOF files of this pnm_name, it must be determined if
their CVTFOF processes are still pending. The subfof table is updated
by CVTFOF after it completes processing the entire FOF file. The
following query counts the number of fofs with no subfof records.
SELECT COUNT(*) FROM fofpnms f
WHERE f.pnm_name=@pnm_name and
NOT EXISTS (SELECT * FROM subfof s
WHERE s.fof_name = f.fof_name)
If the missing fof count is zero, then the CLNFOF column of the PNM
class OSF having the pnm_name, is set to 'W'. At the end of all
processing the FOF class OSF that is marked with a "W" will be set to
"C". The completed FOF class OSF will have the following status.
PN FT ZI CV CL CLASS
== == == == == =====
C C C C C FOF
ii. Error Handling
Should a I/O or database error occur during the processing of a
PNM OSF, the PNM OSF will be set to "F".
Should a I/O or database error occur during the processing of a
FOf OSF, the FOF OSF will be set to "F".
iii. Mode
This process will allow pipeline processing only.
iv. Interfaces
1) pipeline
eg.
PNMFOF -p opus_definitions_dir:your.path
(this command is contained in the process resource file)
PNMFOF: command name
-p : denotes path file will be used to direct the process
opus_definitions_dir:your.path: path file to use
To examine details of the CLNFOF Process Design, reference the
CLNFOF Process Design Document
fofpnms Relation
================
Field name type size description
---------- ---- ---- -----------
pnm_name char 17 pnm file name
pnm_notification_date char 20 pnm notification time stamp
pnm_receipt_date char 20 date received time stamp
fof_name char 14 FOF file name
fof_transfer_date char 20 FOF transfer time stamp
fofsub Relation
===============
This database relation, populated by the CVTFOF process, is used to
track what Subset Files were generated from what FOF files and when.
In addition, a join of this relation and the OMS OMS_SU_TABLE relation
can be used to provide a mapping between Subset files and the OMS
Scheduling Units they were processed under.
Field name type size description
---------- ---- ---- -----------
fof_name char 14 name of FOF file
sub_name char 13 name of subset file
sub_start_time char 17 start time of subset file
sub_stop_time char 17 stop time of subset file
sub_create_date char 17 time subset file was created
Example field values are as follows:
fof_name - t1997314050000
sub_name - shba0500w.vc0
sub_start_time - 1997.314:05:00:00
sub_stop_time - 1997.314:05:59:59
sub_create_date - 1997.317:09:00:00