Each HST observation consists of several packets of data which are time-tagged on the spacecraft, sent to the ground by a Tracking and Data Relay Satellite (TDRS) link to Goddard Space Flight Center (GSFC), and sorted by the PACket processOR (PACOR) at GSFC into a packetized data set called a "POD" file (Packetized Observation Data). The individual POD files are sent by PACOR-A to the Operational Processing and Uplink System (OPUS), at STScI. This is the system which is used to process and calibrate the data for retrieval through the HST through the Archive.
"On-The-Fly-Reprocessing" (OTFR) is a software interface to the HST Archive that generates "raw" (uncalibrated) science data files in FITS format from the POD files, and which then applies the most up-to-date calibration of the data possible to the Archive user. Since OTFR starts with the POD files, any changes or additions to header keywords needed by the pipeline software are automatically performed. The OTFR system uses the best available calibration reference files and the most recent versions of the OPUS processing pipeline and calibration software (calnica & calnicb). OTFR is transparent to the Archive user.
OTFR is now the standard mode of data retrieval for observations taken with NICMOS, WFPC2, STIS, and ACS. NICMOS OTFR went into operation on 26 September 2001. Prior to that date, data retrieved from the Archive was processed through the conventional, "static" OPUS pipeline. All new retrievals since that time (including re-retrievals of data taken earlier, i.e., in NICMOS Cycles 7 and 7N) are automatically reprocessed through OTFR.
After OTFR converts the POD files to FITS format and generates the image headers, the images are then processed and calibrated by a suite of software programs known as the pipeline. The purpose of pipeline processing is to provide data products to observers and the HST Data Archive in a form suitable for most scientific analyses. Pipeline processing is also applied to engineering data and calibration data.
The basic sequence of steps in the STScI pipeline system is:
Note that under OTFR, the raw and calibrated FITS data sets are no longer stored in the Archive. They are always regenerated, starting from the POD files and using the most up-to-date reference files and pipeline software, whenever a user requests the data.
The pipeline must also handle exceptions (e.g., incomplete data) and perform a general data evaluation and quality control step. Final delivery of data to observers is accomplished by the data distribution mechanisms of the Archive system.
The calibration step has several goals:
Data retrieval via the Archive and OTFR provides data calibrated to a level suitable for initial evaluation and analysis for all users. Sometimes, however, users may wish to reprocess the data themselves in order to improve the reductions for particular data sets. Observers frequently require a detailed understanding of the calibrations applied to their data and the ability to repeat the calibration process at their home institution. For example, the best calibration reference files for a given data set may not be available until some time after the data were obtained. Although new reductions, using the most up-to-date reference files, can always be obtained by re-retrieving the data via OTFR, it will sometimes be more convenient for users to simply retrieve the new reference files and do the reprocessing locally, on their home computers. Further, certain types of image artifacts can appear in NICMOS data, which require processing with specialized tools to remove. To support these goals, the calibration software is available within the IRAF/STSDAS system and the calibration reference files (e.g., flat fields) are available from the HST Archive via StarView so that observers have the ability to repeat and customize the calibration processing to meet the specific needs of individual observations.
To improve the utility of the pipeline processing for the second and third generation science instruments (NICMOS, STIS, ACS) and future instruments (COS, WF3) several significant changes were made to the structure of the calibration pipeline. The largest of these changes was to enable the combination of multiple observations during the calibration process. This permits the pipeline to both generate a combined product and to use calibrations obtained contemporaneously with the science observations. This capability is designed to support the cosmic ray event removal, mosaicing, and background subtraction for NICMOS observations. As discussed in Chapter 11, mechanisms exist for compactly requesting such observations in the Phase II proposal.
The basic element in the HST ground system has historically been the exposure. The first generation HST science instruments were commanded to generate single exposures, which result from a recognizably distinct sequence of commands to the instrument. This creates a flow of data which is assembled into a single dataset. Each dataset is given a unique 9 character identifier (an IPPPSSOOT in STScI terminology) and is processed by the pipeline, calibrated, and archived separately from all other datasets.
The second generation instruments present many instances in which the combination of data from two or more exposures is necessary to create a scientifically useful data product. NICMOS falls into this category due to several factors. It is strongly recommended to combine exposures in order to remove cosmic rays and to improve flat fielding (by dithering). Further, because the HST thermal background contributes a significant signal at wavelengths longward of 1.9 µm, as well as in especially biased filters for NIC2 and NIC3, multiple exposures (dithered for small targets and offset onto blank sky-chopped-for extended targets) are necessary to measure and remove this background.
Associations exist to simplify the use of HST data by observers. This starts from the proposal phase, continues with a more complete calibration process than would be possible without associations, carries into the archiving and retrieval of associated data, and includes the use of HST data by observers within the IRAF/STSDAS
system.
An association is a set of one or more exposures along with an association table and, optionally, one or more products. We define the following terms:
The first generation instruments all had a one-to-one correspondence between exposures and datasets. They do not have products. NICMOS, STIS and subsequent instruments use the association structure as a meta-dataset. Further, they use the information in multiple exposures during the calibration process to create products.
From a high level, an association is a means of identifying a set of exposures as belonging together and being, in some sense, dependent upon one another. The association concept permits these exposures to be calibrated, archived, retrieved, and reprocessed (within OPUS or STSDAS
) as a set rather than as individual objects. In one sense, this is a book-keeping operation which has been transferred from the observer to the HST data pipeline and archive.
Associations are defined by optional parameters on a single exposure logsheet line. That is, there is a one-to-one correspondence between proposal logsheet lines and associations (although it is possible to have exposures which are not in associations).
Observers may obtain one or more associations at each of one or more positions on the sky using the NICMOS proposal grammar (e.g. dither patterns). Typically usage will be:
A set of predefined patterns are provided in the proposal instructions for these types of observations or a combination of both types ( Chapter 11). The Institute ground system will expand the observer's single line request into multiple exposures each with its own identifying name (IPPPSSOOT) and populate the necessary database relations to describe this association for the OPUS system.
For the second generation and subsequent science instruments the format of the data products from the pipeline is FITS (Flexible Image Transport System) files with image extensions. The IRAF
/STSDAS
system was modified to operate directly on these files. Each NICMOS image is expressed as a set of five image extensions representing the image, its variance, a bit encoded data quality map, the number of valid samples at each pixel, and the integration time at each pixel. This structure is used at all stages of the calibration process which permits the re-execution of selected elements of the pipeline without starting from the initial point. Finally, the calibration code itself is written in the C programing language (rather than IRAF's SPP language). This greatly simplifies the modification of the pipeline code by users and the development of new NICMOS specific data processing tasks. See Section 12.3 NICMOS Data Products and the NICMOS Data Handbook for more information regarding data products and structure.
Space Telescope Science Institute http://www.stsci.edu Voice: (410) 338-1082 help@stsci.edu |