Advanced Camera for Surveys Instrument Handbook for Cycle 14
 |
 |
 |
 |
 |
|
| Advanced Camera for Surveys Instrument Handbook for Cycle 14 | |||||
| ||||||
11.1 OverviewChapter 11:
Pipeline Calibration
11.1.1 On The Fly Reprocessing (OTFR)
11.1.2 When is OTFR not Appropriate or Sufficient?
11.1.3 Distortion Correction and Dither Combining
11.2 ACS Pipeline
11.3 ACS Data Products
11.4 System Requirements for ACS Data
This Chapter describes the ACS pipeline's calibration software, CALACS and MultiDrizzle. Developed at STScI, CALACS removes instrumental signatures, combines CR-SPLIT or REPEAT-OBS exposures and updates certain header keyword values when calibrating ACS data. MultiDrizzle completes the calibration process by removing uncalibrated hot-pixels and cosmic-rays when combining associated images and by removing geometric distortion from all images when creating a final calibrated, possibly combined, product. This chapter is meant only as a high level overview. The "
HST Data Handbook for ACS" version 3.0, July 2004, provides a more thorough discussion of data reduction and analysis issues for ACS.11.1 Overview
11.1.1 On The Fly Reprocessing (OTFR)
All data taken by HST are run through STScI's calibration pipeline. This consists of two main software systems, the Operations Pipeline Unified System (OPUS) and the Data Archive and Distribution System (DADS). Raw spacecraft telemetry from Goddard Spaceflight Center (GSFC) is transmitted to STScI in the form of POD files. When a user requests data from the archive through Starview or the archive web interface, OTFR uses these POD files as input to the OPUS step named Generic Conversion generating the uncalibrated "raw" data. CALACS and MultiDrizzle is then run by OPUS to process the uncalibrated data, using specific ACS reference images and tables from the Calibration Data Base System (CDBS), into calibrated data. DADS populates a database from these data that is accessible to users via StarView. DADS then distributes any data requested for download to the user completing the OTFR process.
OTFR is now the standard way to process data requested from the STScI archive. It provides the best calibrated products by reprocessing the raw spacecraft telemetry files through OPUS "on the fly" for distribution each time any data is requested. Exceptions to this include observations expected to be heavily requested by the community for which the archive will maintain current calibrated versions for quicker access.
The most appropriate versions of the ACS reference files are used by CALACS and Multidrizzle each time OTFR is run. Since reference files such as CCD biases and darks are frequently updated, OTFR will use different reference files depending on the date of reprocessing. The user waits until the contemporaneous reference files are in place, and requests the data via StarView. Dark and bias reference files will be updated on a regular basis (usually within 2-3 weeks). The uncalibrated and calibrated data's header keywords are updated with the filenames of the reference files used during that specific OTFR run.
OTFR also enables the user to avoid downloading outdated archived data due to the software changes made for bug fixes, improved algorithms, new capabilities or header keyword changes. Once a code change is made, OTFR will reprocess and distribute the corrected data using the latest software versions available.
OTFR can distribute all files associated with an ACS observation, including raw and uncalibrated. It will also allow the users to select certain parts of the dataset for download, such as only the uncalibrated or only the calibrated data, when the user does not need all the products.
11.1.2 When is OTFR not Appropriate or Sufficient?
Although OTFR produces a data product which will be suitable for many uses there are several occasions when it is not ideal and offline interactive processing by the user is required. The main most frequent reasons for this are:
- Running CALACS or Multidrizzle with personal versions of reference files
- Running CALACS with non-default calibration switch values
- Running Multidrizzle with improved pointing information or other non-default parameter settings
OTFR will always use the most appropriate ACS calibration reference files by default. In order to use non-default calibration reference files, manual re-calibration is required. The calibration reference file keywords in the uncalibrated data need to be updated manually with the non-default filenames before running CALACS.
The selection criteria in table 11.1 are used to set the values for the calibration switch header keywords in uncalibrated ACS data. In order to use non-default calibration switch values, manual re-calibration is required. The calibration switches in the uncalibrated data need to be updated with the non-default values before running CALACS.
A few of the calibration capabilities, specifically an analogue-to-digital conversion correction controlled by ATODCORR, and a correction for shutter shading (gradient of exposure time at short values) as controlled by SHADCORR are not invoked. These capabilities are supported in CALACS code, but instrument characterization has shown that these corrections are fortunately not needed for ACS, and supporting reference files do not exist. The switches for these should therefore always be left as OMIT.
Table 11.1: Calibration Switch Selection Criteria
The goal of the ACS pipeline is to provide data calibrated to a level suitable for initial evaluation and analysis for all users. Observers frequently require a detailed understanding of the calibrations applied to their data and the ability to repeat, often with improved calibration products, the calibration process at their home institution. Therefore, the CALACS and PyDrizzle packages used in this pipeline can also be used to calibrate ACS data off-line and are available within the STSDAS system. In addition, the calibration reference files (e.g. flat fields) are available from the HST Archive via the
Archive web pages. The most recent version of STSDAS, which includes CALACS and MultiDrizzle, can be downloaded athttp://www.stsci.edu/resources/software_hardware/stsdas. A tutorial for running CALACS can also be viewed athttp://www.stsci.edu/hst/acs/analysis.Many ACS datasets will not have multiple exposures at the same pointing. The increasing prevalence of hot pixels, particularly in the WFC camera, make dithering in most cases highly desirable. This use of dithering circumvents the algorithms built into CALACS for detecting and removing cosmic-rays. To address this problem, the task MultiDrizzle, originally designed to be run interactively (Koekemoer et al. 2002 HST Calibration Workshop), has been developed to run within OTFR. It will be used in OTFR to process all ACS datasets and remove hot-pixels and cosmic-rays from all associated ACS images, regardless of whether they were dithered or taken as CR-SPLIT or REPEAT-OBS pointings, while also correcting for geometric distortion.
MultiDrizzle provides a single-step interface to the complex suite of operations originally performed by the tasks in the STSDAS dither package. These are used for initial image registration, creation of a clean median image, transformation back to the input image plane, creation of cosmic ray masks, and final drizzling. The goal of MultiDrizzle is to provide a high-level task which has an extensive suite of user-adjustable parameters which, if left at their default values, will allow the task to perform all these steps in a single operation with no user intervention. At the same time, the parameters allow the user a large amount of flexibility in controlling the relevant aspects of these steps, in case the default parameters are not sufficient for specific scientific applications. MultiDrizzle relies on PyDrizzle extensively to control the operation of the task 'drizzle' which performs the actual distortion correction and image combination.
MultiDrizzle relies on the image header world coordinate system (WCS) to deduce the image-to-image offsets. However, user-supplied offsets may also be applied in a flexible way in offline use, offsets computed using whatever tasks the user finds most appropriate and suitable for their data. For OTFR use, though, image header WCS information for images taken as part of a defined association generally provides image registration good enough for initial analysis.
The most recent version of MultiDrizzle, updated since the last release of STSDAS, can be downloaded from:
http://www.stsci.edu/resources/software_hardware/pydrizzleIn addition, examples of using Multidrizzle with ACS data, along with available documentation, can be found at:
http://www.stsci.edu/hst/acs/analysis/drizzle.11.1.3 Distortion Correction and Dither Combining
All ACS data will automatically be corrected for distortion during standard pipeline processing using a task called PyDrizzle called from MultiDrizzle. PyDrizzle relies on the IDCTAB reference file for the description of the distortion model. It also understands the ACS association tables to allow MulitDrizzle to combined dithered observations in the pipeline. Observation sets which use CR-SPLIT or REPEAT-OBS options for a single pointing or use the dither patterns provided in the proposal instructions will be automatically associated for combining in the pipeline. Programs which rely on explicit POS TARG commands will NOT be associated in the pipeline, resulting in separately calibrated image for each position. PyDrizzle, as called by MultiDrizzle, automatically produces images which are both astrometrically and photometrically accurate regardless of whether they were taken as part of an association or as a single exposure. Calibrated individual images, with the _flt.fits extension (FLT files) are also produced by the pipeline and are used as the input for MultiDrizzle.
It is important to note that PyDrizzle, when combining dithered images, does NOT remove cosmic-rays. MultiDrizzle was developed to perform this additional work of removing cosmic-rays from associated images, relying on PyDrizzle to perform the actual distortion correction and image combination.
For ACS WFC observations, both WFC chips will be combined into a single image in the output by PyDrizzle. All ACS distortion-corrected, possibly dither-combined, images will have a single SCI extension, a weight (WHT) extension, and a context (CTX) extension, replacing the standard SCI, ERR and DQ arrays in the CALACS calibrated products. The WHT extension records the combined input weights for each output pixel. In standard processing, the values will be the effective exposure time for each output pixel. The CTX extension encodes information about which input image contributes to a specific output pixel. This is done using a bitmask for each output pixel where `bit set' means that the image, in the order they were combined, contributed with non-zero weight to that output pixel. The context image starts as a 32-bit integer image but is extended as a cube with additional 32-bit deep planes as required to handle all the input images.
Processing comments are recorded in the trailer file for the DRZ image, including which version of drizzle was used, what parameters were used, and which images were combined (if dithered). The same default parameters, however, are used for all observations in the pipeline. These parameters were chosen to avoid introducing any scale changes, or shifts relative to the original point while returning a corrected product oriented with North aligned with the Y axis for most ACS images. The size of a single, distortion-corrected image, prior to rotating it to align with North, can be found in Table 11.2. These products will be properly corrected for distortion. However, for dithered observations, the combination may not be ideal as many defects resulting from small pointing errors may still be present. Subsequent reprocessing with PyDrizzle or MultiDrizzle offline, using updated image registration, may be required to obtain the desired scientific value. PyDrizzle and MultiDrizzle can be obtained as part of STSDAS, which requires the PyRAF environment to run. These packages can also be obtained from the
Figure 11.1: Flow Diagram for ACS data shown with CALACS task names.STSDAS webpages.
11.2 ACS Pipeline
The ACS calibration pipeline consists of 2 major packages run one after the other: namely, CALACS and MultiDrizzle. The CALACS package itself consists of 4 calibration tasks which can all be run separately on individual exposures. Since ACS can also produce associated data, such as
CR-SPLITorREPEAT-OBSexposures, the task CALACS can be used to process these associated exposures, or even individual exposures, automatically by calling the 4 individual tasks in the package as needed. The PyDrizzle package then processes the output from CALACS, both single exposures and associated datasets. These tasks apply the basic calibrations necessary for ACS data. The flow of data through the ACS pipeline, and what decisions are made while working with associated data, can be seen in Figure 11.1.The following calibration steps are performed in order for ACS data:
Figure 11.2: Flow diagram for CCD data in CALACS
- Calculate a noise model for each pixel and record estimated noise value in error array
- Flag known bad pixels and saturated pixels in the data quality array
- Subtract bias-level determined from overscan regions (CCD data only)
- Subtract bias image (CCD data only)
- Subtract post flash image (CCD data only)
- Perform cosmic-ray rejection and combination of
CR-SPLITdata (CCD data only)- Perform global linearity corrections (MAMA data only)
- Scale and subtract dark image and calculate mean dark value
- Perform flat-fielding
- Calculate values for photometry keywords
- Calculate image statistics
- Remove cosmic-rays, combine associated observations, and apply distortion correction (using MultiDrizzle)
As shown in Figure 11.1, the calibration tasks have been split to handle CCD-specific calibrations separately from those steps which can be applied to any ACS data. MAMA data obtained from the SBC do not have the overscan regions found in CCD data, and therefore those steps pertaining to the use of the overscan regions were split into a separate task. The initial processing performed on CCD data alone is shown in Figure 11.2, with the result being processed like the rest of the ACS data through the processing steps shown in Figure 11.3.
Figure 11.3: Flow diagram for MAMA and CCD data in CALACS.
11.3 ACS Data Products
The inputs for CALACS processing come out of the Generic Conversion stage of OTFR and include:
Figure 11.4: Data formats for Calibrated and Drizzled ACS Modes. Note that for calibrated data, WFC1 (chip 1) corresponds to extension [SCI,2] (or equivalently, extension [4]).
- raw exposure - FITS formatted, 16-bit integer data (see Figure 11.4)
- association table (only for associated data)
- trailer file from Generic Conversion (optional for reprocessing)
Processing Single Exposures with CALACS
Processing single exposures will result in the creation of a single fully calibrated ACS exposure. Figure 11.4 illustrates the basic format used to store ACS images, however, differences exist between uncalibrated and calibrated observations. A single, uncalibrated (RAW) ACS exposure starts out with a Primary header plus, for each chip, a SCI extension with the data in 16-bit integer format, an empty ERR extension, and a DQ extension which may or may not have 16-bit integer data. CALACS then processes this observation using the steps outlined in Figure 11.2 and Figure 11.3 to generate the fully calibrated (FLT) image. This FLT image consists of a Primary header, and again for each chip, a SCI extension with 32-bit float data, an ERR extension with 32-bit float data, and a DQ extension with 16-bit integer data. The data in the calibrated SCI extension and the corresponding ERR extension have units of ELECTRONS, after accounting for the gain correction performed during the flat-field correction.
Processing Associated Exposures with CALACS
The CALACS will also recognize and correctly process
CR-SPLITorREPEAT-OBSexposures by interpreting the association table and determining which exposures should be combined during calibration. As illustrated in Figure 11.1,REPEAT-OBSexposures are individually processed to create an FLT file for each input exposure, then those FLT images are summed to create the final (SFL) product. However, since these exposures are not run through ACSREJ, no cosmic-ray rejection is performed whenREPEAT-OBSimages are summed together to form the final SFL product.Input RAW exposures taken for a CR-SPLIT association, on the other hand, are only calibrated using ACSCCD, then they are combined into a single exposure during cosmic-ray rejection with ACSREJ. The final calibration steps are then run on cosmic-ray cleaned products to produce the final calibrated, cosmic-ray cleaned (CRJ) image. Thus, processing an association results in a single CALACS-calibrated product created from combining the individual exposures in the association.
Since the same processing is performed on combined images as single exposures, the data in the calibrated SCI extension and the corresponding ERR extension also have units of ELECTRONS for both REPEAT-OBS and CR-SPLIT data.
MultiDrizzle Processing in the Pipeline
The final calibration performed during pipeline processing corrects for the geometric distortion and pixel area effects in the flat-fielded calibrated observations produced by CALACS. It also removes cosmic-rays and combines all associated observations, whether taken as CR-SPLIT, REPEAT-OBS or dither patterns, into a single distortion-corrected product. Both steps are performed using MultiDrizzle in the pipeline with the flat-fielded (FLT) files from CALACS as input.
A single ACS exposure calibrated by CALACS results in an FLT file. This image can be processed by MultiDrizzle only by running PyDrizzle to remove the effects of the geometric distortion to produce a drizzle calibrated (DRZ) images. This DRZ image is a multi-extension FITS file with the format shown in Figure 11.4 for the DRZ product. The SCI extension contains the distortion-corrected data as 32-bit float data. The weight image gets stored as a WHT extension in the form of 32-bit float data, while the final CTX extension contains the 32-bit integer data for the context data.
Associations can be produced for dither patterns, CR-SPLIT, or REPEAT-OBS exposures or a combination of CR-SPLIT or REPEAT-OBS exposures at each point in a dither pattern. CALACS, though, will always produce an FLT image for each input in the association, regardless of the pointing, while CRJ or SFL images will also be produced for each CR-SPLIT or REPEAT-OBS pointing, respectively. MultiDrizzle processes only the FLT images from the association to remove the cosmic-rays, apply the correction for geometric distortion, then combine them into a single combined DRZ product with the same format as a single calibrated exposure. More details on this can be found in Chapter 4 of the "HST Data Handbook for ACS".
11.4 System Requirements for ACS Data
The large size of ACS WFC exposures may present problems for observers using ACS, especially when dealing with data that requires associations. Raw ACS exposures which serve as input to the pipeline have the sizes given in Table 11.2.
The total size of a WFC image includes both the SCI arrays, while the HRC and SBC detectors only have one chip/array. The file sizes given in Table 11.2 presume that both the SCI array and DQ array are populated with short integer values, but that the ERR array is NULL with all pixels having a value of zero. During processing, the SCI arrays are converted to floating point data from the input integer data. The ERR array also is populated with floating point values. The size of the calibrated images become quite large, as noted in Table 11.2. The final size of the distortion corrected images, though, can be considerably larger due to the expansion to account for the distortion, as noted in the last column of Table 11.2. The final image size as produced by OTFR can be up to 50% larger than the distortion corrected size since most ACS images get rotated to align the Y axis with North. Thus, the final size depends on the initial orientation of the image, but will never be smaller than the value listed in Table 11.2.
Table 11.2: Final sizes of unrotated calibrated ACS exposures for each detector.
FITS file
(Sraw)
1This size assumes no dither offset or scale change.
While the size of the final calibrated HRC or SBC exposures are comparable to those of STIS or WFPC2, the ACS WFC exposures are over 16 times as large. In addition, the following equation should be used to estimate the minimum amount of free storage that should be available during processing of associated ACS data:
- Dmin is the minimum free disk space required for processing,
- Scal is the size of the calibrated exposure (from Table 11.2),
- Sraw is the size of the raw exposure (from Table 11.2), and
- n is the number of exposures in each
CR-SPLITset orREPEAT-OBSset.- Sdrz is the size of the distortion corrected, dither combined (if needed) exposure (from Table 11.2).
- p is the percentage shift (in pixels) across all dither positions.
Size of Reference Files for Re-Processing
Another additional concern when processing ACS observations may be the amount of storage taken for reference files. For ACS WFC observations, a complete set of reference files could exceed 520MB by themselves. HRC and SBC reference files, in comparison, only require about 45MB of disk space.
Speed of Pipeline Processing
Some observers will eventually want to re-calibrate their ACS data locally. Reprocessing HRC or SBC data will not put a burden on most computing systems since the data sizes are relatively small, both for the science data and for the reference files. Processing ACS WFC observations, on the other hand, will require more computing power, both in terms of CPU runtime and disk space. Great care has been taken to minimize the memory requirements of the pipeline software to accommodate most computing configurations. Even so, CALACS requires up to 130MB of memory, while PyDrizzle requires up to 400MB to process WFC data. This involves relying on line-by-line I/O to read the input data and reference files, placing an extra burden on the I/O capabilities of the computing system.
Overall, though, an Ultra-10 class workstation (300 MHz, 512MB RAM) can still run CALACS to process ACS WFC data in reasonable times; namely, about 13 minutes for a CR-SPLIT=2 ACS/WFC association or about 4.5 minutes for a single ACS/WFC exposure. The PyDrizzle processing only requires working on the output image, not all individual input images, thus requiring only about 5 minutes to process a single ACS/WFC exposure itself. These results can be scaled by the processor speed for the computing system, resulting in even more acceptable runtimes for faster, newer workstations.
|
|||||
|
Space Telescope Science Institute http://www.stsci.edu Voice: (410) 338-1082 help@stsci.edu |
