The actual pipelines that call individual correction steps in various orders are defined as python classes within python code modules. The pipelines can be executed by referencing their class name or through the use of a configuration (.cfg) file that in turn references the class. The table below shows the pipeline classes that are currently available, the corresponding pre-defined configurations that make use of those classes, and the instrument modes to which they can be applied.
|Class Name||Configuration File||Used For|
|Detector1Pipeline||calwebb_detector1.cfg||Stage 1 processing: all modes|
|DarkPipeline||calwebb_dark.cfg||Stage 1 processing: darks|
|Image2Pipeline||calwebb_image2.cfg||Stage 2 processing: imaging modes|
|Spec2Pipeline||calwebb_spec2.cfg||Stage 2 processing: spectroscopy modes|
|Image3Pipeline||calwebb_image3.cfg||Stage 3 processing: imaging modes|
|Spec3Pipeline||calwebb_spec3.cfg||Stage 3 processing: spectroscopy modes|
|Ami3Pipeline||calwebb_ami3.cfg||Stage 3 processing: NIRISS AMI mode|
|Coron3Pipeline||calwebb_coron3.cfg||Stage 3 processing: Coronagraphic mode|
|TSO3Pipeline||calwebb_tso3.cfg||Stage 3 processing: Time Series mode|
Input Files, Output Files and Data Models¶
An important concept used throughout the JWST pipeline is the Data Model. Nearly all data used by any of the pipeline code is encapsulated in a data model. Most input is read into a data model and all output is produced by a data model. When possible, this document will indicate the data model associated with a file type, usually as a parenthetical link to the data model in question. For some steps, the output file may represent different data models depending on the input to those steps. As a result, the data models listed here will not be an exhaustive list.
Stage 1 Pipeline Step Flow (calwebb_detector1)¶
Stage 1 processing applies basic detector-level corrections to all exposure
types (imaging, spectroscopic, coronagraphic, etc.). It is applied to one
exposure at a time. The pipeline module for stage 1 processing is
calwebb_detector1 (the equivalent pipeline class is
Detector1Pipeline). It is
often referred to as
ramps-to-slopes processing, because the input raw data
are in the form of one or more ramps (integrations) containing accumulating
counts from the non-destructive detector readouts and the output is a corrected
countrate (slope) image. The list of steps applied by the Build 7.1 calwebb_detector1
pipeline is as follows.
- Raw 4D product: The input to
calwebb_detector1is a single raw exposure file, e.g.
jw80600012001_02101_00003_mirimage_uncal.fits, which contains the original raw data from all of the detector readouts in the exposure (ncols x nrows x ngroups x nintegrations).
- 2D Countrate product: All types of inputs result in a 2D countrate product,
resulting from averaging over all of the integrations within the exposure.
The output file will be of type
- 3D Countrate product: If the input exposure contains more than one integration
(NINTS>1), a 3D countrate product is created that contains the individual
results of each integration. The 2D countrate images for each integration are
stacked along the 3rd axis of the data cubes (ncols x nrows x nints). This
output file will be of type
calwebb_detector1 pipeline has one optional argument:
which is a boolean argument with a default value of
False. If the user sets
True, the pipeline will save intermediate data to a file as it
exists at the end of the
jump step (just before ramp fitting). The data at
this stage of the pipeline are still in the form of the original 4D ramps
(ncols x nrows x ngroups x nints) and have had all of the detector-level
correction steps applied to it, including the detection and flagging of
Cosmic-Ray hits within each ramp (integration). If created, the name of the
intermediate file will be constructed from the root name of the input file, with
the new product type suffix
Dark Pipeline Step Flow (calwebb_dark)¶
The stage 1 dark (
calwebb_dark) processing pipeline is intended for use
with dark exposures. It applies all of the same detector-level correction steps
calwebb_detector1 pipeline, but stops just before the application of the
- Raw 4D Dark product: The input to
calwebb_darkis a single raw dark exposure.
- 4D Corrected product: The output is a 4D (ncols x nrows x ngroups x nints)
product that has had all corrections up to, but not including, the
dark_currentstep, with a product file type of
calwebb_dark pipeline does not have any optional arguments.
Stage 2 Imaging Pipeline Step Flow (calwebb_image2)¶
Stage 2 imaging (
calwebb_image2) processing applies additonal corrections
that result in a fully calibrated individual exposure. The list of correction
steps applied by the calwebb_image2 imaging pipeline is as follows.
- 2D or 3D Countrate product: The input to the
calwebb_image2pipeline is a single countrate exposure, in the form of either a
_rateintsfile. If the latter (data on a per-integration basis), the steps in the pipeline are applied individually to each integration, where appropriate.
- 2D or 3D Calibrated product: The output is a single calibrated exposure, using
the product type suffix
_calints, depending on the type of input (e.g.
calwebb_image2 pipeline does not have any optional arguments.
Stage 2 Spectroscopic Pipeline Step Flow (calwebb_spec2)¶
Stage 2 spectroscopic (
calwebb_spec2) processing applies additional
corrections to countrate products that result in fully calibrated individual
The list of correction steps is shown below. Some steps are only applied to
certain instruments or instrument modes, as noted in the table.
resample_spec step produces a resampled/rectified product for non-IFU
modes of some kinds of spectroscopic exposures. If the
is not applied to a given exposure, the
extract_1d operation will be
performed on the original (unresampled) data. The
cube_build step produces
a resampled/rectified cube for IFU exposures.
The input to the
calwebb_spec2 pipeline can be either a single countrate
_rateints) exposure or an Association (ASN) file
listing multiple exposures. The background subtraction (
imprint subtraction (
imprint_subtract) steps can only be executed when
the pipeline is supplied with an association of exposures, because they rely
on multiple exposures to perform their tasks. The ASN file must not only list
the input exposures, but must also contain information that indicates their
relationships to one another.
The background subtraction step can be applied to an assocation containing nodded exposures, such as for MIRI LRS fixed-slit, NIRSpec fixed-slit, and NIRSpec MSA observations, or an association that contains dedicated exposures of a background source. The step will accomplish background subtraction by doing direct subtraction of nodded exposures from one another or by direct subtraction of dedicated background expsoures from the science target exposures.
The imprint subtraction step, which is only applied to NIRSpec MSA and IFU exposures, also requires the use of an ASN file, in order to specify which of the inputs is to be used as the imprint exposure. The imprint exposure will be subtracted from all other exposures in the association.
If a single countrate product is used as input, the background subtraction and imprint subtraction steps will be skipped and only the remaining regular calibration steps will be applied to the input exposure.
Two or three different types of outputs are created by
- Calibrated 2D product: All types of inputs result in a fully-calibrated 2D
product at the end of the
photomstep, which use the
_calintsproduct type suffix, depending on whether the input was a
- Resampled 2D product: If the input is an exposure type that gets
resampled/rectified by the
resample_specstep, the rectified 2D spectral product created by the
resample_specstep is saved as a
- Resampled 3D product: If the data are NIRSpec IFU or MIRI MRS, the
results of the
cube_buildstep will be saved as a
- 1D Extracted Spectrum product: All types of inputs result in a 1D extracted
spectral data product, which is saved as a
If the input to
calwebb_spec2 is an ASN file, these products are created
for each input exposure.
calwebb_spec2 pipeline has one optional argument:
which is a Boolean argument with a default value of
False. If the user sets
True, the results of the background subtraction step (if applied) are
saved to an intermediate file of type
_bsubints, as appropriate.
Stage 3 Imaging Pipeline Step Flow (calwebb_image3)¶
Stage 3 processing for imaging observations is intended for combining the data
from multiple exposures (e.g. a dither or mosaic pattern) into a single
rectified (distortion corrected) product.
Before being combined, the exposures receive additional corrections for the
purpose of astrometric alignment, background matching, and outlier rejection.
The steps applied by the
calwebb_image3 pipeline are shown below.
- Associated 2D Calibrated products: The inputs to
calwebb_image3will usually be in the form of an ASN file that lists multiple exposures to be processed and combined into a single output product. The individual exposures should be in the form of calibrated (
_cal) products from
- Single 2D Calibrated product: It is also possible use a single
_calfile as input to
calwebb_image3, in which case only the
source_catalogsteps will be applied.
- Resampled 2D Image product (DrizProductModel): A resampled/rectified 2D image product of type
_i2dis created containing the rectified single exposure or the rectified and combined association of exposures, which is the direct output of the
resamplestep. This is the
- Source catalog: A source catalog produced from the
_i2dproduct is saved as an ASCII file in
ecsvformat, with a product type of
- CR-flagged products: If the
outlier_detectionstep is applied, a new version of each input calibrated exposure product is created, which contains a DQ array that has been updated to flag pixels detected as outliers. This updated product is known as a CR-flagged product and the file is identified by appending the association candidate ID to the original input
_calfile name, e.g.
Stage 3 Aperture Masking Interferometry (AMI) Pipeline Step Flow (calwebb_ami3)¶
The stage 3 AMI pipeline (
calwebb_ami3) is intended to be applied to
associations of calibrated NIRISS AMI exposures and is used to compute fringe
parameters and correct science target fringe parameters using observations of
The steps applied by the
calwebb_ami3 pipeline are shown below.
- Associated 2D Calibrated products: The inputs to
calwebb_ami3are assumed to be in the form of an ASN file that lists multiple science target exposures, and optionally reference target exposures as well. The individual exposures should be in the form of calibrated (
_cal) products from
- LG product (AmiLgModel): For every input exposure, the fringe
parameters and closure phases caculated by the
ami_analyzestep are saved to an
_lgproduct type file.
- Averaged LG product (AmiLgModel): The LG results averaged over all science or reference
exposures, calculated by the
ami_averagestep, are saved to an
_lgavgt(for the science target) or
_lgavgr(for the reference target) file. Note that these output products are only created if the pipeline argument
save_averages(see below) is set to
- Normalized LG product (AmiLgModel): If reference target exposures are included in the input
ASN, the LG results for the science target will be normalized by the LG
results for the reference target, via the
ami_normalizestep, and will be saved to an
calwebb_ami3 pipeline has one optional argument:
which is a Boolean parameter set to a default value of
False. If the user
sets this agument to
True, the results of the
ami_average step will be
saved, as described above.