Step

class jwst.stpipe.Step(name=None, parent=None, config_file=None, _validate_kwds=True, **kws)[source]

Bases: object

Create a Step instance.

Parameters:
  • name (str, optional) – The name of the Step instance. Used in logging messages and in cache filenames. If not provided, one will be generated based on the class name.
  • parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
  • config_file (str path, optional) – The path to the config file that this step was initialized with. Use to determine relative path names.
  • **kws (dict) – Additional parameters to set. These will be set as member variables on the new Step instance.

Attributes Summary

prefetch_references
reference_file_types
spec

Methods Summary

__call__(*args) Run handles the generic setup and teardown that happens with the running of each step.
call(*args, **kwargs) Creates and runs a new instance of the class.
closeout([to_close, to_del]) Close out step processing
from_cmdline(args) Create a step from a configuration file.
from_config_file(config_file[, parent, name]) Create a step from a configuration file.
from_config_section(config[, parent, name, …]) Create a step from a configuration file fragment.
get_reference_file(input_file, …) Get a reference file from CRDS.
load_spec_file([preserve_comments])
make_output_path(step, data[, basepath, …]) Make up a path based on data and user specification
merge_config(config, config_file)
print_configspec([stream])
process(*args) This is where real work happens.
reference_uri_to_cache_path(reference_uri) Convert an abstract CRDS reference URI to an absolute file path in the CRDS cache.
resolve_file_name(file_name) Resolve a file name expressed relative to this Step’s configuration file.
run(*args) Run handles the generic setup and teardown that happens with the running of each step.
save_model(model, suffix, *args, **kwargs) Saves the given model using the step/pipeline’s naming scheme
search_attr(attribute[, parent_first]) Return first non-None attribute in step heirarchy
set_input_filename(path) Sets the name of the master input file.

Attributes Documentation

prefetch_references = True
reference_file_types = []
spec = '\n pre_hooks = string_list(default=list())\n post_hooks = string_list(default=list())\n output_dir = string(default=None) # Directory path for output files\n output_file = output_file(default=None) # File to save output to.\n skip = boolean(default=False) # Skip this step\n save_results = boolean(default=False) # Force save results\n suffix = string(default=None) # Default suffix for output files\n '

Methods Documentation

__call__(*args)

Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

classmethod call(*args, **kwargs)[source]

Creates and runs a new instance of the class.

To set configuration parameters, pass a config_file path or keyword arguments. Keyword arguments override those in the specified config_file.

Any positional *args will be passed along to the step’s process method.

Note: this method creates a new instance of Step with the given config_file if supplied, plus any extra *args and **kwargs. If you create an instance of a Step, set parameters, and then use this call() method, it will ignore previously-set parameters, as it creates a new instance of the class with only the config_file, *args and **kwargs passed to the call() method.

If not used with a config_file or specific *args and **kwargs, it would be better to use the run method, which does not create a new instance but simply runs the existing instance of the Step class.

closeout(to_close=None, to_del=None)[source]

Close out step processing

Parameters:
  • to_close ([object(, ..)]) – List of objects with a close method to execute The objects will also be deleted
  • to_del ([object(, ..)]) – List of objects to simply delete

Notes

Other operations, such as forced garbage collection will also be done.

static from_cmdline(args)[source]

Create a step from a configuration file.

Parameters:args (list of str) – Commandline arguments
Returns:step – If the config file has a class parameter, the return value will be as instance of that class.

Any parameters found in the config file will be set as member variables on the returned Step instance.

Return type:Step instance
classmethod from_config_file(config_file, parent=None, name=None)[source]

Create a step from a configuration file.

Parameters:
  • config_file (path or readable file-like object) – The config file to load parameters from
  • parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
  • name (str, optional) – If provided, use that name for the returned instance. If not provided, the following are tried (in order): - The name parameter in the config file - The filename of the config file - The name of returned class
Returns:

step – If the config file has a class parameter, the return value will be as instance of that class. The class parameter in the config file must specify a subclass of cls. If the configuration file has no class parameter, then an instance of cls is returned.

Any parameters found in the config file will be set as member variables on the returned Step instance.

Return type:

Step instance

classmethod from_config_section(config, parent=None, name=None, config_file=None)[source]

Create a step from a configuration file fragment.

Parameters:
  • config (configobj.Section instance) – The config file fragment containing parameters for this step only.
  • parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
  • name (str, optional) –

    If provided, use that name for the returned instance. If not provided, try the following (in order):

    • The name parameter in the config file fragment
    • The name of returned class
  • config_file (str, optional) – The path to the config file that created this step, if any. This is used to resolve relative file name parameters in the config file.
Returns:

step – Any parameters found in the config file fragment will be set as member variables on the returned Step instance.

Return type:

instance of cls

get_reference_file(input_file, reference_file_type)[source]

Get a reference file from CRDS.

If the configuration file or commandline parameters override the reference file, it will be automatically used when calling this function.

Parameters:
  • input_file (jwst.datamodels.ModelBase instance) – A model of the input file. Metadata on this input file will be used by the CRDS “bestref” algorithm to obtain a reference file.
  • reference_file_type (string) – The type of reference file to retrieve. For example, to retrieve a flat field reference file, this would be ‘flat’.
Returns:

reference_file

Return type:

path of reference file, a string

classmethod load_spec_file(preserve_comments=False)[source]
static make_output_path(step, data, basepath=None, suffix=None, ext=None, result_id=None, ignore_use_model=False)[source]

Make up a path based on data and user specification

Parameters:
  • step (Step) – The step which produced the data
  • data (obj) – Unused by this routine
  • basepath (str or None) – The output file name. If None or empty string, create a filename based on the data.
  • suffix (str or None) – The suffix to append to the basename.
  • ext (str or None) – The file format extension
  • result_id (str or None) – If a suffix cannot be determined, use this as the suffix. If the result is still None, raise ValueError
  • ignore_use_model (bool) – Ignore configuration parameter output_use_model
Returns:

output_path – The fully qualified output path

Return type:

str

classmethod merge_config(config, config_file)[source]
classmethod print_configspec(stream=<open file '<stdout>', mode 'w'>)[source]
process(*args)[source]

This is where real work happens. Every Step subclass has to override this method. The default behaviour is to raise a NotImplementedError exception.

reference_uri_to_cache_path(reference_uri)[source]

Convert an abstract CRDS reference URI to an absolute file path in the CRDS cache. Reference URI’s are typically output to dataset headers to record the reference files used.

e.g. ‘crds://jwst_miri_flat_0177.fits’ –>
‘/grp/crds/cache/references/jwst/jwst_miri_flat_0177.fits’

The CRDS cache is typically located relative to env var CRDS_PATH with default value /grp/crds/cache. See also https://jwst-crds.stsci.edu

resolve_file_name(file_name)[source]

Resolve a file name expressed relative to this Step’s configuration file.

run(*args)[source]

Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

save_model(model, suffix, *args, **kwargs)[source]

Saves the given model using the step/pipeline’s naming scheme

Parameters:
  • model (jwst.datamodels.Model instance) – The model to save.
  • suffix (str) – The suffix to add to the filename.

Notes

This routine is used to save data outside of the normal step cycle, where results are saved at the end of a step.

Serious consideration should be given to the step design if this call is needed. In particular, if such data is important to save, consider making the producing code its own step, such that it is subject to the output controls of the step infrastructure.

search_attr(attribute, parent_first=False)[source]

Return first non-None attribute in step heirarchy

Parameters:
  • attribute (str) – The attribute to retrieve
  • parent_first (bool) – If True, allow parent definition to override step version
Returns:

value – Attribute value or None if not found

Return type:

obj

set_input_filename(path)[source]

Sets the name of the master input file. Used to generate output file names.