7.2 The HST Data Archive

All science and calibration data, along with a large fraction of the engineering data, are placed in the HST Data Archive. Science data become immediately available to a program's Principal Investigator and those designated by him/her. These data may be retrieved after the PI has registered as an archive user, and are normally proprietary for a period of one year (see Section 5.1 of the Call for Proposals for information on data rights).

Typically the science data from HST flow through the production pipeline and into the Data Archive within 2-3 days after execution on the telescope. The observer is notified by e-mail when the first datasets reach the archive, and is provided with web tools to track a visit's completeness as well as to retrieve the data generated by it. The time for retrieving data from the Archive is nominally a few hours, but can be more than a day depending upon the data load and system status. However, occasional software or system failures may lengthen the processing/retrieval times.

If you have strict scientific requirements for data receipt within days after execution, such as to provide pointing corrections for tightly scheduled visits, there are resource-intensive methods to expedite delivery of data. If you have such requirements, they must be stated in your proposal so that the resource needs can be determined and reviewed.

As of June 1, 2003, the HST Archive contained over 559,000 individual observations. These observations, along with engineering and other supporting information, comprise over 13.75 Terabytes of data. About 200 new observations (and 10-15 Gbytes of data) are archived every day. The heart of the Archive is the Data Archive and Distribution Service (DADS)-a collection of magneto-optical disks on which the data are stored, the databases that comprise the Archive catalog, and the hardware and software that support the ingest and distribution of HST data.

7.2.1 StarView and Web Access to the HST Data Archive

Most of the data in the HST Archive are public and may be retrieved by any user. The Archive can be accessed either through the MAST Web page or by using a special user interface called StarView. The Web interface can do simple searches by object name or location or by lists of names or locations, and it can retrieve data and calibration files. StarView is a JAVA-based tool that runs on most operating systems. It provides a wider range of options for searching the Archive, including the ability to create custom queries, to cross-correlate lists of objects observed by HST, and to use the Visual Target Tuner (VTT) for display of the apertures of multiple, individual HST observations on a single DSS image. The StarView Web page provides information and instructions for downloading the StarView program. Updates on the HST Archive are provided in both StarView and on the MAST web site.

The MAST web site and StarView allow you to preview most of the publicly available images and spectra. Both interfaces also offer integrated access to the Digitized Sky Survey (DSS) and allow the user to access the Set of Identifications, Measurements and Bibliography for Astronomical Data (SIMBAD) or NASA/IPAC Extragalactic Database (NED) to look up the coordinates of an object by name.

All data requested from WFPC2, ACS, STIS, and NICMOS will be reprocessed in the pipeline at the time of the request. This On-The-Fly Reprocessing (OTFR) takes the science data in spacecraft packet format and generates calibrated files. On-The-Fly Reprocessing prepares a new FITS file with each request, which allows a clean integration of new headers with updated keywords which are fully compatible with the latest version of the pipeline and calibration software. The Archive no longer saves the initial versions of the calibrated data, but does archive the initial version of the uncalibrated FITS data. The next version of DADS, which we expect to release for public use by 2004, will allow users to filter their retrievals to only the files they need (as well as allowing secure ftp retrievals for most users and retrieval of compressed data.)

STScI maintains an "Archive Hotseat" to which all Archive-related questions, problems, or comments should be referred. The Archive Hotseat can be reached by email at archive@stsci.edu or by phone at 410-338-4547.