%%%%%%%%%%%%%%%
% Preamble
%%%%%%%%%%%%%%%

%\documentstyle[12pt,aaspp4,epsf]{article}
\documentstyle[11pt,aaspp4,epsf]{reportj}

\renewcommand{\familydefault}{\rmdefault}
\slugcomment{{\bf Technical Instrument Report CDBS 2008-02}}
\lhead{}
%\cfoot{\rm Copyryght 1999 The Association of Universities for Research in Astronomy. All Rights Reserved.}
%\setlength{\headrulewidth}{0pt}
%\setlength{\footrulewidth}{0pt}
\lefthead{Diaz}
\righthead{Assessment and Delivery of Calibration Reference Files}

%%%%%%%%%%%%%%%
% User defined macros
%%%%%%%%%%%%%%%

\def\ssection#1{\section{\hbox to \hsize{\large\bf #1\hfill}}}
\def\ssectionstar#1{\section*{\hbox to \hsize{\large\bf #1\hfill}}}
\def\ssubsection#1{\subsection{\hbox to \hsize{#1\hfill}}}
\def\ssubsectionstar#1{\subsection*{\hbox to \hsize{#1\hfill}}}
\def\ssubsubsection#1{\subsubsection{\hbox to \hsize{#1\hfill}}}
\def\ssubsubsectionstar#1{\subsubsection*{\hbox to \hsize{#1\hfill}}}

\newcommand{\etal}{{et al.~}}

\newcommand{\lta}{\lesssim}
\newcommand{\gta}{\gtrsim}

\renewcommand{\footnoterule}{\null}

%%%%%%%%%%%%%%%
% Useful definitions during manuscript preparation
%%%%%%%%%%%%%%%

% \def\epsfbox#1{}

\def\xx{$^{[xx]}$}
\def\tbd#1{{\baselineskip=9pt\medskip\hrule{\small\tt #1}\smallskip\hrule\medskip}}
\def\com#1{{\baselineskip=9pt\medskip\hrule{\small\sl #1}\smallskip\hrule\medskip}}


\begin{document}

\null
\vskip-2.0truecm
\epsfxsize=5.2truecm
\epsfysize=3.5truecm
\hbox to \hsize{\epsfbox{STLogo.eps}\hfill}

\vskip-0.7truecm

\title{\LARGE \bf
\hbox to \hsize{\hfill Delivery of Reference Files to the  }
\hbox to \hsize{\hfill  Data Management Systems}}
\vskip-1.0truecm
\hbox to \hsize{\hfill \hbox to 5truecm{\bf \hrulefill}} 
%\smallskip
\author{{\hfill  \bf R. I. Diaz, M. Cracraft}}
\medskip
%\author{\hbox to \hsize{\hfill \Large R. I. Diaz-Miller}}
\affil{ \hbox to \hsize{\hfill \bf rmiller@stsci.edu,cracraft@stsci.edu}}
\vskip-0.3truecm
\affil{ \hbox to \hsize{\hfill \bf June 24, 2008}}



%%%%%%%%%%%%%%%
% Use a small baselineskip
%%%%%%%%%%%%%%%

\baselineskip=14pt
%\baselineskip=15pt


%%%%%%%%%%%%%%%
% Abstract
%%%%%%%%%%%%%%%

\smallskip
{\bf \hrule}
\noindent {\Large\bf Abstract}\\
\noindent 
\emph{This TIR describes the  INS/CDBS Team's responsibilities. It defines the standard procedures for the delivery of calibration pipeline 
and SYNPHOT reference files to the Data Management Systems (DMS). It  provides guidelines for test and validation of reference files by the INS/CDBS Team. This is an update to TIR CDBS 2005-02A. It clarifies the procedures to deliver all types of SYNPHOT files, including Atlases and bandpasses.}
\medskip
\hrule

%%%%%%%%%%%%%%%
% Beginning of main text
%%%%%%%%%%%%%%%

% SEC 1
\ssectionstar{Introduction} 
\label{s:intro}

% **************************************************************
% **************************************************************
% **************************************************************
% *****
% *****
% *****
% *****  SEC  1. INTRODUCTION
% *****
% *****
% *****
% **************************************************************
% **************************************************************
% **************************************************************
% Put the ISR number here as well... and make sure this falls in the second page
\lhead{\rm Instrument Science Report CDBS 2005-02}
\cfoot{\rm\thepage}
\rhead{}

In order for reference 
files to be used by the OTFR pipeline,  these files have to be  copied to OPUS disks. Pointers to these files are based on instrument modes and applicability dates.
On the other hand, their ingestion in the Data Archive and Distribution Services (DADS), disk media and database, allows users to retrieve them from the archive. 
The main function of the CDBS database is to allow selection of the correct reference files based on an instrument's configuration and date. The selection is based on data file keyword values and criteria outlined in ICD-47
 (http://www.stsci.edu/instruments/observatory/cdbs/documents/icd-47RevF.pdf or 
 http://www.ess.stsci.edu/projects/distribution/ICD47/ICD-47\_RevF.pdf.). More information can be found in the CDBS Documents web page (http://www.stsci.edu/hst/observatory/cdbs/documents).

The person creating the file will be checking and assessing the quality of the reference files. The official delivery of the files to CDBS is handled by the INS/CDBS Team. Here we describe detailed steps and procedures for ingesting reference files into the databases. 

% Put the ISR number here as well... and make sure this falls in the second page
\lhead{\rm Instrument Science Report CDBS 2005-02}
\cfoot{\rm\thepage}
\rhead{}

% SEC 2
\ssectionstar{Summary: Test and Validation of Reference Files}

\begin{enumerate}
\item Transfer the reference files to a directory in the CDBS delivery area
\item Check the permissions of the files
\item Make sure that all relevant header keywords and history section are present in the header [0] of the FITS file
\item Verify that the files are in standard FITS format

\vskip 0.4truecm
\hskip -1truecm
For calibration spectra or spectral models, skip all the steps below and go to appendix  B
\vskip 0.4truecm

\item Run the CDBS \verb|certify| tool on the FITS files or GEIS header files
\item Create the ``load'' files
\item Populate the ``load'' files
\item Certify the ``load'' files
\item Run \verb|check_load|
\item Rename the pipeline reference files to have an unique name identifier
\begin{itemize}
\item In the case of SYNPHOT files, make sure that the throughput or bandpass files have a number that is greater than the last delivered version of the file
\end{itemize}
\item In the case of SYNPHOT bandpasses, make sure all the instrument teams have tested these files. Obtain sign-off from all of them before delivery.
\item Put the files in a delivery directory and deliver them to the DMS
\item Fill out the template delivery form and e-mail it to the DMS (if OPUS\_FLAG = Y)
\item Transfer the files to the centralized storage area
\item In the case of SYNPHOT files, create the TMC table and deliver it together with the TMG and TMT files (if applicable)
\begin{itemize}
\item Transfer the TMC, TMT, and TMG files to the centralized storage area for further testing
\item Send a message to Vicki Laidler and wait for acknowledgment that they passed testing
\item Deliver the file to DMS
\item e-mail the delivery form to DMS
\end {itemize}
\item Run \verb|cdbs_report|
\item Check the size of the files in the archive
\item Check that the files are correctly used in the archive
\item Send notification to deliverer (when applicable)
\item Fill out delivery information on CDBS WIKI
\end{enumerate}

\ssectionstar{Detailed description of the preparation steps}

The following steps assume that you are using the special account for CDBS deliveries and that you are working in the \verb|smalls.stsci.edu| domain. In this domain, the account settings have the appropriate permissions to access the databases and the different disk locations used for the delivery process.
We suggest following these steps in the order they appear, and whenever a problem is found in one of them, try to solve it before proceeding with the next step.
A log file should be kept documenting the tests done on the files. As a team convention, save the log with the output of the CDBS and other commands in a file named ``delivery.log''. This file will be used by the team as a status report for deliveries. In the following sections, all the examples provided will be given assuming that the outputs are re-directed to this log file. Given that some of the steps described here will use IRAF tasks, open an IRAF session too.

\noindent
\ssubsectionstar{1. Transfer the reference files to a directory in the CDBS delivery area}

Create a delivery directory. Currently the 
deliveries are done from the \verb|smalls.stsci.edu| domain. The area assigned to test and validate the reference files prior to delivery is located in the directory /calib/cdbs\_delivery/. Here, each instrument 
team has a particular area assigned. For example, ACS deliveries
are in the /calib/cdbs\_delivery/ACS/ directory while STIS deliveries are in /calib/cdbs\_delivery/STIS/. The delivery directories are named after the date when the files were delivered to the CDBS Team, with format {\it yyyy\_mm\_dd}, where {\it yyyy} is the year, {\it mm} the 
month, and {\it dd} the day.
These files will remain here for safekeeping until the files are ingested into the databases. Complete steps
2 to 18 in this directory. To save disk space, gzip \verb|lod| files and erase the \verb|FITS| files after the files have been  ingested. 
A log file should be kept with the output of all the scripts and tasks used to test the 
files. For this, redirect the outputs to a log file using the append redirection command:
``$>>\&$''. Note the ``\&'' character, this is used to record all the output flags, including those that are
sent to the
standard error instead of standard output.

Transfer the FITS files from the deliverer directory to this directory using FTP or copy (if they are already in the \verb|smalls.stsci.edu| domain). When using FTP remember
that you are transferring binary files and that the transferring mode has to be binary.

In the case of WFPC2 files (except for the IDC reference file), the delivery will have instead of FITS files, four GEIS files (two files ``*d'' and two files ``*h'') and one ``*.lod'' file per delivered reference file. Transfer all to the smalls working directory. Note also that these files do not have the usual extensions (e.g. ``drk''). In this case, the format of the files is {\it {\bf rootname}}\verb|.r#x|; where \verb|#| can be a digit between 0 and 6, and \verb|x| will be the letter ``d'' or ``h''.  In the case of WFPC2 "drk" files, the root name of the files should be unique, i.e. the WFPC2 Team has already renamed them using the \verb|uniqname| script. This is because the "drk" files are generated automatically. Other type of files should be renamed with the \verb|uniqname| script by us.

The WFPC2 team delivers two types of GEIS files, one that can be used withe the Solaris and MacOS systems, and another that can be used by Linux systems.  Only those used in Solaris systems should be tested and delivered.  The Linux files are only transferred to the appropriate directory in the centralized storage area; see step 13.

If the delivery is of SYNPHOT throughput files and you also receive a ``Master Graph Table'' (TMG; for the file extension name) and the ``HST Thermal Components Master Table'' (TMT; for the file extension name), make sure that you put the throughput and the TMG and TMT files in different directories, for example, a subdirectory labeled {\bf tmtables}. This is because the TMG and TMT files have to be delivered together with the ``Master Component Table''  (TMC; for the file extension name) after all the throughput files are in the system. Perform the following steps for the throughput tables only. The TMG and TMT files will be tested later, together with the TMC file. 

\noindent
\ssubsectionstar{2. Check the permissions of the files}
Using the command \verb|ls -la|, make sure that all the files in the delivery directory have``user'', ``group'', and ``other'' read permissions. For example, in the following list:

\vspace{0.3cm}
\noindent
\texttt{-rw-r--r--   1 srefpipe    31680 May 11 17:31 p5b1731aj\_idc.fits\\
-rw-r--r--   1 srefpipe    22068 May 11 17:31 p5b1731aj\_idc.lod
}
\vspace{0.3cm}

\noindent
the string \verb|-rw-r--r--| indicates that the files can be read by anybody. This is necessary for the files to be correctly transfered to the OPUS and test areas.

\noindent
\ssubsectionstar{3. Make sure that relevant header keywords and history section are present. }

As mentioned in TIR CDBS 2005-01A, there are four header keywords  that should be present
and correctly populated in all reference files: \verb|PEDIGREE|, \verb|USEAFTER|, \verb|DESCRIP| and \verb|COMMENT|. For SYNPHOT data files, the header keywords: \verb|INSTRUME|, \verb|COMPNAME|, and \verb|DBTABLE| should also be checked. To do this check, the IRAF tasks \verb|hedit| or \verb|hselect|, or command \verb|more| can be used. Examples of \verb|hedit| and \verb|hselect| tasks are:

\vspace{0.3cm}
\noindent
\texttt{\verb|hedit| *.fits[0] pedigree,useafter,descrip,comment .\\
or\\
\verb|hselect| *fits[0] \$i,pedigree,useafter,descrip,comment  yes}
\vspace{0.3cm}

\noindent
An example using \verb|more|:

\vspace{0.3cm}
\noindent
\texttt{\verb|more| nameoffile.fits}
\vspace{0.3cm}


In the \verb|more| case, only one file at a time can be checked. To escape \verb|more| mode, type ``q''. In the case of WFPC2 GEIS reference files, only the GEIS header files have to be checked. The GEIS header files are ASCII files and have extensions ending in ``h''. In this step, the two header files per dataset  have to be checked to make sure they have the same and complete information.

 If  any of the relevant keywords are missing from the header of the files, contact the deliverer
and request the needed information. If the field \verb|COMMENT| is missing, it can be filled with the name of the deliverer as the creator of the file; e.g.,

\noindent
\vspace{0.3cm}
``Reference file created by J. Smith.''

\noindent
This can be done using the IRAF command \verb|hedit| and selecting the add option:

\vspace{0.3cm}
\noindent
\texttt{\verb|hedit| filename\_xxx.fits[0] \verb|COMMENT| ``Reference file created by J. Smith.'' add+}
\vspace{0.3cm}


Note, however, that some FITS reference files have a default \verb|COMMENT| section that refers to the FITS file format and which cannot be modified or erased. The FITS default  \verb|COMMENT| section  is different than the \verb|COMMENT| section (header keyword) referred to here. The way to distinguish between these two is by their format. In the case of the CDBS required \verb|COMMENT| line, the word \verb|COMMENT| is followed by an ``\verb|=|'', as in the example above and should list the people who created the file. For the cases when the FITS COMMENT line exists, the CDBS COMMENT  can not be added with the IRAF task \verb|hedit|, but in the following two ways.  

Using IRAF, you can first delete the default FITS COMMENT lines that appear in the file and then add the new one.  The commands needed to do this are:

\noindent
\texttt{cl$>$ thedit file.fits[0] comment delete+\\
cl$>$ hedit file.fits[0] comment "= 'comment string'" add+}

\noindent
Note that the "=" should be added at the beginning, or a comment section line would be added rather than the header keyword you were trying to create.   If this command does not work, try removing the = in the "= 'comment string'" and just add the comment you want, but double-check the header file to be sure the comment shows up with an equal sign.  The other way to add the header keyword is by using Pyraf as follows.

\noindent
\texttt{import pyfits\\
hdulist=pyfits.open(myfile.fits,mode=update)\\
hdulist[0].header.add\_comment(= comment string,before=origin)\\
hdulist.flush()}

\noindent
This last one will add a header keyword \verb|COMMENT| even if a comment section already existed.  Another task that will allow you to manually edit the headers of the files if necessary is the IRAF task \texttt{eheader}.  This task opens the header in a "vi" environment for editing.

In the case of the history section, check that it has a section relevant
to the current delivery.  This can be checked by comparing with the information provided in the delivery form. If this information was not provided
request  it from the deliverer. Check TIR CDBS 2005-01 for the relevant information needed in  the ``history'' section and how to update it within IRAF.

When checking the \verb|HISTORY| lines, keep in mind that there is a known bug in the CDBS software which could make the delivery fail. The information of the 
tracking file, known as a ``load'' file and accompanying each delivered file, is extracted from some of the  header keywords 
and history lines of the FITS file. Before they go in the ``load'' file, those are stripped of the word \verb|HISTORY|, extra spaces, and blank lines. Therefore, if a history line has the word ``go'' at 
the beginning of a line, it would be mistaken for the ``go'' SQL command and the ingest 
will fail with no clear error. Check all the lines of the history to make sure that none
 starts with this word. 

For WFPC2 GEIS reference files,  the ``*h'' files contain the header information. These are ASCII files and can be checked all at once using your favorite text editor or with grep. For example,

\noindent
\vspace{0.3cm}
\noindent
\texttt{grep -n USEAFTER *h\\
or\\
 grep -n COMMENT *h}
\vspace{0.3cm}

\noindent
Check all the header keywords and HISTORY section this way.

\noindent
\ssubsectionstar{4. Verify that the files are in standard FITS format }

Although the person creating the reference files has already verified that these are in standard FITS format,  double check them, as files that are not in standard FITS format cannot be ingested into the databases. (Note that GEIS  files should not be tested with this command.)
For this, run the \verb|fitsverify| script on the files: 

\vspace{0.3cm}
\noindent
\texttt{\verb|fitsverify| filename >>\& delivery.log}
\vspace{0.3cm}

\noindent
In the STScI Science Cluster, the \verb|fitsverify| version is different than that in \verb|smalls.stsci.edu| domain and some of the files could fail this test even though they pass the \verb|fitsverify| test in \verb|smalls.stsci.edu|. Therefore, in the Science Cluster, the FITS format verification has to be done with  \verb|farris_fitsverify|, which is the same
one as the \verb|fitsverify| version in \verb|smalls.stsci.edu|.

\vspace{0.3cm}
\noindent
\texttt{\verb|farris\_fitsverify| filename >>\& delivery.log}
\vspace{0.3cm}

\noindent
Wildcards may be used instead of file names, e.g., filename can be *.fits. 
A sample output from this script looks like this: 

\noindent
\vspace{0.3cm}
\texttt{{=================================================================== \\
FITS Verification for file: lbq1211ao\_bia.fits \\
=================================================================== \\
Summary contents of FITS file: lbq1211ao\_bia.fits \\
0: Primary Array ( SHORT ) \\
0 bytes, 108 header lines, 3 FITS blocks \\
1: Image Extension ( FLOAT ) [IMAGE,SCI,1] 2 dims [1024,1024] \\
4194304 bytes, 36 header lines, 1458 FITS blocks \\
2: Image Extension ( FLOAT ) [IMAGE,ERR,1] 2 dims [1024,1024] \\
4194304 bytes, 36 header lines, 1458 FITS blocks \\
3: Image Extension ( SHORT ) [IMAGE,DQ,1] 2 dims [1024,1024] \\
2097152 bytes, 36 header lines, 730 FITS blocks \\
No special records. \\
=================================================================== \\
No problems were encountered.}
}
\vspace{0.3cm}

\noindent
Examples of problems encountered with the files in this verification include: 
\begin{itemize}
\item extra spaces in keyword fields 
\item incorrect format for \verb|DATE| keyword field (18/12/00 instead of Dec 18, 2000) 
\item missing \verb|PCOUNT| and \verb|GCOUNT| keywords in extension headers. 
\end{itemize}

If any problems are found at this stage, send a message to the deliverer (Cc: cdbs@stsci.edu) with an 
explanation of the problem. Notify the deliverer that with this message you are canceling the delivery and that you need to receive a new delivery form when the file(s) has been fixed. If you are able to identify the problem include this information in your e-mail.  Remember that it is the responsibility of the deliverer to make sure that
the delivered files are FITS format compliant.

\noindent
\ssubsectionstar{5. Run the CDBS \verb|certify| tool on the files.}

The CDBS \verb|certify| tool performs further checking on the syntax and keyword values in 
the reference file, ensuring adherence to ICD-47 specifications for each type of reference file. Instrument  specific header keywords and columns (in a table file) that are necessary for the
correct selection of a reference file will be checked. In the particular case of WFPC2 GEIS files, only the GEIS header files (extension *.*h) should be run against \verb|certify|. For SYNPHOT Atlas files or HST Calibration Spectra files (CALSPEC),  the \verb|certify| tool is not run, as these are not recognized by the CDBS tools. For all the other cases, any errors in this file should be resolved before proceeding with the next step.
 Note that most CDBS scripts can also be accessed through IRAF in the stlocal.cdbsutil package; but those are likely an older version than the command line versions, so do not use them.  When using the tools in this document on a Solaris machine, be sure that it is a Solaris 10 machine (\verb|smalls| is a Solaris 10), as the tests are likely to fail on an earlier version of Solaris.
The \verb|certify| tool is run by typing in the command line: 

\vspace{0.3cm}
\noindent
\texttt{certify filename.fits >>\& delivery.log \\
or\\
certify filename.*h >>\& delivery.log}
\vspace{0.3cm}

\noindent
Wildcards may be used for filenames; e.g.,  *.fits or *.*h for WFPC2 header files. 
More detailed documentation on the \verb|certify| task is available, in postscript format, in the 
CDBS web page (http://www.stsci.edu/hst/observatory/cdbs/documents/).
The \verb|certify| tool does not check all the keyword syntax and values in the reference file, but 
only those that are specifically used in CDBS, OPUS, and DADS for selecting and tracking 
the reference files. A complete list of the instrument-dependent standard header keywords can 
be found in ICD-47. 

These required keywords are accessed by \verb|certify| via CDBS template files; template files end with ``.tpn''. There is a pair of files for each reference file type. One is for the FITS or GEIS files and one is for the ``load'' files (*\_ld.tpn). These files are located in the CDBS working  areas of the Science Cluster and the \verb|smalls.stsci.edu| domain. In the
smalls domain, these files are currently in the /store/smalls/cdbs/tools/data/  directory, while in the centralized storage the files are located in the /grp/hst/cdbs/tools/data/ directory. Note that whenever a template file is updated in the smalls domain, it should also be updated in the Science Cluster, otherwise the person delivering the file and working in the Science Cluster will not be using the most up to date version. A more detailed explanation on the procedures to change these files will be given in another CDBS TIR. A sample of the template file for the STIS PHT  reference file looks like this: 

\vspace{0.3cm}
\noindent
\texttt{{\# Template file used by \verb|certify| to check reference files \\
\# Some fields may be abbreviated to their first character: \\
\#\\
\# keytype = (Header|Group|Column) \\
\# datatype = (Integer|Real|Logical|Double|Character) \\
\# presence = (Optional|Required)\\
\#\\
\# NAME KEYTYPE DATATYPE PRESENCE VALUES \\
\#---------------------------------------------------------- \\
INSTRUME H C R STIS \\
FILETYPE H C R "PHOTOMETRIC CONVERSION TABLE" \\
DETECTOR H C R CCD,NUV-MAMA,FUV-MAMA \\
OBSTYPE H C R IMAGING,SPECTROSCOPIC \\
OPT\_ELEM C C R G140L,G140M,E140M,E140H,G230L,$\backslash$ \\
G230M,E230M,E230H,PRISM,G230LB,G230MB,G430L,G430M,G750L,G750M,$\backslash$  \\
MIRCUV,MIRFUV,MIRNUV,MIRVIS,X140H,X140M,X230H,X230M,N/A \\
CENWAVE H I R \\
1173,1200,1218,1222,1234,1271,1272,$\backslash$  \\
1307,1321,1343,1371,1380,1387,1400,1416,1420,1425,1453,1470,1489,$\backslash$  \\
1518,1526,1540,1550,1562,1567,1575,1598,1616,1640,1665,1687,1713,1714,$\backslash$  \\
1763,1769,1813,1851,1854,1863,1884,1913,1933,1963,1978,1995,2013,$\backslash$  \\
2014,2063,2095,2113,2124,2125,2135,2163,2176,2213,2257,2263,2269,$\backslash$  \\
2276,2313,2338,2363,2375,2376,2413,2415,2416,2419,2463,2499,2513,$\backslash$  \\
2557,2561,2563,2579,2600,2613,2659,2663,2697,2707,2713,2739,2762,$\backslash$  \\
2794,2800,2812,2818,2828,2836,2862,2898,2912,2962,2976,2977,3012,$\backslash$  \\
3055,3115,3165,3305,3423,3680,3843,3936,4194,4300,4451,4706,4781,$\backslash$  \\
4961,5093,5216,5471,5734,6094,6252,6581,6768,7283,7751,7795,8311,$\backslash$  \\
8561,8825,8975,9286,9336,9806,9851,10363,$\backslash$  \\
1232,1269,1305,1341,1378,1414,1451,1487,1523,1560,1587,1760,$\backslash$  \\
2010,2261,2511,2760,3010,1975,2703,-1,-999 \\
USEAFTER H C R \&SYBDATE  \\
PEDIGREE C C R \&PEDIGREE \\
DESCRIP C C R }
}

\noindent
A sample of the \verb|certify| output for a file that has a problem is: 

\vspace{0.3cm}
\noindent
\texttt{{== Checking mama2\_PFL.fits ==\\ 
Could not match keywords in header (mama2\_PFL.fits)\\ 
Cannot determine reference file type (mama2\_PFL.fits)}
}
\vspace{0.3cm}

\noindent
If you encounter a problem at this stage; first check to see if there are any obvious problems 
with the file header keywords or keyword 
values. A complete list of required and valid values for the header keywords can be found in the template files
or in ICD-47. If you identify the cause of the error, contact the person delivering the file to describe the problem and solicit input to fix the file. You could also reject the delivery and request the deliverer to send a new delivery form once the reference file has been fixed. In this case, you will have to re-start the process from step 1.

\ssubsectionstar{6. Create the ``load'' file}

In order to correctly ingest the files in CDBS, an ASCII ``load''  (*.lod) file is created for each 
reference file. This ``load'' file contains information from the reference file header, and information 
from the database about existing reference files. Exceptions to this are deliveries for WFPC2 data 
composed of GEIS files, SYNPHOT Atlas files, and HST Standard Calibration spectra. The process for these files will be explained at the end of this section.

 The information in the ``load'' file is used in the
delivery process to create SQL command scripts that populate the databases with the 
necessary information for the correct selection of the files. The ``load'' file will have 
the same root name as the FITS reference file, but with the extension ``lod''. The file consists of 
two sections: the header section and the row section. For image reference files, there 
is one header section followed by one row section. For table reference files there is one header 
section followed by one or more row sections, each corresponding to a row, or group of rows, in 
the reference table. The number of rows for table files is usually determined by the selection
criteria for the given reference file; therefore, regardless of the number of rows in the table,
some table reference files will have several 
row sections in the ``load'' file while others will have only one.
To create the ``load'' file type the following command: 

\noindent
\vspace{0.3cm}
\texttt{\verb|mkload| filename >>\& delivery.log}
 
\noindent
Wildcards may be used for filenames, e.g., filename can be *.fits. In the case of WFPC2 GEIS files, filename is the name of the GEIS header file with extension ``.r*h'', and will be discussed later. An example of a ``load'' file for a reference file image: 

\noindent
\vspace{0.3cm}
\noindent
\texttt{FILE\_NAME = 11x1\_2001\_1120\_1125\_ref\_bia.fits\\ 
INSTRUMENT = stis \\
REFERENCE\_FILE\_TYPE = bia \\
USEAFTER\_DATE = Nov 20 2001 00:00:00 \\
COMPARISON\_FILE = lbq12111o\_bia.fits \\
OPUS\_FLAG = \\
COMMENT = \\
ENDHEADER \\
\\
CHANGE\_LEVEL = \\
PEDIGREE = INFLIGHT \\
OBSERVATION\_BEGIN\_DATE = Nov 20 2001 \\
OBSERVATION\_END\_DATE = Nov 25 2001 \\
BINAXIS1 = 1 \\
BINAXIS2 = 1 \\
CCDAMP = D\\
CCDGAIN = 1 \\
CCDOFFST = 3 \\
DETECTOR = CCD \\
COMMENT = \\
ENDROW \\
ENDFILE}
\vspace{0.3cm} 

The \verb|mkload| command will use information contained in the FITS file to fill some of the fields of the ``load'' file.
There are a few more things about this file and command that are worth mentioning. The \verb|mkload| command automatically fills the \verb|USEAFTER_DATE| field with
the \verb|USEAFTER| header keyword information in the FITS file, while the \verb|COMPARISON_FILE| parameter is obtained from
the CDBS database. In the latter case, the information from the  header and row level information is used to determine the correct comparison reference file. If no file of the same type is found (e.g. when a new type or new mode is being delivered) this parameter will be filled with the value \verb|(INITIAL)|. This prevents other CDBS commands from trying to compare the fields of the new reference file with those of an old file.
Note also that in the case when the reference file has a \verb|PEDIGREE| value of \verb|INFLIGHT|, the \verb|mkload| task will populate the \verb|OBSERVATION_BEGIN_DATE| and \verb|OBSERVATION_END_DATE| with the dates given in the FITS file header keyword.
More detailed documentation 
on the \verb|mkload| task is available in postscript format in the CDBS web page (http://www.stsci.edu/instruments/observatory/cdbs/documents/).

In the case of dark and bias WFPC2 files, we receive four GEIS files for each
reference file and the ``load'' file, so we do not have to create it. For other cases, the ``load'' file should be created using the GEIS header file with extension ending in ``r?h'' (where ? can be a digit between 0 and 6).
In the case of WFPC2 ``IDC'' reference tables, standard FITS files are delivered and those can be treated as any other FITS reference file mentioned at the beginning of this section.

In the case of  SYNPHOT Atlas files (e.g. Kurucz) and HST Calibration Standards spectra, the files are not recognized by the CDBS tools, so the ``lod'' file cannot be created. These files are not  delivered to the CDBS, OPUS, or DADS databases; however, these need to be copied to the corresponding directory in the centralized storage (refer to section 13).

\ssubsectionstar{7. Populate the ``load'' files and check them}

The ``load'' file has several important fields that should be populated. As we mentioned in step 6, some of the fields are automatically populated by \verb|mkload|  using the information from the primary and extension headers of the FITS file.  Here we will describe those fields for which the
content is common to all instruments. These fields are: \verb|OPUS_FLAG| and \verb|COMMENT| in the header section, 
and \verb|CHANGE_LEVEL|, \verb|PEDIGREE|, \verb|OBSERVATION_BEGIN_DATE|, and 
\verb|OBSERVATION_END_DATE| fields in the row section. Other  fields in the ``load'' file vary from file to file and therefore will not be mentioned. Note also that an IRAF task, \verb|setlodkeywd|, has been developed 
to help populate the ``load'' files and will be explained in detail later in this section.

\ssubsubsectionstar{OPUS\_FLAG }

Set this to \verb|Y| or \verb|N| to indicate whether the files should be stored in the archive and by OPUS or not. We 
do expect to deliver reference files that for special reasons, should not be stored in the archive or by OPUS; for example, in the first stages of development, the WFC3 and COS teams requested that the SYNPHOT WFC3 and COS files not be stored in the archive. The \verb|OPUS_FLAG| should be set to \verb|N| for such cases, and 
the files will not be delivered to the OPUS and DADS databases. This information should be given
by the deliverer via the delivery form. {\bf Note} that the  \verb|OPUS_FLAG| for the TMG, TMT, and TMC should always be set to \verb|Y|, so the latest version is always stored in the archive.

\ssubsubsectionstar{\verb|COMMENT|  (in the header section)}

The \verb|COMMENT| section in the ``load'' file is the equivalent to the \verb|HISTORY| section in the 
data header. The information included here will appear in the StarView forms and on the reference file webpages; therefore, it is recommended to fill this section with information relevant to the delivered file only. This information should be provided by the deliverer or be contained in the \verb|DESCRIPTION| and  \verb|HISTORY| section in the FITS reference file.

\ssubsubsectionstar{\verb|CHANGE\_LEVEL|} 

This keyword defines the level of change of the reference file with respect to the last 
delivered file (given in the field \verb|COMPARISON_FILE|). Note that for table reference files, the changes could affect only a few rows in the file. In
this case, only the modified rows should have a value other than \verb|TRIVIAL| (the rows that
were not modified should always be \verb|TRIVIAL|); however, in some cases even the change level of the modified rows could also be \verb|TRIVIAL|. In the case of image reference files,
there is only one row section. The \verb|CHANGE_LEVEL| should be set to one of three values: \verb|SEVERE|, \verb|MODERATE|, or \verb|TRIVIAL|. 
The criteria for each are: 

\indent 
{\it SEVERE}

\indent
{\it i} Initial delivery of any file 
\vspace{-0.2cm}

\indent 
{\it ii} Change that requires existing data to be recalibrated 
\vspace{-0.2cm}

\indent
{\it iii} The row-level field for a table has changed by more than 50\% compared to the 
\verb|COMPARISON_FILE| 

\indent {\it MODERATE} 

\indent 
{\it i}  Changes are significant, but do not warrant data recalibration 
\vspace{-0.2cm}

\indent
{\it ii}  The row-level field for a table has changed by ~10-50\% compared to the 
\verb|COMPARISON_FILE| 

{\it TRIVIAL}
 
{\it i}  Changes are insignificant (e.g., fixing typos; removing erroneous but unused rows 
from a table), and do not warrant data recalibration.
\vspace{-0.2cm}

\indent
{\it ii}  No changes made to the row or image

\ssubsubsectionstar{\verb|PEDIGREE|}

This should be \verb|GROUND|, \verb|DUMMY| or \verb|INFLIGHT| (\verb|MODEL| is accepted in some cases) and is populated with the value given in the header keyword of the FITS file. Note that this field should have been filled already by the CDBS script \verb|mkload| for pipeline reference files and SYNPHOT throughput tables.  In the case of the TMC file, however,  this field is not filled by the \verb|mkload| script.  The value will have to be entered manually so that the PEDIGREE value for the TMC table matches the value in the SYNPHOT throughput tables that triggered the remake of the the TMC table. 

\ssubsubsectionstar{\verb|OBSERVATION\_BEGIN\_DATE| and \verb|OBSERVATION\_END\_DATE|}

These are the actual start and end date of acquisition of the calibration data used to 
create the reference file. The format should be Month Day Year (e.g., April 12, 2001) to be 
consistent with the \verb|USEAFTER| date format. These fields are populated by the \verb|mkload| script and
are left blank when the \verb|PEDIGREE| values are \verb|DUMMY|,  \verb|GROUND|, or \verb|MODEL|.

\ssubsubsectionstar{\verb|COMMENT| (in the row section)}

The \verb|COMMENT| field in the row 
section can be blank if there are no relevant comments at the row level, but use of comments 
at the row level is strongly encouraged. We will be delivering reference file tables 
where only a few rows of the table have changed significantly as compared to the old reference file. In 
such cases, row-level comments may be more appropriate than header-level comments, and 
they are required under such circumstances.

An IRAF task called \verb|setlodkeywd| has been developed for 
use in populating keyword fields in the ``load'' files automatically. This is particularly useful 
if a large number of files need to have fields populated in an identical manner. This task has
been defined within the delivery account IRAF tasks.  An lpar of the task looks like this: 

\vspace{0.3cm}
\noindent
\texttt{infile = "@filelist " File or list of lod files to fix \\
comments = yes Add comments? (yes/no/append) \\
change\_level = " " Change level value: SEVERE, MODERATE, TRIVIAL \\
opus\_flag = " " Opus flag (Y/N) \\
pedigree =" " Pedigree entry: GROUND, DUMMY, IN-FLIGHT \\
(inlist = " ") \\
(inlod= " ") \\
(mode = " q")}
\vspace{0.3cm}

\noindent where {\it infile} should have one ``load'' file name or a list of ``load'' files to be edited. You can create a list of ``load'' files with the command

\vspace{0.3cm}
\noindent
\texttt{ls *.lod >>\& filelist}
\vspace{0.3cm}


\noindent
If a list of files is 
used, the '@' symbol has to precede the list name (as in the example). DO NOT use '*.lod' in the paramenter 'infile' because it won't work.  Also, be aware that if you put the names of the FITS files here accidentally, this task will overwrite the .fits files and corrupt them, making it necessary to re-retrieve them from the deliverer's directory before you can continue. 

\noindent
{\it comments = yes} will copy \verb|DESCRIP| and all \verb|HISTORY| lines from the FITS reference file 
header, deleting what is currently present in this entry. If {\it comments = no}, nothing will 
be copied from the reference file and information present in this field will not be changed. Always set
this to {\it comments = yes} unless the deliverer specifically indicated that the comment section of the file should have information different than that of the header of the FITS file.

\noindent{\it change\_level} is the change level value. Refer to explanation above for the appropriate 
value to use. If it is left blank, the current entry will remain unchanged.

\noindent{\it opus\_flag = " " } is the opus flag value. Refer to explanation above. If it is left blank, 
the current entry will remain unchanged. 

\noindent{\it pedigree = " "} is the pedigree value. Refer to above explanation. If it is left blank, the 
current keyword value will be retained.  Since the \verb|mkload| task extracts this information from the header of the FITS file, we can leave this parameter blank.

\noindent {\it inlist, inlod} are list parameters used internally by the task. Do not enter any value 
here. 

An example of a filled ``load'' file looks like this: 

\noindent
\vspace{0.3cm}
\texttt{FILE\_NAME = 11x1\_2001\_1120\_1125\_ref\_bia.fits \\
INSTRUMENT = stis \\
REFERENCE\_FILE\_TYPE = bia \\
USEAFTER\_DATE = Nov 20 2001 00:00:00 \\
COMPARISON\_FILE= lbq12111o\_bia.fits \\
OPUS\_FLAG = Y \\
COMMENT = Superbias created by R. Diaz-Miller \\
Created on Dec 19, 2001 using the cl scripts \\
``refbias'' and "refaver", which are available \\
in the (local) xstis package within STSDAS. \\
Superbias image, combination of 98 input bias frames \\
taken in CCDGAIN=1, BINAXIS1=1, BINAXIS2=1 mode.\\ 
All input frames were from Proposal(s): \\
8901/8903 "CCD Bias Monitor". \\
The following input files were used:\\ 
o6hn2b010 \\
o6hn2c010 \\
o6hn2d010 \\
o6hn2e010 \\
o6hn2f010 \\
o6hn2g010 \\
cl script "refbias" was run on these input files, \\
after having split them up into sub-lists of less  \\
than 30 imsets each. After running "refbias" on the  \\
individual sub-lists, script "refaver" was run to  \\
average the reference files resulting from the  \\
individual "refbias" runs together.  \\
ENDHEADER \\
 \\
CHANGE\_LEVEL = SEVERE \\
PEDIGREE = INFLIGHT  \\
OBSERVATION\_BEGIN\_DATE = Nov 20 2001  \\
OBSERVATION\_END\_DATE = Nov 25 2001  \\
BINAXIS1 = 1  \\
BINAXIS2 = 1  \\
CCDAMP = D  \\
CCDGAIN = 1  \\
CCDOFFST = 3  \\
DETECTOR = CCD  \\
COMMENT =  \\
ENDROW  \\
ENDFILE }
\vspace{0.3cm}

Note that in some cases a new reference table may be identical to its predecessor with the 
exception of some rows within the table. In this case, use the \verb|CHANGE_LEVEL| from 
these rows as indicated in the delivery form and set to \verb|TRIVIAL| the unchanged row groups. Currently, the task \verb|setlodkeywd| 
can only change all of the lines in a ``load'' file to the same value. Should the file 
require multiple values, the ``load'' file will need to be edited ``by hand'' with your favorite text editor. 
An example of such a situation follows. A new PHT  table where the 
G230LB mode was updated while the G230MB mode was unchanged has the following  row 
sections: 

\vspace{0.3cm}
\noindent
\texttt{CHANGE\_LEVEL = SEVERE \\
PEDIGREE = INFLIGHT \\
OBSERVATION\_BEGIN\_DATE = May 21 1997 \\
OBSERVATION\_END\_DATE = Jul 1 1997 \\
CENWAVE = -1 \\
DETECTOR = CCD \\
OBSTYPE = SPECTROSCOPIC \\
OPT\_ELEM = G230LB \\
\verb|COMMENT| = New calibration from program 9117\\
ENDROW \\
\\
CHANGE\_LEVEL = TRIVIAL \\
PEDIGREE = GROUND \\
OBSERVATION\_BEGIN\_DATE = \\
OBSERVATION\_END\_DATE = \\
CENWAVE = -1 \\
DETECTOR = CCD \\
OBSTYPE = SPECTROSCOPIC \\
OPT\_ELEM = G230MB \\
\verb|COMMENT| = \\
ENDROW }
\vspace{0.3cm}

\ssubsubsectionstar{Check the \verb|COMPARISON\_FILE| parameter}

There are cases when the  \verb|mkload| script puts more than one entry in the \verb|COMPARISON_FILE| parameter. In those cases, all the entries should be erased manually, except
for one. Leave the most recent reference file from that list. If more than one file is in this parameter, the
delivery will fail.

\noindent
\ssubsectionstar{8. Certify the ``load'' files.}
 
After creation of the ``load'' files they also need to be certified: 

\vspace{0.3cm}
\noindent
\texttt{certify filename.lod >>\& delivery.log}
\vspace{0.3cm}

\noindent
where ``filename.lod'' can be replaced by a wildcard (*.lod). If the reference FITS file, and consequently 
the ``load'' file, uses wildcard values, -1, -999, ANY, or N/A, for any of the header keywords 
(see ICD-47), \verb|certify| will report the following ``error'' and the ``load'' file needs to be 
``expanded'': 

\vspace{0.3cm}
\noindent
\texttt{Error in opt\_elem[1]: ``any'' is not a legal value. May need to run expload \\
Error in cenwave[1]: ``-1'' is not a legal value. May need to run expload}
\vspace{0.3cm}

\noindent
{\bf Note that this applies to image reference files only}, table reference files are expanded appropiately by the \verb|mkload| script. With the term ``expanded'', we mean that the wild cards in the image ``load'' file have to be replaced with actual values.  To ``expand'' the image ``load'' files, run the CDBS task \verb|expload|. (The wild cards are usually more than one value and therefore the ``expload'' name.)

\vspace{0.3cm}
\noindent
\texttt{expload filename\_in.lod filename\_out.lod /store/smalls/cdbs/tools/data/\#\#\#\#.rule }
\vspace{0.3cm}

\noindent
where the ``{\it filename\_in.lod}'' file can be the same as ``{\it filename\_out.lod}''; in which case the changes will be written in the same file (note that this task does not take wildcard syntax on the command line). Expload expands 
the ``load'' file in those cases where a single reference file is to be used for many 
modes. For example, suppose we have a reference file that is applicable for ANY optical element (\verb|OPT_ELEM|) of
the STIS spectroscopic observing modes. \verb|Expload| will ``expand'' the ``load'' file to cover all legal \verb|OPT_ELEM|
values, providing one row section in the ``load'' file for each of the \verb|OPT_ELEM| values. The expansion of the files 
is governed by the so-called ``$\#\#\#\#$.rule'' file, where $\#\#\#\#$ is replaced by the instrument name. The rule files for each instrument are located in the CDBS data directory /store/smalls/cdbs/tools/data/ in the \verb|smalls.stsci.edu| domain or  /grp/hst/cdbs/tools/data/ for Solaris and Mac systems. This file shows the current legal values that will be used to replace wildcard values in 
the expansion. In the above example, the expanding rule for combination \verb|OBSTYPE=SPECTROSCOPIC| 
and \verb|OPT_ELEM=ANY| (taken from the stis.rule file) 
is: 

\vspace{0.3cm}
\noindent
\texttt{OBSTYPE = SPECTROSCOPIC \&\& OPT\_ELEM = ANY => \\
OPT\_ELEM=G140L || OPT\_ELEM=G140M || OPT\_ELEM=E140M || \\ 
OPT\_ELEM=E140H || OPT\_ELEM=G230L || OPT\_ELEM=G230M || \\
OPT\_ELEM=E230M || OPT\_ELEM=E230H || OPT\_ELEM=PRISM || \\
OPT\_ELEM=G230LB || OPT\_ELEM=G230MB || OPT\_ELEM=G430L || \\ 
OPT\_ELEM=G430M || OPT\_ELEM=G750L || OPT\_ELEM=G750M || \\
OPT\_ELEM=X140H || OPT\_ELEM=X140M || OPT\_ELEM=X230H || \\
OPT\_ELEM=X230M; }
\vspace{0.3cm}

\noindent
That is, the row section with \verb|OPT_ELEM = ANY| will be replaced by several row sections, one for \verb|OPT_ELEM=G140L|, another for \verb|OPT_ELEM=G140M|, etc.  Once the file has been properly exploaded, run \verb|certify| again until there are no errors or 
expansions required. Repeat this step as necessary until \verb|certify| does not report any missing 
keyword information. 

In order to simplify this work, one script has been created to expand several files of the same kind at once. The script is called \verb|multi_expload| and is located in the bin/ directory of the delivery account in SMALLS. This script needs the information of the instrument to which these files apply and the extension of the file (in this case it is the last characters of the file name and not the type of reference file). For example, to run this command for ACS dark reference files with names ``*drk\_new.fits'', type:

\noindent
\vspace{0.3cm}
\texttt{multi\_expload drk\_new acs}
\vspace{0.3cm}

\noindent
Note that you first have to give the extension of the file and then the instrument name.  If you are delivering files with different extensions, you have to run this script for each extension. 

\ssubsectionstar{9. Run \verb|check\_load|}

The CDBS task, \verb|check_load|, must be run on the ``load'' files before they can be delivered. 
This task takes wildcards. 

\vspace{0.3cm}
\noindent
\texttt{check\_load *.lod >>\& delivery.log}
\vspace{0.3cm}

\noindent
The output from this task will look like the following: 

\vspace{0.3cm}
\noindent
\texttt{starting check\_load \\
database cdbs\_ops \\
server CATLOG \\
Thu Feb 26 11:25:26 EST 1998 \\
load file: i2916173o\_drk.lod \\
header file: i2916173o\_drk.fits \\
Wrote file (i2916173o\_drk.lod) \\
no differences for file i2916173o\_drk.lod}
\vspace{0.3cm}

\noindent
\ssubsectionstar{10. Rename the files to have an unique name identifier}

Once the reference files are ready to be ingested into the databases, the files have to be renamed with an unique name identifier. 

\ssubsubsectionstar{SYNPHOT}

In the case of SYNPHOT  tables an incremental number format is used for the naming of new data files. When
the SYNPHOT files were delivered they are likely to have been already re-named to the next value; however, make sure that this is the case by checking the SYNPHOT disk area ( /grp/hst/cdbs/comp/ in the centralized storage area or /store/smalls/ref/ in the \verb|smalls.stsci.edu| domain). Identify the type of file by its name
and check that indeed the number of the new file does not exist. In some cases, the instrument teams
choose to skip values.  This is not a problem and you can deliver the files that way provided that the
number does not previously exist and it is greater than that of any previously delivered file of that type.   If these files do not have an unique name, however, use the task \texttt{uniqname} as described for the pipeline reference files.  Be sure in this case that the assigned number is larger than that of the older files.  We have found instances where \texttt{uniqname} assigns a value of ``001" when this type of file has been previously delivered many times.

Note that if SYNPHOT Atlas files (e.g. Kurucz) and HST Calibration Standards spectra are delivered, those have special names that cannot be changed and therefore the same file name should be used to replace the old file.

\ssubsubsectionstar{Pipeline Reference Files}
For the case of calibration reference files, assign an unique name using the CDBS script called \verb|uniqname|:

\vspace{0.3cm}
\noindent
\texttt{uniqname *.lod >>\& delivery.log}
\vspace{0.3cm}

\noindent
The files will be renamed to CDBS style reference file names.  More details about the naming conventions used by the script are described in the CDBS documentation web page (http://www.stsci.edu/instruments/observatory/cdbs/documents/).

Note that the WFPC2 team usually renames the files themselves. This is because, at least for bias and darks reference files, they use an automatic script that does this step. In these cases we don't have to run \verb|uniqname| on the files .

\noindent
\ssubsectionstar{11. SYNPHOT Bandpases}

Non HST bandpasses files are a special case of SYNPHOT files. These are delivered to CDBS and are part of the SYNPHOT package. Unlike the atlases, these are in the CDBS database and as such should be included in the TMC table.  Examples of these files are the Landolt b, i, r, u, v filter bandpasses and the  Stromgren B, U, V, Y.
These files are located in the directory /grp/hst/cdbs/comp/nonhst/ and follow the same numbering convention as any of the SYNPHOT files. Given that those files are non HST filters and used for normalization by all the HST instruments, these should in practice be tested by all the instrument teams. 

If the person making the update or requesting it has not communicated about this change to all the instrument teams, we should make sure we let all the instrument teams know the reason behind this change, the person making the change,  the location of these file and the timeline for testing and sign-off for the delivery of these files.  The instrument teams have to test SYNPHOT and if possible the ETCs, before this file can be delivered. The test procedures should be left to the instrument teams; however, if this work is done by only one person familiar with the use of these files  with SYNPHOT, the instrument teams should agree that this person will be the tester and sign-off  to this effect via an e-mail to the INS/CDBS team. If the instrument teams decide to perform the test themselves, they should sign-off the files for delivery once testing is completed. These files cannot be delivered until sign-off from all the instrument teams is obtained. There might be cases where there is a pressing need to deliver these files and therefore it is important to make sure a deadline is set for the delivery of these files.  This deadline should be clearly given to the instrument teams when letting them know about the change in the files. If any of the teams has a problem with meeting this deadline, contact the CDBS Lead for assistance in working out an schedule with them. 

\noindent
\ssubsectionstar{12. Put the files in a delivery directory and deliver them to the DMS}

In the case of pipeline reference files or SYNPHOT Throughput tables, copy the FITS and ``load'' files to be delivered to an empty directory. The delivery script does not work if there are other files in this directory. Some empty directories already exist for this purpose. These are under 
/calib/cdbs\_delivery/ directory and have names ``deliverfiles*''. Deliver the reference files to the CDBS database, and when applicable to the OPUS and DADS databases, using the \verb|sendit| script:

\vspace{0.3cm}
\noindent
\texttt{sendit >>\& ../workingdir/delivery.log}
\vspace{0.3cm}

\noindent
where the ``workingdir'' is the delivery path where the files were tested before delivery. This script will re-check that the files are FITS and CDBS compliant, will create SQL command inputs for the databases, and will copy the delivered files to a fixed  location from where the Data Management System Teams  will collect the data.

In the case of WFPC2 files, \verb|sendit| converts the GEIS files to ``waiver" FITS type before they are delivered to the databases or DMS disks.
In the particular case of  SYNPHOT Atlas files (e.g. Kurucz), the files cannot be delivered this way so skip this step. 
 
More detailed information on the steps performed by \verb|sendit|  will be described in another TIR.
An example of the \verb|sendit| output for a successful delivery is:

\vspace{0.3cm}
\noindent
\texttt{{\footnotesize You start your delivery process at:Mon Apr  4 19:39:02 GMT 2005\\
starting deliver\_cdbs\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:02 GMT 2005\\
---------------------------------------------\\
starting certify\_delivery\\
Mon Apr  4 19:39:02 GMT 2005\\
== Checking p441909no\_pht.fits ==\\
== Checking p441909no\_pht.lod ==\\
certify\_delivery succeeded\\
---------------------------------------------\\
starting loopfits\_delivery\\
Mon Apr  4 19:39:03 GMT 2005\\
converting:\\
created output file loopfits.out\\
loopfits\_delivery succeeded\\
---------------------------------------------\\
starting farris\_fitsverify\_delivery\\
Mon Apr  4 19:39:03 GMT 2005\\
no errors or warnings reported by farris\_fitsverify\\
created output file farris\_fitsverify.out\\
fitsverify\_delivery succeeded\\
---------------------------------------------\\
starting check\_load\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:03 GMT 2005\\
load file: p441909no\_pht.lod\\
header file: p441909no\_pht.fits\\
Wrote file (p441909no\_pht.lod)\\
no differences for file p441909no\_pht.lod\\
lcheck\_load succeeded\\
---------------------------------------------\\
starting cdbs\_sql\_gen\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:05 GMT 2005\\
{\bf delivery\_number = 11560}\\
lock acquired\\
load file(s) p441909no\_pht.lod\\
Processing p441909no\_pht.lod ...\\
\\
Warning: No comparison file records matched mode values for row 6.\\
         New equivalence class values and a SEVERE change\_level were used\\
\\
Processing complete -- cdbs\_delivery11560.sql generated\\
cdbs\_sql\_gen succeeded\\
---------------------------------------------\\
starting run\_delivery\_sql\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:08 GMT 2005\\
/calib/cdbs\_delivery/deliverfiles2/cdbs\_delivery11560.sql.out\\
using file /calib/cdbs\_delivery/deliverfiles2/cdbs\_delivery11560.sql\\
no errors in processing sql file /calib/cdbs\_delivery/deliverfiles2/cdbs\_delivery11560.sql\\
created output file /calib/cdbs\_delivery/deliverfiles2/cdbs\_delivery11560.sql.out\\
run\_delivery\_sql succeeded\\
---------------------------------------------\\
starting check\_cdbs\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:09 GMT 2005\\
delivery 11560 in progress\\
missing modes check\\
uni check\\
synphot compname check\\
expansion number check\\
archive date check\\
general availability date check\\
opus load date check 1\\
opus load date check 2\\
row check\\
file check\\
reject check 1\\
reject check 2\\
current reject check 3\\
current delivery number check\\
reject check 4\\
\\
output file check\_cdbs\_11560.out created\\
       4 warning(s): see output file check\_cdbs\_11560.out\\
No errors.\\
check\_cdbs succeeded\\
---------------------------------------------\\
starting opus\_sql\_gen\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:39 GMT 2005\\
\\
created opus\_11560\_o.sql\\
opus\_sql\_gen succeeded\\
---------------------------------------------\\
starting update\_ga\_date\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:39 GMT 2005\\
delivery number =  11560\\
general availability date was updated\\
cdbs\_ops\\
lock released\\
update\_ga\_date succeeded\\
---------------------------------------------\\
starting opus\_catalog\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:39:40 GMT 2005\\
delivery\_number= 11560\\
instr= o\\
catalog file= opus\_11560\_o.cat\\
opus\_catalog succeeded\\
---------------------------------------------\\
deliver\_cdbs completed\\
Mon Apr  4 19:39:41 GMT 2005\\
\\
total execution times:\\
\\
real       39.1\\
user        7.0\\
sys         7.4\\
\\
\\
\\
\#\#\#\#\#  \#\#\#\#\# \\
\\
\\
...CDBS process done...Making links for delivery pick-up...
linking p441909no\_pht.fits\\
linking opus\_11560\_o.cat\\
linking opus\_11560\_o.sql\\
\\
\\
 \#\#\#\#\#  \#\#\#\#\# \\
\\
...You have successfully finished the delivery process...\\
\\
 \#\#\#\#\#  \#\#\#\#\#   \#\#\#\#\#  \#\#\#\#\# \\
\\
Process finished at:Mon Apr  4 19:39:42 GMT 2005\\
\\
\\
 \#\#\#  \#\#\#   \#\#\#  \#\#\# \\
 \#\#\#  \#\#\#   \#\#\#  \#\#\# \\
 \#\#\#  \#\#\#   \#\#\#  \#\#\# \\
 \#\#\#  \#\#\#   \#\#\#  \#\#\#}
}
\vspace{0.3cm}

  
The \verb|sendit| script performs the basic tests and creates SQL command input files. After these are completed, a unique delivery number is assigned (indicated in bold in the above output). If the delivery happens to fail after this number has been assigned, the delivery has to be cancelled before another delivery or redelivery can occur. This is done running the command \verb|delete_delivery|:

\vspace{0.3cm}
\noindent
\texttt{delete\_delivery >>\& ../workingdir/delivery.log}
\vspace{0.3cm}

\noindent
 This will unlock the databases and will correctly exit the delivery process.  An example of a failed delivery is:
 
 \vspace{0.3cm}
\noindent
\texttt{{\footnotesize run\_delivery\_sql succeeded\\
---------------------------------------------\\
starting check\_cdbs\\
database  cdbs\_ops\\
server  CATLOG\\
Mon Apr  4 19:16:25 GMT 2005\\
delivery 11559 in progress\\
missing modes check\\
uni check\\
synphot compname check\\
expansion number check\\
archive date check\\
general availability date check\\
opus load date check 1\\
opus load date check 2\\
row check\\
file check\\
reject check 1\\
reject check 2\\
current reject check 3\\
current delivery number check\\
reject check 4\\
\\
output file check\_cdbs\_11559.out created\\
       4 warning(s): see output file check\_cdbs\_11559.out\\
       1 error(s): see output file check\_cdbs\_11559.out\\
CDBS ERROR: check\_cdbs failure. Exiting.\\
\\
real       33.7\\
user        6.4\\
sys         5.3\\
FAILURE of deliver\_cdbs.}}
\vspace{0.3cm}

\noindent
In this example, the first successful lines of the \verb|sendit|'s output are not shown.
 If the delivery fails before the delivery number has been assigned, the \verb|delete_delivery| command does not need to be run.
 
 
\noindent
\ssubsectionstar{13.  Fill the delivery form and e-mail it to DMS.}

If the \verb|*.lod| files have \verb|OPUS_FLAG = Y|  (e.g. all pipeline reference files), immediately after the delivery software (or \verb|sendit|) successfully populated the CDBS database, you have to notify DMS of the delivery; so they can check that the files are ingested properly in the different DMS areas. When \verb|OPUS_FLAG = N|, the files are sent only to the CDBS database, as those do not affect the pipeline calibration products and should only be used by SYNPHOT. In the latter case DMS should not be notified. For those cases when a notification is needed, submit an e-mail to the e-mail address {\bf cdbs\_datamng@stsci.edu} using the following formatted form (a template of this form is in the file /calib/cdbs\_directory/form):

 \vspace{0.3cm}
\noindent
\texttt{Date: \\
By: \\
Instrument: \\
File\_type(s) (e.g. PHT, DRK): \\
Directory where data is found:\\
/calib/cdbs\_delivery/.../2005\_...\_.../\\
\\
Description of data delivered:\\
\\
\\
Delivery\_number: \\
\\
Opus ingest date:\\
Opus signoff}
\vspace{0.3cm}

\noindent
where {\it Date} is today's date. In {\it By:}, put your name; in {\it Instrument}, put the name of the instrument or team for which you are delivering the reference files; in {\it File\_type(s),} put the extension (e.g. PHT, DRK) of the file delivered. The information in {\it Directory where data is found:} is for our records. This directory is the directory where you tested the files before delivery, i.e. your working directory. In this case replace the ``...'' by the appropriate values according to the instrument and the date of delivery. In the section {\it Description of data delivered:} list the datasets  delivered and the {\it opus*} ASCII files that were created by the script \verb|sendit|. For example, use the \verb|ls -la|:

\vspace{0.3cm}
\noindent
\texttt{-rw-rw-rwx   4 srefpipe      108 Mar 25 16:56 opus\_11556\_j.cat*\\
-rw-rw-rwx   4 srefpipe        1499 Mar 25 16:56 opus\_11556\_j.sql*\\
-rw-rw-rwx   4 srefpipe   164160 Mar 25 16:53 p3p1650tj\_mdz.fits*\\
-rw-r--r--       1 srefpipe        1046 Mar 25 16:53 p3p1650tj\_mdz.lod}
\vspace{0.3cm}

\noindent
and copy and paste this information to the delivery form. In the case of WFPC2 files, list only the ``waiver" FITS files and ``opus*'' files. Finally, in {\it Delivery\_number} put the number of the delivery. The subject of this e-mail has to be:
{\it Delivery \#\#\#\#\#}, where \#\#\#\#\# is the number of the delivery. The last two fields (opus ingest date and opus ingest signoff) are left blank and will be filled by the OPUS team's person ingesting the file.   Save a copy of the completed form in the directory \verb|/calib/cdbs_directory/deliveryforms/| with a filename of the format {\it yyyy\_mm\_dd\_ins} where ins is the instrument, such as acs, wfpc2, etc.

\noindent
\ssubsectionstar{14. Transfer the files to the centralized storage area}

Currently  the deliveries are made in the SunFire15K system and these disks cannot be mounted in the Science  Cluster. Therefore, a copy of the reference files  have to be transfered to the directories located in the centralized storage area. In this area, each of the instrument teams has an assigned area to store the reference files, one for pipeline reference files and another for SYNPHOT reference files. Each of these directories can be accessed from the /grp/hst/cdbs/ directory. The specific disk location for each of the instrument teams is:

\noindent
\begin{table}[htbp]
  \centering
  \begin{tabular}{lcc} % column spec
  \hline
  \hline
  &\multicolumn{2}{c}{Directory Path} \\
   &\multicolumn{2}{c}{---------------------------------------}\\  
  &\multicolumn{2}{c}{reference files} \\
  &\multicolumn{2}{c}{/grp/hst/cdbs/}\\
 Instrument & calibration &SYNPHOT \\
                     
\hline
ACS & jref/ &comp/acs/ \\
STIS& oref/ & comp/stis/ \\
WFPC2&uref/ &comp/wfpc2/ \\
NICMOS&nref/ &comp/nicmos/ \\
WFC3&iref/&comp/wfc3/\\
COS&lref/&comp/cos/\\
non-HST&-&comp/nonhst/\\
OTA&-&comp/ota/\\
TMC/TMG&-&mtab/\\
Atlases& - & grid/\\
CALSPEC&calspec/&current\_calspec/\\
 \hline
  \end{tabular}
%  \caption{Basic column alignment}
%  \label{tab:basic-align}
\end{table}

In the case of WFPC2, transfer both the GEIS and ``waiver'' FITS files to the centralized storage area.  With some types of WFPC2 files, there is a second set of files to be used only on linux systems.  This will be indicated in the delivery form sent by the WFPC2 team.  The linux version of the files will be in a separate directory to be retrieved by the CDBS team.  The CDBS deliverer will download the linux files to a linux directory within the delivery directory, but the linux files do not go through the delivery process with the Solaris files.  Once the Solaris files have been delivered,  the GEIS linux files (not the lod files) will need to be transferred to their own directory, \verb|/grp/hst/cdbs/uref_linux/|.

In the case of SYNPHOT data files, this step should be done before creating the TMC file with the \verb|MKCOMPTAB| task, as the software looks in the above mentioned disk location for the files that appear as active in the CDBS database. If the files
are not present in these disk locations, the task will fail. Currently, this task runs in \verb|smalls|, however, we need to copy these throughput files to the \verb|smalls| reference files area (\verb|/store/smalls/ref/thu/|) because these are not copied automatically by the ``sendit" script.

In the case of SYNPHOT Atlases, FTP the files to their corresponding \verb|grid| directory in the centralized storage area.  This will make them accessible to all internal users. In order for these atlases to be used by SYNPHOT outside the Science Cluster (including MAC users), users have to copy them to their corresponding /grid/ directories (where all the SYNPHOT Atlases and Libraries live) or wait for the next release where the files will be automatically delivered to them with the SYNPHOT package. Note that copying them to the /grp/hst/cdbs/grid/ directory in the centralized storage area makes these files available to the STSDAS Group for future STSDAS releases or for download from their web site. In any case, {\bf it is necessary to notify the STSDAS Group} that these Atlases or the HST Calibration Standards spectra have changed so they can package them in the appropriate tarball. See Apendix C  for further information regarding the needed test and delivery procedures for these files.

To transfer the files, SFTP or FTP to the centralized storage area. Use the {\bf srefpipe} account name and password. Change the transfer mode to {\it binary}  (if using FTP) and put in all the FITS files (GEIS and ``waiver'' FITS in the case of WFPC2), into the respective directory. For example,

\vspace{0.3cm}
\noindent
\texttt{mymac> sftp srefpipe@tib.stci.edu\\
Connecting to tib.stsci.edu...\\
Password: \\
sftp> cd /grp/hst/cdbs/jref/ \\
sftp> mput *.fits
}
\vspace{0.3cm}

\noindent
\ssubsectionstar{15. In the case of SYNPHOT files: Create the TMC table and deliver it  with the TMG}
\vspace{-0.5cm}
{\bf and TMT tables}

\vspace{0.2cm}
In the case of SYNPHOT data files, once these have been delivered, it is necessary to re-create the TMC table using the CDBS script \verb|mkcomptab|.  In your testing directory or a subdirectory created just for the TMC, TMG and TMT tables, run:

\vspace{0.3cm}
\noindent
\texttt{mkcomptab new\_tmc.fits}
\vspace{0.3cm}

\noindent This script will recreate the TMC table using the information located in the CDBS database. It will use the most up to date files to fill each of the \verb|COMPONENT| rows in this file.  However, if it encounters more than one file with the same \verb|USEAFTER| date, it will list all of them in the TMC table. The MKCOMPTAB task will list these files from older to newer. This order is correct for SYNPHOT, however, Pysynhot uses the files in the opposite direction. Currently, Pysynphot supports only some COS calculations (in order to support the ETCs), however, we are in the process of transitioning from SYNPHOT to Pysynphot  and  therefore the TMC files should not have repeated entries, to avoid a conflict between Pysynphot and SYNPHOT. In order to prevent repeated entries the USEAFTER date has to be different for each of the files delivered. This is why it is important to set the USEAFTER date of SYNPHOT throughput tables to the date when the file is created or delivered. Make sure this is the case when you check the changes in this file in order to prevent repeated entries.
More details about this script can be found in the CDBS Documentation web page (http://www.stsci.edu/instruments/observatory/cdbs/documents/).  

Once the TMC file has been created, check that the changes in this file are for those SYNPHOT data files just delivered. For this, run the IDL procedure \verb|compare_table.pro|. The script is located in the Science Cluster in the STScI IDL area (\verb|/data/garnet2/idl/stsci/|) and in the smalls area (\verb|/store/smalls/srefpipe/useful_scripts/|). Keep in mind that to run IDL in smalls, you must go into the .setenv file for smalls (srefpipe account) and comment out the line that says ``source \verb|~/def/opus_login.csh|", save the file, and open a new window that will then run IDL.  To compare the old and new TMC tables type in IDL:

\vspace{0.3cm}
\noindent
\texttt{IDL> .compile /store/smalls/srefpipe/useful\_scripts/compare\_table.pro}

\noindent
\texttt{IDL> compare\_table,'path1/new\_tmc.fits','path2/old\_tmc.fits',\$\\
COLUMNS=['compname','filename'],SAVEFILE=1}
\vspace{0.3cm}

\noindent
The older TMC tables are located in \verb|/store/smalls/ref/mul|, and the current version of the TMC table is the last one listed.  The \verb|COLUMNS| parameter indicates which set of columns to use for the comparison of each row of the file. When the \verb|SAVEFILE| parameter is set equal to 1, it will direct the output of the procedure to a file called \verb|compare_table.out| in the current directory. This script looks for missing elements in the table by checking differences in each row.  
If no unexpected differences are found, fill the \verb|HISTORY| section of the FITS file documenting the reason for the file to be re-created.

If the TMG and/or TMT tables were received together with the SYNPHOT data files, copy these files to the directory where the TMC file was created and check that the header keywords are correctly populated. Also check that the changes in the file are as expected. For this, use the same IDL procedure \verb|compare_table.pro|. For the TMT table the \verb|COLUMNS| parameters are set identical to those used for the TMC table, while for the TMG table these should be \verb|['compname','keyword','innode','outnode','thcompname']|. We do expect the deliverer to have done this test already; but we have to confirm the changes.

 If no problems or unexpected differences are found, \verb|fitsverify| and \verb|certify| the TMG, TMT, and TMC files; as in steps 4 and 5. Create the ``load'' files as in step 6. Fill the fields  \verb|CHANGE_LEVEL| and \verb|PEDIGREE| of the TMC ``load'' file value used by the throughput tables that were just delivered. That is, if the throughput  tables had \verb|CHANGE_LEVEL = SEVERE| use this value for the \verb|CHANGE_LEVEL| of the TMC ``load" file. For the TMG and TMT ``load'' files always use SEVERE. For these three files use \verb|OPUS_FLAG=Y|. Run \verb|certify|,  \verb|check_load|,  and \verb|uniqname| on the ``load'' files according to steps 8, 9 and 10.  Before these files can be delivered, however, they must be tested against the SSB suite of regression tests.  To accomplish this, the current procedure is to place the TMC, TMG and the latest TMT file in /grp/hst/cdbs/work/vicki/ for Vickie Laidler and e-mail her so that she knows where the files are and that they need to be tested. Note that if the TMT file and TMG files were not delivered, you need to find the latest delivery files and put them in Vicki's directory together with the new TMC file. This is because she needs these files to be in the same directory to do her test. Only after receiving an e-mail from her confirming that the files are ready for delivery should the files be delivered as described in step 11.  Then send a delivery form to DMS as in step 12, and finally transfer the files to centralized storage as in step 13  and copy only the new file(s) to the \verb|/store/smalls/ref/mul/| directory. 

\ssubsectionstar{16. Run cdbs\_report}

After the CDBS delivery Pipeline completes all the stages succesfully, an e-mail acknowledging the completion of the delivery is sent back to the INS/CDBS member delivering the files. (A copy is sent to the cdbs@stsci.edu e-mail address.) This usually happens the same day the files were delivered. Note that in the case when the files were delivered late in the day, the acknowledgment of the ingest will arrive the next day. If the reply e-mail is not received within the expected time, investigate the reason for the delay. The OPUS
team usually notifies us of the successful ingestion after the files have been properly transferred to the Archive,
OPUS, and the mirror sites (ECF and CADC) disks. Although a problem in any of these steps can delay the notification, after the files have been ingested into the Archive and OPUS disk areas, the files will be used in the OTFR pipeline and will be available for retrieval. But before these files can be recommended as the best reference files for a given dataset, it is necessary to run another script that updates the archive database. The *\_ref\_data tables in the archive database are used to select the best reference files via the ``Best Reference Files'' option in the archive retrieval form.  The script that updates these tables is run only once a day (usually at night) and therefore there is a period of time when the files used in OTFR are different than those selected by the ``Best Reference Files'' option. This does not affect our delivery, but it is something to keep in mind. In any case, once OPUS has ingested the files, we can assume the files are in the system. The information on when the files were ingested into the Archive and OPUS system can be obtained running the \verb|cdbs_report| script:

\vspace{0.3cm}
\noindent
\texttt{cdbs\_report \#\#\#\#\#}
\vspace{0.3cm}

\noindent
where \#\#\#\#\# is the delivery number. For example, running this script for delivery number 11160 shows all the information relevant to that delivery number:

\noindent
\begin{table}[htbp]
\centering
  \begin{tabular}{lccccc} % column spec
  \multicolumn{6}{c}{{\footnotesize CDBS Installation Report }} \\
   \multicolumn{6}{c}{---------------------------------------}\\  
{\footnotesize Instrument}& {\footnotesize Reference }&{\footnotesize File Name}  &{\footnotesize  Useafter Date}&{\footnotesize  Archive Date}&{\footnotesize  OPUS Load Date}\\
&{\footnotesize File Type}&&&& \\
------------&------------&---------------&-------------&--------------&-----------------\\  
{\footnotesize STIS}& {\footnotesize PHT}&{\footnotesize P441909NO\_PHT.FIT}&{\footnotesize Mar 15 1999 12:00AM }&{\footnotesize Apr  4 2005  8:32PM }&{\footnotesize Apr  4 2005  8:32PM}\\
{\footnotesize STIS}&{\footnotesize PHT}&{\footnotesize P441909OO\_PHT.FIT}&{\footnotesize Oct  1 1996 12:00AM }&{\footnotesize Apr  4 2005  8:32PM} &{\footnotesize Apr  4 2005  8:32PM}\\
{\footnotesize STIS}&{\footnotesize TDS}&{\footnotesize P441909PO\_TDS.FIT}&{\footnotesize Oct  1 1996 12:00AM }&{\footnotesize Apr  4 2005  8:32PM} &{\footnotesize Apr  4 2005  8:32PM}\\
  \end{tabular}
%  \caption{Basic column alignment}
%  \label{tab:basic-align}
\end{table}

\ssubsectionstar{17. Check the size of the files in the archive}

We have found several cases where the tables that are ingested into the archive are corrupted. We believe that this happened when the files were ftped to the archive media. Since we are now delivering the files from the SunFire15K system, where the operational and archive domains reside, this problem is likely to have been solved; however, we should still perform this check to verify the integrity of the files that are being archived. This can be done by comparing the size of the archived reference files to that of the files we have in our delivery directory. This verification can be done in three different ways, all by performing a query to the {\it archive\_files} table in the database {\bf dadsops} in the ZEPPO server. 

The simplest way is by running the CSHELL script \verb|search_size_csh| .
This script was created to check the size of the files and is available in the special
CDBS account only. It makes the appropriate calls to the database using the provided dataset name search string. To run this script type:

\vspace{0.3cm}
\noindent
\texttt{smalls> search\_size\_csh NNNNN
}
\vspace{0.3cm}

\noindent
where \verb|NNNN| should be the file name or file name prefix to search. Note it has to be entered as caps. If the file
prefix is not provided, the script will request it. The output to this script is a file called \verb|size_out.txt| which has the commands and output of the SQL database search.

If the script is not available, you have to perform each step manually. The first two steps would be to set the environment variable \verb|DSQUERY| and load the {\it dadsops} database. For this, type:

\vspace{0.3cm}
\noindent
\texttt{mymac>setenv DSQUERY ZEPPO\\
mymac>isql\\
1>use dadsops\\
2>go}
\vspace{0.3cm}

\noindent
The column we want to search here is: {\it afi\_data\_set\_name}. The rows we want to examine are those that have values  equal to the name of the reference file we just delivered. For example, if we just delivered reference files with names p441909so\_drk.fits and p441909to\_bia.fits, we can check the size of the files in the archive with the command:

\vspace{0.3cm}
\noindent
\texttt{1>select * from archive\_files where afi\_data\_set\_name like ``P441909\%''\\
2>go}
\vspace{0.3cm}

\noindent
where ``\%'' is a wild card. In this case, using the wild card will simplify the verification by showing us the size of all the reference files  with prefix \verb|"P441909\%"|. 
Note that we are using uppercase  for the file name. This is because the names of the reference files are stored as uppercase. The output of this command looks like this:

\vspace{0.3cm}
\texttt{       afi\_data\_set\_name      \ \ \ \ \ \ \ \                   afi\_archive\_class\\
\ \ \ \ \ \   afi\_generation\_date \ \ \ \ \ \        afi\_mission afi\_file\_extension\\  
 afi\_file\_name                                                                            \\           
\hspace{0.5cm}        afi\_file\_type afi\_pre\_compress\_size afi\_post\_compress\_size\\
\hspace{0.5cm} afi\_checksum \ \ afi\_verify\_status \ \  \ afi\_virtual \\
 --------------------------------------- -----------------\\
         -------------------------- ----------- --------------------\\
         ----------------------------------------------------------------------------------------------------\\
         ------------- --------------------- ----------------------\\
         ------------ ----------------- ----------- \\
 P441909NO   \ \ \ \ \ \ \ \ \ \ \                             CTB              \\
                Apr  4 2005  8:06PM \ \ \ HST     \ \ \ \    PHT   \\              
         p441909no\_pht.fits                                    \\                                              
         FITS    \ \ \ \ \ \ \              4518720.000000   \ \ \ \      2700547.000000\\
           1336725531 \ \ NULL  \ \ \ \ \             N           \\
 P441909OO             \ \ \ \ \ \ \ \ \ \ \ \                   CTB              \\
                Apr  4 2005  8:06PM\ \ \ HST \ \ \ \         PHT\\                 
         p441909oo\_pht.fits                                     \\                                             
         FITS        \ \ \ \ \ \ \         4518720.000000 \ \ \ \        2700418.000000\\
            759764626 \ \ NULL        \ \ \ \ \       N           
            }
\vspace{0.3cm}

To simplify the output you could select the {\it afi\_pre\_compress\_size} column only. For this the
command should be:

\vspace{0.3cm}
\noindent
\texttt{1>select afi\_data\_set\_name,afi\_pre\_compress\_size  \\
2> from archive\_files where afi\_data\_set\_name like "P441909\%"\\
2>go}
\vspace{0.3cm}

\noindent
The output will look like this:

\vspace{0.3cm}
\texttt{       afi\_data\_set\_name    \ \ \ \ afi\_pre\_compress\_size \\
 ---------------------------------------  ------------- \\
 P441909NO  \ \ \ \  4518720.000000  \\
 P441909OO   \ \ \ \  4518720.000000 }
\vspace{0.3cm}

Another way to do this is by entering the SQL commands listed above in an ASCII file; for the example used here the file is called \verb|size_query.sql|. Once the file has been created, run the following in the command:

\vspace{0.3cm}
\noindent
\texttt{smalls> isql -e -i size\_querry.sql  -o size\_out.txt -S ZEPPO
}
\vspace{0.3cm}

\noindent
the output will be directed to a file called \verb|size_out.txt|.

In all cases, the last step is to compare the {\it afi\_pre\_compress\_size} column value with the size of the file you have in your delivery directory. If these values are not identical, it is likely that the file in the archive is corrupted and  the OPUS team has to be informed of the problem.

\ssubsectionstar{18. Verify the correct usage of the reference files in the operational environment.}

Another problem that we have seen in the past has to do with the way the new reference files were recommended. In a few cases, the files were 
not ingested properly and the old reference file was still being recommended for some 
datasets when it should not have been. Therefore, it was decided to verify that the reference files were used correctly by OPUS and properly recommended in the archive.  This should be done only after the nightly script that updates the *ref\_data tables, containing information about the Best reference files has completed. To make sure that this has run, wait until the day after the files were delivered to CDBS to perform this test. Note that since there is no science data for COS or WFC3 yet, this step should be skipped for these instruments until after SM4. When HST starts to take science observations with these instruments, their {\it ref\_data} tables will start to get entries and therefore this step can be performed.

For the other instruments, the simplest way to do this is by running the CSHELL script \verb|search_best_reffile|.  This script was created to facilitate verification about the usage of the reference files.  Currently, it only supports ACS, WFPC2, and STIS reference files.  This script sends the request to the DADSOPS database for the best reference files used after a given date.  It returns the list of best reference files used for a particular instrument, detector, reference file type, and time of observation.  Just like the script that checks the size of the reference files, this script can only be run in the special CDBS account.

\noindent
This script can be run as follows:

\noindent
\texttt{smalls> search\_best\_reffile INSTRUMENT TYPE\_FILE USEAFTERDATE DETECTOR}

\noindent
where INSTRUMENT is the INSTRUMENT name for which the search is made.  TYPE\_FILE is the type of reference file we want to use in the search; e.g. DRK, BIA, PHT, or drk, bia, pht.  USEAFTERDATE is the date after which we want to verify the usage of the reference files.  In this case, it should correspond to the earliest USEAFTER date of the reference file type we are delivering.  The format for the USEAFTERDATE is as follows: Month Day Year Time.  Finally, DETECTOR is the name of the detector for which this file applies.  If there is no detector distinction for the reference files (e.g. WFPC2 DRK reference files), this parameter can be left blank.

For the moment, this script does not seem to work in an xterm or xgterm.  You may need to open a basic terminal in order for the script to work.  An example of using this script to find WPFC2 darks is:

\noindent
\texttt{smalls> search\_best\_reffile WFPC2 DRK Jan 21 2008 04:10:00}

\noindent
If the script is not available, you will have to perform each step manually.  In order to do this, first we need to know the  prefix of the field name, in the tables with reference file data, that is associated to the reference files. That is, the reference file records are located in tables named ``\#\#\#\_ref\_data''; where \#\#\# is the name of the instrument (e.g. acs\_ref\_data).  Within these tables the reference files are listed in columns named after the calibration reference file name that appears in the header of the observation FITS files. For example, the ``Pixel to pixel flat field'' file for ACS data is assigned by the keyword PFLTFILE. The column in the {\it acs\_ref\_data} that contains this information is {\it acr\_best\_pfltfile}. Note the prefix used for the reference file column, these change from instrument to instrument but are the same for all the reference files of that instrument. Here is the list of prefixes :

\noindent
\begin{table}[htbp]
  \centering
  \begin{tabular}{lcc} % column spec
    \hline
  \hline
  \multicolumn{2}{c}{ZEPPO database} \\
   \multicolumn{2}{c}{table \#\#\#\_ref\_data}\\  
   &\\
    \hline
 Instrument &prefix  \\
\hline
 ACS &acr\_best\_\#\#\# \\
STIS& ssr\_best\_\#\#\#\\
WFPC2&w2r\_best\_\#\#\# \\
NICMOS&nsr\_best\_\#\#\#\\
\hline
  \end{tabular}
\end{table}


\noindent
where \#\#\# is the reference file table identifier. In appendix A of this document the corresponding names for the current reference files for all the instruments are listed. The DSQUERY environment variable as well as the database should be set as in step 17.
Once the table identifier is known, the verification can be done using the  SQL command:


\vspace{0.3cm}
\noindent
\texttt{select distinct {\it reference\_file\_column} from {\it instrument\_best\_ref\_table}  where {\it prefix\_expstart\_field}  >= "{\it USEAFTER\_date}"}
\vspace{0.3cm}

\noindent
where for the above example {\it reference\_file\_column}  is ``acr\_best\_pfltfile'' and {\it instrument\_best\_ref\_table}  is ``acs\_ref\_data''
The field {\it prefix\_expstart\_field}  is the table column name with the information of the useafter date. In the case of ACS , {\it prefix\_expstart\_field}  should be replaced by ``acr\_expstart'', while for STIS it is
``ssr\_texpstrt''  (see Appendix A for a complete list of all the parameters). Finally,  ``{\it USEAFTER\_date}'' is the useafter date reference file header keyword and after which the reference file has to be used; e.g., ``MAR 05 2005 08:44:17''. Note that we are using uppercase letters and specifying the hour, minute and second after which the file should be used. If uppercase does not work, use the same format as the file that was delivered.
An example of the command to check the ACS darks recommended after the useafter date ``Mar 05 2005 08:44:17'' is:

\vspace{0.3cm}
\noindent
\texttt{1>select distinct acr\_best\_darkfile from acs\_ref\_data where acr\_expstart >= "MAR 05 2005 08:44:17"\\
2> go\\
acr\_best\_darkfile  \\
 ------------------ \\
 NULL               \\
 P3V2228OJ\_DRK.FITS \\
 P3V2228PJ\_DRK.FITS \\
 P3V2228QJ\_DRK.FITS
}
\vspace{0.3cm}

This can be done by querying for the best reference files of any one kind in the {\it \bf dadsops} database in the ZEPPO server. This command should list only the reference files that are active. Those that have been superseded by the current delivery should not appear in the list.  If any of the old reference files appears (this means that there was a problem with the script that updates the *\_ref\_data tables), contact Mike Swam so he can re-run the cron job that updates this table. If possible, look for some examples of data have the erroneous reference file. For the latter you can use the StarView  forms that list the best reference files; in those forms search for the old reference file in the corresponding field. Note also that in the above output there is a  {\it``NULL''} value. When datasets are ingested in the archive, the best reference values are all set to {\it ``NULL''}. This value is automatically changed later to the appropriate reference file value when the nightly cron job, that updates the best reference files tables, runs.

\ssubsectionstar{19. Send notification}

Forward the acknowledgment e-mail from OPUS mentioned in step 15 to the deliverer along with a copy of the CDBS installation report (cdbs\_report). This will serve as a confirmation that the files are in the system. Copy the ``opus\_*'' files created by \verb|sendit| to the testing directory and compress the files.  Once the ``opus\_*" files are copied to the testing directory, delete all the files from the delivery directory, ie deliverfiles, deliverfiles2, deliverfiles3, etc, so that the next deliverer can find an empty directory in which to deliver their files.

\ssubsectionstar{20. Fill out delivery information on WIKI}

Once the delivery is finished, the deliverer should report the completed delivery on the CDBS WIKI page.  The main WIKI page for CDBS is located at 
http://www.stsci.edu/wiki/INS\-CDBS/CDBSGroupNotes .  From the main page, go to the link 'Status Reference File Deliveries'.  On this page, you will find a table with the following headers; Date, Instrument, Delivery Number, Number of files delivered, Type of files delivered, and Name of Deliverer.  Click on the edit button to add your current information.  When the edit screen shows up, copy the format (html tags) of the previous line and insert your information into the html formatting.  When you are done, you can Preview or Save the information.  

This Wiki page will help all the members of the CDBS team keep informed as to how many and what types of deliveries are being made.



\bigskip

%\acknowledgements
%\noindent {\bf Acknowledgements} 

%\noindent I thank ........

\ssectionstar{References}

\noindent C. Cox, \& C. Tullos TIR OSG-CAL-97-02 (updated 7/1/98)

\noindent R. Diaz-Miller TIR CDBS 2005-02

%\noindent Pasquali, A., Pirzkal, N., \& Walsh, J.\ R. STECF ISR
%ACS 2003-07.

\newpage
\begin{center}
{\bf Appendix A}
\end{center}

\begin{center}
{\bf ACS}
\end{center}
%\vspace{-1.0cm}

\begin{center}
{\bf Table A1: acs\_ref\_data table reference useful keywords}
\end{center}
\vspace{-1.2cm}
\noindent
\begin{table}[htbp]
  \centering
  \begin{tabular}{ll} % column spec
    \hline
  \hline
   Column\_name & comment \\  
\hline
\hline
acr\_aperture & Aperture Name\\
acr\_ccdamp & CCD Amplifier Readout Configuration\\
acr\_ccdchip & CCD chip\\
acr\_ccdgain & Commanded gain of CCD\\
acr\_crsplit & number of cosmic ray split exposures\\
acr\_detector & Detector\\
acr\_expstart &  UT date of start of observation (MMM DD YYYY hh:mm:ss)\\
acr\_filter1 &  element selected from filter wheel 1\\
acr\_filter2 & element selected from filter wheel 2\\
acr\_flashcur &  Post flash current: OFF, LOW, MED, HIGH\\
acr\_fwoffset &  computed filter wheel offset\\
acr\_fwerror &  filter wheel position error flag:  F or T\\
acr\_obstype &  Observation type - imaging or spectroscopic\\
acr\_proposid &  PEP proposal identifier\\
acr\_shutrpos &  Shutter position: A or B\\
acr\_sclamp &  lamp status, NONE or name of lamp which is on\\ 
\hline
\end{tabular}
\end{table}

\begin{center}
{\bf Table A2: acs\_ref\_data table reference file identifier}
\end{center}
\vspace{-1.2cm}
\noindent
\begin{table}[htbp]
  \centering
    \begin{tabular}{llc} % column spec
    \hline
  \hline
  Column\_name & Reference file type & Reference file extension \\
    \hline
\hline
acr\_best\_biasfile  &  BIAS IMAGE  &  BIA\\
acr\_best\_cfltfile &  CORONAGRAPHIC SPOT  FLAT  IMAGE & CFL\\
acr\_best\_darkfile &  DARK  IMAGE  & DRK\\
acr\_best\_dfltfile &  DELTA  FLAT IMAGE & DFL\\
acr\_best\_dgeofile &  GEOMETRIC DELTA IMAGE (DISTORTION) & DXY\\
acr\_best\_flshfile &  POST  FLASH IMAGE &  FLS\\
acr\_best\_lfltfile &  LOW ORDER FLAT IMAGE &  LFL\\
acr\_best\_pfltfile &  PIXEL TO PIXEL FLAT  FIELD IMAGE &  PFL\\
acr\_best\_shadfile &  SHUTTER SHADING IMAGE &  SHD\\
acr\_best\_atodtab  &  ANALOG-TO-DIGITAL TABLE  &  A2D\\
acr\_best\_bpixtab &  BAD PIXEL TABLE & BPX\\
acr\_best\_ccdtab &  CCD PARAMETERS TABLE & CCD\\
\hline
  \end{tabular}
\end{table}
\newpage

\begin{center}
{\bf Table A2: acs\_ref\_data table reference file identifier (cont)}
\end{center}
\vspace{-1.0cm}
\noindent
\begin{table}[htbp]
  \centering
    \begin{tabular}{llc} % column spec
    \hline
  \hline
  Column\_name & Reference file type & Reference file extension \\
    \hline
\hline
acr\_best\_comptab &  THE HST MASTER COMPONENT TABLE  & TMC\\
acr\_best\_crrejtab &  COSMIC RAY REJECTION PARAMETER TABLE & CRR\\
acr\_best\_graphtab &  THE HST GRAPH TABLE  & TMG\\
acr\_best\_idctab &  IMAGE DISTORTION COEFFICIENTS TABLE & IDC\\
acr\_best\_mdriztab &  MULTIDRIZZLE PARAMETER TABLE & MDZ\\
acr\_best\_mlintab &  MAMA LINEARITY TABLE  & LIN\\
acr\_best\_oscntab &  CCD OVERSCAN REGION TABLE & OSC\\
acr\_best\_phottab &  PHOTOMETRY and THROUGHPUT TABLE & PHT\\
acr\_best\_spottab &  SPOT POSITION TABLE &  CSP\\
\hline
  \end{tabular}
\end{table}

\begin{center}
{\bf STIS}
\end{center}

\begin{center}
{\bf Table A3: stis\_ref\_data table reference useful keywords}
\end{center}
\vspace{-1.3cm}
\noindent
\begin{table}[htbp]
  \centering
  \begin{tabular}{ll} % column spec
    \hline
  \hline
   Column\_name & comment \\  
\hline
\hline
ssr\_aperture &  Aperture name\\
ssr\_binaxis1 & axis1 data bin size in unbinned detector pixels\\
ssr\_binaxis2 & axis2 data bin size in unbinned detector pixels\\
ssr\_ccdamp & CCD Amplifier\\
ssr\_ccdgain & CCD commanded Gain\\
ssr\_ccdoffst & Commanded bias offset of CCD\\
ssr\_cenwave & Central wavelength in Angstroms\\
ssr\_crsplit & Number of CR split exposures\\
ssr\_detector & Detector\\
ssr\_lampset & spectral cal lamp current value (milliamps) \\
ssr\_obstype & Observation Type (Imaging or Spectroscopic)\\
ssr\_opt\_elem & Optical Element used for observation\\
ssr\_texpstrt & UT time of the start of exposure (MMM DD YYY hh:mm:ss)\\
ssr\_wavecal & wavecal image file name  \\
\hline
\end{tabular}
\end{table}
\newpage 

\begin{center}
{\bf Table A4: stis\_ref\_data table reference file identifier}
\end{center}
\vspace{-1.0cm}
\noindent
\begin{table}[htbp]
  \centering
    \begin{tabular}{llc} % column spec
    \hline
  \hline
  Column\_name & Reference file type & Reference file extension \\
    \hline
\hline
ssr\_best\_biasfile & Bias image file &  BIA  \\
ssr\_best\_darkfile & Dark image file  & DRK  \\
ssr\_best\_pfltfile & Pixel-to-pixel flat file &  PFL\\
ssr\_best\_dfltfile & Delta flat image file &  DFL\\
ssr\_best\_lfltfile & Low-order flat image file  & LFL\\
ssr\_best\_shadfile & Shutter shading correction image file  & SSC  \\
ssr\_best\_sdstfile & Small scale distortion image file  & SSD \\
ssr\_best\_atodtab & A2D Correction Table  & A2D \\
ssr\_best\_apdstab & Aperture Description Table  &  APD\\
ssr\_best\_apertab & Aperture Throughput Table  &  APT  \\
ssr\_best\_bpixtab & Bad Pixel Table  & BPX\\
ssr\_best\_ccdtab & CCD Parameters Table  & CCD\\
ssr\_best\_crrejtab & Cosmic Ray Rejection Parameters Table & CRR\\
ssr\_best\_disptab & Dispersion Coefficients Table  &  DSP\\
ssr\_best\_inangtab & Incidence Angle Correction Table & IAC \\
ssr\_best\_idctab & Image Distortion Correction Table  & IDC  \\
ssr\_best\_mlintab & MAMA Linearity Table  &  LIN  \\
ssr\_best\_lamptab & Calibration Lamp Table  & LMP  \\
ssr\_best\_mofftab & MAMA Offset Correction Table & MOC  \\
ssr\_best\_pctab & Photometric Correction Table  & PCT  \\
ssr\_best\_phottab & Photometric Conversion Table  & PHOT  \\
ssr\_best\_sdctab & 2-D Spectrum Distortion Correction  & SDC  \\
ssr\_best\_cdstab & Cross-Disperser Scattering Table & CDS\\
ssr\_best\_echsctab & Echelle Scattering Table & ECH\\
ssr\_best\_ecstab &  Echelle Cross-Dispersion Scattering Table & EXS\\
ssr\_best\_halotab &  Detectore Halo table & HAL\\
ssr\_best\_riptab &  Echelle Ripple Table & RIP\\
ssr\_best\_srwtab &  Scattering reference Wavelenght Table & SRW\\
ssr\_best\_psftab &  Telescope Point Spread Function Table & TEL \\
ssr\_best\_tdctab & NUV Dark Correction Table  & TDC\\
\hline
  \end{tabular}
\end{table}

\newpage
\begin{center}
{\bf Table A4: stis\_ref\_data table reference file identifier (cont)}
\end{center}
\vspace{-1.2cm}
\noindent
\begin{table}[htbp]
  \centering
    \begin{tabular}{llc} % column spec
    \hline
  \hline
  Column\_name & Reference file type & Reference file extension \\
    \hline
\hline
ssr\_best\_tdstab & Time Dependent Sensitivity Table & TDS  \\
ssr\_best\_wcptab & Wavecal Parameters Table  & WCP  \\
ssr\_best\_sptrctab & 1-D Spectrum Trace Table &  1DT  \\
ssr\_best\_xtractab & 1-D Extraction Parameters Table  & 1DX \\
\hline
  \end{tabular}
\end{table}

\begin{center}
{\bf WFPC2}
\end{center}

\begin{center}
{\bf Table A5: wfpc2\_ref\_data table reference file identifier}
\end{center}
\vspace{-1.0cm}
\noindent
\begin{table}[htbp]
  \centering
    \begin{tabular}{llc} % column spec
    \hline
  \hline
  Column\_name & Reference file type & Reference file extension \\
  & & (type)\\
    \hline
\hline
w2r\_best\_atodfile&Analog to Digital Converter Lookup Table&R1?\\
w2r\_best\_biasfile&Bias Frame&R2?\\
w2r\_best\_darkfile&Dark Frame&R3?\\
w2r\_best\_flatfile&Flat Field File&R4?\\
w2r\_best\_maskfile&Static Mask File&R0?\\
w2r\_best\_shadfile&Shutter Shading Correction& R5? \\
w2r\_best\_comptab&Master Component Table&TMC.FITS\\
w2r\_best\_graphtab&The Master Graph Table (SYNPHOT)&TMG.FITS\\
w2r\_best\_idctab&Image Distortion Coefficients File&IDC.FITS\\
w2r\_best\_offtab&not used& -\\
\hline
  \end{tabular}
\end{table}
\newpage
\begin{center}
{\bf Table A6: wfpc2\_ref\_data table reference useful keywords}
\end{center}
\vspace{-1.3cm}
\noindent
\begin{table}[htbp]
\centering
\begin{tabular}{llc} % column spec
\hline
\hline
Column\_name & comment & type \\  
\hline
\hline
w2r\_obset\_id &observation set id &-\\
w2r\_obsnumA&observation number &base 36 \\
w2r\_atodgain & A-D  Gain & electrons\\
w2r\_equinox &equinox of celestial coord. system &-\\
w2r\_expstart & Exposure start time &Modified Julian Date \\
w2r\_filter1 &First filter Numberf& -\\
w2r\_filter2 &Second filter Number& -\\
w2r\_filtnam1 &First filter Name& -\\
w2r\_filtnam2 &Secondfilter Name& -\\
w2r\_mode &instrument mode& FULL (full res.), AREA (area int.) \\
w2r\_orientat\_1,2,3,4& Orientation of the image 1, 3, or 4&posangle\\
w2r\_serials& serial clocks&ON, OFF\\
w2r\_shutter &Shutter in place at beginning of the exposure&-\\
w2r\_atodcorr &A-D correction applied &PERFORM, OMIT,COMPLETE\\
w2r\_biascorr &Bias correction applied&PERFORM, OMIT,COMPLETE\\
w2r\_blevcorr &Bias level correction applied&PERFORM, OMIT,COMPLETE\\
w2r\_darkcorr&Dark correction applied&PERFORM, OMIT,COMPLETE\\
w2r\_dophotom &Fill Photometry keywords&PERFORM, OMIT,COMPLETE\\
w2r\_flatcorr &Flat correction applied&PERFORM, OMIT,COMPLETE\\
w2r\_maskcorr &Mask correction applied&PERFORM, OMIT,COMPLETE\\
w2r\_shadcorr &Shaded Shutter correction applied&PERFORM, OMIT,COMPLETE\\
\hline
  \end{tabular}
\end{table}
\newpage
\begin{center}
{\bf NICMOS}
\end{center}
\begin{center}
{\bf Table A7: nicmos\_ref\_data table reference file identifier}
\end{center}
\vspace{-1.0cm}
\noindent
\begin{table}[ht]
  \centering
    \begin{tabular}{llc} % column spec
    \hline
  \hline
  Column\_name & Reference file type & Reference file extension \\
  & & (type)\\
    \hline
\hline
nsr\_best\_darkfile& Dark Current File&DRK\\
nsr\_best\_flatfile& Flat Field&FLT\\
nsr\_best\_illmfile&Illumniation Patern File&ILM\\
nsr\_best\_maskfile&On-Orbit MASK for NCS data\\
nsr\_best\_nlinfile&Deterctor Linearity File&LIN\\
nsr\_best\_noisfile&Detector Read-Noise File&NOI\\
nsr\_best\_saadfile&Post SAA Dark&Assoc. Name\\
nsr\_best\_tempfile&temperature-dependent dark reference file&TDD\\
nsr\_best\_backtab&Background Model Table&-\\
nsr\_best\_phottab& Phototmetric Calibration Talbe&PHT\\
nsr\_best\_idctab&Image Distortion Coefficents File&IDC\\
\hline
  \end{tabular}
\end{table}

\begin{center}
{\bf Table A8: nicmos\_ref\_data table reference useful keywords}
\end{center}
\vspace{-1.3cm}
\noindent
\begin{table}[htbp]
\centering
\begin{tabular}{llc} % column spec
\hline
\hline
Column\_name & comment & type \\  
\hline
\hline
nsr\_obset\_id& Observation Set ID&-\\
nsr\_camera&Camera in use &1, 2, or 3\\
nsr\_expstart& Exposure Start Timei&MJD\\
nsr\_filter& Filter Wheel Element &varchar\\
nsr\_nread&Number of Initial and Final Readouts&small int\\
nsr\_readout&Detector readout rate & FAST, SLOW\\
nsr\_samp\_seq&Number of Samples & int\\
\hline
  \end{tabular}
\end{table}

\newpage
\begin{center}
{\bf Appendix B}
\end{center}

From time to time we receive deliveries of new HST Calibration Spectra files or updates to the libraries of stellar models that SYNPHOT uses. Given that these files are used by all the instrument teams, there is no instrument in charge of the assessment and delivery of these files. Therefore, the INS/CDBS Team has taken the lead  to support these activities, making sure that the files are CDBS compliant and tested by all the instrument teams. The bulk of the work on these files is on the testing. Although, it is not the duty of the INS/CDBS Team to do this work, it is important that the team makes sure that these files are tested with SYNPHOT (and if possible with the ETCs) and that the appropriate documentation describing the reason for the update appears in the header of the file.
  
Once the test has been completed by the instrument teams, the files can be copied to their respective directories in the CDBS area of the centralized storage. There is no need to deliver these files to the CDBS database; however, there are a series of steps that should be followed to make sure these files have the appropriate documentation and are in the appropriate directories.

First of all, there are two important steps that should always be followed when updating these files
\begin{enumerate}
\item The history of the new file should clearly state the reason for the update.
\item Since these files are used by all the instrument teams, they should sign-off the delivery of these files.
We should request that instrument teams test these files; however, they might choose to sign-off without further
testing. In any case, when they are aware of the changes and the possible implications the delivery of these files might
have in the SYNPHOT or ETC calculations, it should be enough for us to make the delivery. Even in this case, a formal signoff, via e-mail should be required and kept in our records.
\end{enumerate}

\begin{center}
{\bf HST Standard Stars Spectra}
\end{center}

In the case of HST Calibration Spectra, there are two independent directories containing the spectra of standard stars: CALOBS  (/grp/hst/cdbs/calobs) and CALSPEC (/grp/hst/cdbs/calspec). CALOBS contains original as well as updated versions of the ultraviolet (IUE and VOYAGER2) and optical (Oke, Tapia or Stone) spectra of standard stars, while CALSPEC contains composite ultraviolet and optical absolute calibrated reference spectra of the HST standards. 

Although these files are not delivered to the CDBS database, a set of several steps has to be followed when copying these to the centralized storage. The CALSPEC files have to be copied not only to the CALSPEC directory mentioned above but also to another directory called CURRENT CALSPEC (/grp/hst/cdbs/current\_calspec/). This second directory can be found in the CDBS area in the centralized storage and has the same files that are in the CALSPEC directory. These two directories exist now to support SYNPHOT and ETC. In this case the directory CURRENT CALSPEC supports the ETCs while CALSPEC supports SYNPHOT. However, due to the current policies and procedures created by the HST Mission Office and the ETC Team for the ingest of all the SYNPHOT data files in the ETC servers, the CURRENT CALSPEC directory might become obsolete soon, except maybe for testing purposes. The need for this directory will have to be evaluated at a later time. In the mean time we still have to make the copy of these files in these two directories. Note that there is another directory called SUPPLEMENTAL CALSPEC ( /grp/hst/cdbs/supplemental\_calspec/) which contains a subset of HST calibration spectra. The files in these directories are different to those in the CALSPEC area and  these are for calibration spectra of less accuracy than those in CALSPEC.

There are other things to consider about these files. The new files will follow the same version control as any other SYNPHOT file; that is, the version number increases by one for the new file being delivered.  These are delivered by R. Bohlin, who is the person working on the HST standard spectra. He also helps us with the update of the CSBS Web page information for the CALSPEC files. Therefore, an update to the CALSPEC web page should be made every time a new HST Standard star is delivered.

\begin{center}
{\bf SITS FUV-MAMA Dark Current Glow Image Files}
\end{center}

The STIS FUV-MAMA dark current glow images files are in the CDBS area (/grp/hst/cdbs/stis\_aux/) but are not delivered to the  CDBS database. These files are auxiliary FUV MAMA dark files provided for users  for five epochs between April 1997 and August 2004. For each of these epochs there is a dark mean dark rate including glow normalized to counts/pixel/second, a dark for hot pixels ($>1e-4 c/s$) plus base level rate measured in dark corner and a dark with glow only (i.e.  Mean dark minus hot pixels minus dark corner average).  If STIS were to decide to update these files in the future, these can just be copied to the holding directory. We should, however, request that instrument team test these files against CALSTIS before delivery.

\begin{center}
{\bf Grid of Spectral Atlases}
\end{center}

CDBS stellar spectral atlases are a collection of theoretical models or composites of observational data that are well known and used by the astronomical community. These are used to model the spectra, to derive the photometry, or to determine the counts  for a particular type of astronomical  object. There is a considerable number of files of this kind, each grouped by the atlas they belong to. These files are rarely updated, so deliveries of this kind might not be seen in a long time. It is more likely that a new grid of models have to be added than that the models need to be changed. In the case a new grid needs to be added, a detailed description of the models have to be included in these directories as a readme file together with the set of models. This README file has to be added to the CDBS web site too. Currently we have 12 atlases in the CDBS GRID directory:

\begin{enumerate}
\item CASTELLI-KURUCZ ATLAS. It contains about 4300 stellar atmosphere models.
\item PICKLES ATLAS. This library of wide spectral coverage, consists of 131 flux calibrated stellar spectra.
\item BUSER-KURUCZ ATLAS. The catalog consists of 1434 files.
\item KURUCZ 1993 ATLAS contains about 7600 stellar atmosphere models.
\item BRUZUAL ATLAS contains 77 stellar spectra. 
\item GUNN-STRYKER ATLAS consists of 175 spectra of stars. 
\item BRUZUAL-PERSSON-GUNN-STRYKER ATLAS contains 175 spectra.
\item JACOBY-HUNTER-CHRISTIAN ATLAS contains 161 spectra of stars. 
\item BRUZUAL-CHARLOT ATLAS is a library of 84 galaxy spectra.
\item KINNEY-CALZETTI ATLAS consists of an homogeneous set of 12 spectral templates of galaxies 
\item AGN ATLAS consists of 5 spectral templates of AGNs ranging from LINER to Seyfert and bright QSO.
\item GALACTIC ATLAS consists of model spectra of the Orion Nebula and of the NGC 7009 planetary Nebula.
\end{enumerate}	

\end{document} 
