ADASS XII Conference

Database systems

P5.1 Optimizing the Performance of ISO and XMM Data Archives

Jose Hernandez, Christophe Arviset, John Dowson, Pedro Osuna, Aurele Venet (ESA)

The ISO Data Archive and the XMM-Newton Science Archive have been developed in Java using a multi-tier architecture, the archives are accessed through a Java Applet running inside any standard browser, they have been used from more than 10 different platforms and thousands of users.

Here we present the experience acquainted in the last 4 years and a few techniques we have used to dramatically improve the performance of the application, as well as the main difficulties we have found along the way. In particular will touch upon the following points:

Techniques used to optimize the performance of the application:

Session management, Objects caching
Minimize number of client-server transactions
Data compression, optimize the object Serialization
How to maintain the interface configurable
When do we stop optimizing: Problems/Bugs introduced during the optimization

Techniques used to optimize the size of the applet:

obfuscation
building a thin client
jar loading on demand

P5.2 ANDES--NOAO's Observatory Database System

David Gasson, Dave Bell, Mia Hartman (NOAO)

ANDES (the Advanced NOAO Database Expert System) is NOAO's new observatory database system. Recent improvements include the phase-out of legacy components, such as our previous Access-based effort called ALPS++. New work focuses on post-TAC procedures such as scheduling, collection of observing reports on the mountain, automatic compilation of various statistics, and publication tracking as ways to extend the usefulness of ANDES. The ultimate goal is to provide an environment which allows a comprehensive understanding of the collection, evaluation, scheduling, observing and post-observing (including publications) of proposals and programs.

P5.3 Chandra Data Archive Download and Usage Database

Emily Blecksmith, Stephane Paltani, Arnold Rots, Sherry Winkelman (CfA/CXC)

In order to support regular operations, the Chandra Data Archive Operations Group has developed a database that records and monitors the user activities that affect the archive servers. This database provides information on the number of users that are connected at a given time, what archive interfaces they use (we have several), and how much data are being downloaded.

The database consists of three tables, populated by a set of three scripts that parse the archive server logs, the ftp logs and the login logs. User activity can be tracked through each of those logs, making information from a given connection easily accessible.

With this tool, the Archive Group will be able to gather statistics and monitor trends, which in the future will improve the accessibility of Chandra data.

This work is supported by NASA contract NAS 8-39073 (CXC).

P5.4 A Simple Rule Based Query Service for Querying Complex Databases

Niall Gaffney, Lisa Gardner, Molly Brandt (STScI)

Moving towards a Virtual Observatory model for archive services has demonstrated that more generic query services will be needed. For many advanced query services that exist today, one must understand the structure of data source to query it - often going as far as making the user hand craft SQL. From the beginning of its JAVA incarnation, StarView has separated the task of getting qualification information from the user from formatting of the SQL. We have recently replaced a very cumbersome and Space Telescope database structure specific LISP program with one based on three simple database tables and a Perl script to build queries for our database. StarView (or any other program) need only submit the equivalent of the SQL SELECT and the qualifiable portion of the WHERE clauses (with no database specific join information). The service takes this information and, for our SQL driven database, adds in all the needed SELECT, FROM, and complete WHERE clauses that is a valid query. With this tool, any program can query the STScI archive database without prior knowledge of how the database tables relate. Only the fields/keywords that are to be queried and returned are specified. Further as new tables and databases are added, the service can be simply updated with the join rules for the new tables without any changes to the software system.

Our poster will outline how the system works in the general case using the STScI databases as an example of a preexisting and complex data source can be simply modeled. We will demonstrate how this system could work for any relational database to be queried by software without knowing its schema and explore possible functionality for such a generic query service in the context of the currently rapidly evolving Virtual Observatory.

P5.5 Performing Dynamic Overlaps Between Astronomical Archives using HTM

Gretchen Greene, Alex Szalay, Antonio Volpicelli, Karen Levay, Paolo Padovani, George Fekete, Wil O'Mullane

An experimental system recently developed provides a method for dynamic computation of overlapping regions between astronomical archives using the Hierarchical Triangulated Mesh (HTM) spatial index. The test bed for this system is based on the SDSS Science Archive and the HST Pointing Catalog. Region boundaries described with HTM polygons provide a common frame for performing joins between the large-scale SDSS survey and the sparse coverage of HST. We expect to provide Web services for performing these intersections to facilitate the navigation of these multi-instrument observations. These methods are valuable in demonstrating the value of HTM standard region specifications for the space-time definitions of astronomical datasets that can be used for NVO discovery technology.

P5.6 An AIPS++-based Archive System for Radio Telescopes

John Benson, Boyd Waters, Tim Cornwell (NRAO)

The NRAO End-to-End (e2e) project has the goal of providing automated, streamlined handling of radio observations on NRAO telescopes all the way from proposal submission to archive access. As an early part of this project, we have constructed an archive system for NRAO telescopes using mainly tools available in the Astronomical Information Processing System (AIPS++). Since the tools are available to anyone using AIPS++, this amounts to a generic archive capability for any telescope for which the AIPS++ data conversion program exists. The rich tool set available in AIPS++ has enabled very rapid development: our entire effort took no more than about 1 FTE-year. Additional capabilities were required to connect AIPS++ to the web. The system is now being deployed at the NRAO as a prototype archive system for the Very Large Array with deployment for the Green Bank Telescope and Very Long Baseline Array planned for 2003.

P5.7 The XMM-Newton SSC Database: Taking Advantage of a Full Object Data Model.

Laurent Michel, Christian Motch (Observatoire Astronomique de Strasbourg) Clive Page, Mike Watson (Leicester University)

One of the main responsibilities of the Science Survey Consortium (SSC) of the XMM-Newton satellite is to carry out a systematic analysis of all instrument data. These high quality products are shipped to the guest observer and eventually enter the XMM-Newton archive. In addition, the SSC compiles a catalogue of X-ray sources and provides identification for the ~ 50,000 new sources detected every year. In order to check product quality and support the catalogue and source identification programmes, all SSC generated products are stored into a data base developed for that purpose.

The database is a powerful tool to browse and evaluate XMM-Newton data and to perform various kinds of scientific analysis. It provides online data views including relevant links between products and correlated entries extracted from many archival catalogues available at CDS and NED.

Because of the large number of transversal links, our data model is difficult to map into relational tables. It has been therefore designed with object oriented technology for both user's interface and data repository (OODBMS O2).

Besides browsing, the Web based user interface provides facilities to select data collections with any constraints on any keywords but also with constraints on correlated data patterns.

P5.8 Representations of DEIMOS Data Structures in FITS

Steven L. Allen, De A. Clarke, Robert I. Kibrick (UCO/Lick Observatory)

DEIMOS (the DEep Imaging Multi-Object Spectrograph) began producing scientific data from the Keck II telescope in 2002 June. The instrument is extremely configurable, and the form of the output data is highly variable. Filters and gratings may be swapped, gratings and mirrors tilt, readout modes and active amplifiers of the 8-CCD mosaic change, and numerous field-specific astrometric slitmasks may be inserted. For archival purposes and to enable fully-automated data reduction, FITS files from DEIMOS document the instrument state, all aspects of the slitmask design, and multiple world coordinate systems for the mosaic images. The FITS files are compatible with existing local conventions for mosaic image display systems and also with incipient FITS WCS standards.