The ORAC Project: Overview and Description

Alan Bridger, Frossie Economou, Gillian Wright and Nick Rees

orac001-wpd, version 01 Created:5 September 1997, modified: 5 September 1997

This paper introduces and describes the UKIRT ORAC project, providing an overview of the project, including a conceptual outline, identifying the expected deliverables and an estimated timescale.

1.0 Introduction

In April 1997 Nick Rees visited ROE and with Alan Bridger described a new UKIRT software project - the Observatory Control System and Data Reduction project - which would be contracted to the ROE. In May this project was approved by the UKIRT board and was set in motion. In July 1997 the project was named ORAC, for Observatory Reduction and Acquisition Control.

This document provides an overview of the project and will also serve as a project work package description.

Section 2.0 gives a brief description of the project, Section 3.0 outlines the expected deliverables and Section 4.0 clarifies various responsibilities. Section 5.0 sets out an initial project plan and manpower estimates, and finally Section 6.0 sets out some project tool standards.

2.0 Project Description

2.1 Aims and Goals

In summary, the stated aim of the project is to 'To increase the observing efficiency and publication rate of UKIRT'. In order to achieve this goal the approach is based around making it easier, friendlier and more efficient to observe at UKIRT: by providing an easy to use observation preparation system; by providing an efficient, powerful and yet flexible on-line data reduction system; and by further automating the acquisition of data at the telescope. A long-term goal is to produce a system that will be capable of fully queue-scheduled observing, should UKIRT decide to implement it.

We realise that it is difficult to quantify the extent to which we fulfil the stated aims. Although changes in the publication rate can be measured, these can be influenced by many factors unrelated to the scope of this project. Observing efficiency is difficult to measure and there are no real metrics for it (at UKIRT) at the moment. Setting up such measures is probably not worthwhile in this context. In general it is the feedback from the users of the system, particularly visiting astronomers, but also staff astronomers that will be the (somewhat subjective) measure of the success of the system. However, some general goals can be set:

2.2 Scope

In order to achieve the aims, all of the software that currently interacts with the observer at UKIRT will be replaced with a more modern, user friendly, UNIX-based system. Thus the project has three main components: a new Observation preparation system; a new data reduction system; and a new overall observatory control system. To facilitate understanding these may be compared with the existing UKIRT control system, though each of these systems will have considerably enhanced functionality. The Observation Preparation system is a direct replacement for the present UKIRT_PREP system, though it is intended to be fully portable and to allow remote preparation. The data reduction system will be a replacement for the existing cgs4dr system, with greater flexibility and extensibility to support all current and planned future instruments. The observatory control system supplants the functionality of the current AIM task that deals with EXECs and configs, but is intended to go further in automating the control of the systems and the acquisition of data. It will be the generic interface between the observer and the specific instrumentation.

It will be noted that although the storage and archiving of data is not specific to an instrument it is none the less not included in the scope of the ORAC project. The data storage facility is required by the Michelle project during their laboratory commissioning (likely to be early 1998) then it remains within the scope of that project, though there will clearly be overlap with the ORAC work and both projects will work to ensure that the data storage and quick-look facilities are generic. Archiving is the responsibility of UKIRT operations, where a suitable system can more easily be implemented.

2.3 Initial System Design

Figure 1 shows a conceptual design for ORAC. Some of the thoughts behind these ideas can be found in the paper [X-1] by Alan Bridger and Gillian Wright presented at the New Observing Modes for the Next Century conference in 1995. The basic ideas of the design are explained further below.

2.3.1 Overall Concept

The design is based around the idea that an observation may be described as consisting of an instrument configuration, which tells the instrument how it should be setup for an observation, a target description, which tells the telescope where to point, and an observing sequence, which describes how telescope movements and data acquisition commands are sequenced to collect the full observation. Taken together these components form an Observation Definition. The components are modular and may be used in multiple Observation Definitions. Thus Target Description BS623 may be used with Instrument Configuration Std_H_Band and Sequence Bright_Star in one observation, and later combined with My_K_Band and Std_Jitter to form a different observation.

To this basic definition of an observation are added further components which describe the data reduction Recipe to be used on the observation and Scheduling Information which may be used in a more sophisticated observation scheduler.

These components, and the full Observation Definitions may be prepared ahead of time (off-line) or at the telescope (on-line) with the preparation system, and then at the telescope the Observatory Control System will read them and distribute the components to the various subsystems, to configure and control the acquisition and the reduction of the data.

2.3.2 Observation Preparation System.

The observation preparation system is a replacement for the existing UKIRT_PREP system. Observers will be able to run it from their home institution without being required to get involved in detailed software installations. The system may also need to change frequently (because of changing instrument configurations). Both of these suggest a web-based application, which avoids the need for software distributions. However, speed of the web, and the loading on the server might be problems. Use of mirror sites could help with this, but the use of automatic software installations and internet push technology to keep them up to date might be a better solution. This is the approach taken by the Gemini group with their Observing Tool.

The output of the preparation system will be one or many files forming an Observation Definition that can be stored along with similar definitions from the same or different observing programmes to form the telescope's observation 'database'. The format of the files and the database is yet to be determined, but initial versions will probably use the existing combination of ASCII text files and a directory tree structure, for backwards compatibility. Gemini have chosen SGML and Sybase to perform the equivalent function and the ORAC preparation system might move in similar directions.

It is likely that much of the observation verification will be done by this system - an observation generated by the preparation system should be guaranteed to be able to be performed by the instrument. This will eliminate many errors at the telescope. Additionally, the preparation system will present the user with an astronomer-oriented interface, but its output may well be at a lower level. It is anticipated that the system's understanding of the instruments will be codified by a set of rules which will be maintained by the instrument support scientist. This will add to the system robustness.

2.3.3 Observatory Control System.

The Observatory Control System is a system that performs the sequencing role between the instruments and telescopes - a role that at UKIRT was traditionally performed by communication between the telescope operator and the observer. The use of the EXEC system at UKIRT has partially automated this, sequencing the telescope and instrument according to predefined EXECs, but the new system will go much further, and may also allow higher level scheduling of the Observation Definitions.

The way the components of the Observation Definitions are to be handled is still to be determined, but a likely approach is to leave the reading and parsing of a component to the system that requires it - e.g. an Instrument Configuration will be read and interpreted by the Instrument Control System. This means that the higher level systems will only pass names to the lower level systems, considerably simplifying them. Other approaches are possible and will be considered.

The Observatory Control System consists of two main components:

  1. The Scheduler. This has no equivalent in the current system and is a higher level system that maintains a queue of Observation Definitions (possibly with definitions from different observing programmes) prioritised according to their scheduling information. Its goal is to intelligently and automatically handle the correct and efficient scheduling of the observations, informing the observation sequencer of the next observation. Additionally, it will provide for monitoring and direct manipulation of the observing queue. The scheduler is not required to run an instrument and can be delivered in an additional phase after the rest of ORAC is in place (and in particular after the delivery of Michelle).
  2. The Observation Sequencer. This task is broadly equivalent to the EXEC handling portion of the AIM task in the current system. It informs instruments of their required configuration and sequences a set of operations to generate a particular observation. This can include taking Object and Sky sequences, mosaicing, nodding and chopping etc. It will define a standard, but very general, instrument interface to which all UKIRT instruments will need to adhere. The observation scheduler is required for instrument operation and so must be provided at the time of Michelle delivery.

It should be noted that the observation sequencer will not immediately be able to control CGS4, IRCAM3 and UFTI, but it should be built such that retrofitting them to the system will be a viable option. However, this retrofit is not considered to be part of the ORAC project.

2.3.4 Data Reduction System

This performs automatic on-line data reduction after the raw data frames are stored. The data reduction strawman is based on the concepts of recipes and scripts:

The user will request a particular reduction sequence to be performed by drawing from a series of observatory defined scripts and recipes. Advanced users will be able to provide their own recipes and scripts to reduce particular aspects of their observations, but it is expected that they will all use standard recipes to remove specific instrument dependencies.

It is anticipated that the scripts will be coded in a standard scripting language, and hoped that the actual data reduction algorithms will be provided by standard data reduction packages. If new or modified algorithms are required then it is hoped that these can be fed back into the standard packages. The aim of this is to reduce the support effort at the Observatory, concentrating it on the automatic aspects of the reduction system. As a side effect all of the actual algorithms used by the system should already be available to the community, in addition to which users may be provided with the actual scripts used to reduce their data.

FIGURE 1. Conceptual Design of the ORAC System

3.0 Project Deliverables

These deliverables are grouped according to the three major project components outlined above. This is not intended to be used as a formally complete list, but as a guide to what should be produced by the project.

The Observation Preparation System

  1. The observation preparation system software and full source code.
  2. Full support for CGS4, IRCAM3 and UFTI. (Michelle added by the Michelle Project).
  3. Backwards compatibility with the existing CGS4 and IRCAM3 Configs and EXECs.
  4. A defined interface for extending the Observation Preparation system to support new instruments.
  5. A maintenance guide for the system (for scientific support staff).
  6. A user guide for the system (paper, electronic and on-line versions).
  7. A programmer's guide for the system.

The Observatory Control System

  1. The sequencer software and full source code.
  2. Full sequencer support for Michelle and future instrumentation, with hooks to support CGS4, IRCAM3 and UFTI if it is decided to retrofit them.
  3. The scheduler software and full source code.
  4. A defined interface for new instrumentation projects to write to so that the new instrument may be controlled by the sequencer.
  5. A maintenance guide for the system (for scientific support staff).
  6. A user guide for the system (paper, electronic and on-line versions).
  7. A programmer's guide for the system.

The Data Reduction System

  1. The data reduction pipeline software and full source code.
  2. Initial recipes and scripts to support CGS4, IRCAM3, UFTI and Michelle data reduction.
  3. A maintenance guide for the system (for scientific support staff).
  4. A user guide for the system (paper, electronic and on-line versions).
  5. A programmer's guide for the system.

4.0 Responsibilities

As the ORAC system interacts strongly with all components of the overall UKIRT control system and both with current instrumentation and new instruments it is appropriate to say something about the boundaries of the system and the responsibilities for some components.

In broad terms the ORAC is responsible for systems that are common to all instruments and for overall control of the observing process. Clearly observation preparation, observation scheduling and sequencing and data reduction fall under this definition, but so does data storage and archiving, and the "quick-look" display, which are not covered by ORAC. The reasons for this are largely practical, relating to the progress of the Michelle project and location of personnel. In order to clarify the situation this is a list of where responsibility lies for the major systems of the UKIRT Control system:

  1. Telescope control system (TCS): UKIRT Operations. The ORAC sequencer must be able to control the TCS, but it will fit to the existing TCS interface. If there is any need for changes to this interface then they will be agreed with UKIRT operations and the change to the TCS performed by operations.
  2. Data archiving: This is entirely the responsibility of UKIRT Operations.
  3. The instrument control system (ICS): This will be the responsibility of each individual instrument project. However, an interface between the ORAC sequencer and each ICS must be defined. It is preferable that this interface be as generic as possible. The responsibility for defining this interface will be shared between ORAC and the Michelle project, with appropriate backwards compatibility for existing instrumentation (IRCAM3, CGS4 and UFTI).
  4. Data Storage and Quick-look display: These will be the responsibility of the Michelle project, though the interfaces with ORAC must be defined in conjunction with the ORAC project, and the products must be suitably generic.
  5. The UKIRT Data Reduction system: This is the responsibility of the ORAC project. The ORAC project will define an interface and procedure for including new data reduction scripts and recipes. The definition and implementation of these scripts and recipes (including any new algorithms) will be the responsibility of instrument projects. (Exception: the recipes and scripts for IRCAM3, CGS4 and UFTI will be within the scope of ORAC.)
  6. The ORAC sequencer: This will be the responsibility of the ORAC project, but its design and implementation will be shared with the Michelle project, and its development timescale driven by the Michelle project.
  7. The ORAC Scheduler: The ORAC project.
  8. The Observation Preparation System: This is the responsibility of the ORAC project. The project will also define an interface and procedure for including new instrumentation in the system and the production of the components for new instruments will be the responsibility of each instrument project. The ORAC project will, however, produce the components for IRCAM3, CGS4 and UFTI, and share the production of the Michelle components with the Michelle project.

5.0 Project Plan

The project will be managed by ROE, with JAC oversight being provided by a series of review meetings staged at roughly 6 month intervals. The definition of "Critical Review" is interpreted broadly by the ORAC project, and these review points are more of a guideline as to roughly where the project should be. The approximate milestones are:

October 1997:

Critical design review for observation preparation system and data reduction system. Preliminary design review for observatory control system.

April 1998:

Delivery of the observation preparation system, and some basic imaging capability of the data reduction system for use on UFTI. Critical design review for the sequencer.

September 1998:

Delivery of the Michelle instrument and the sequencer, preliminary review of the observation scheduler.

February 1999

Critical review of the observation scheduler.

September 1999:

Delivery of the final observatory control system.

More detailed project plans are available from the project.

6.0 Project Standards

The following documentation, design and management standards have been adopted by the ORAC project:

7.0 References

The reference list for the ORAC project may be found in:

  1. orac000-gar, "ORAC: Glossary and Reference List", Alan Bridger, Royal Observatory Edinburgh.