orac022-rev3 Version: 01 Created: 15 March 1999 Modified: 18 March 1999
Comments made at the review of the proposed ORAC scheduler design held at the JAC on 1st March 1999
This document presents a brief summary of the points that were raised at the ORAC Scheduler review meeting on March 1 1999. The document is intended to be read by anyone with an interest in the ORAC project, although knowledge of the documents that were reviewed and of the overall ORAC design is required.
Alan Bridger, Dennis Kelly, Gillian Wright (UK-ATC), Nick Rees, Frossie Economou, Malcolm Currie, Andy Adamson, John Davies (JAC - UKIRT), Remo Tilanus, Richard Prestage (JAC-JCMT), Kim Gillies, Shane Walker (Gemini Project).
Alan Bridger, Dennis Kelly, Gillian Wright (UK-ATC), Nick Rees, Frossie Economou, Malcolm Currie, Andy Adamson, John Davies (JAC - UKIRT), Remo Tilanus, Richard Prestage (JAC-JCMT), Kim Gillies, Shane Walker (Gemini Project), Tim Hawarden, Ian Robson (JAC), Colin Cunningham, Min Tan, Adrian Russell, Malcolm Stewart (UK-ATC).
The meeting was held to review the following documents: orac020-schr: UKIRT Scheduler requirements, orac021-schp: UKIRT Scheduler Preliminary design and orac019-odbc: Observation Database:Concept. The documents describe the potential second stage of ORAC which is to add tools that will provide the functionality necessary for flexibly scheduled observing at UKIRT, and provide an estimate of the required effort. Whether or not UKIRT will implement queue-scheduled observing is still being studied. The review of these UKIRT-specific documents was held immediately prior to a two day meeting that was organised to examine in some detail areas of commonality between UKIRT, JCMT and Gemini plans and software requirements.
Overall the review went well and the proposed designs, with comments taken into account, "passed". However, the issue of funding for ORAC Stage-2 and the detailed remit of the project, such as how/whether it is combined with the JCMT Observation Management and OCS upgrades projects, is still to be decided.
The rest of this document summarises the comments made on each of the review papers in the order in which they were reviewed.
The question of whether there was a real requirement/desire for classical observers to use a scheduling tool was raised. It was agreed that this would be an extremely useful tool for observers, especially if it was easy to use. This of course means that the scheduling tools must be available for distribution.
There was extensive discussion of whether or not there was really a requirement to be able to adjust the weights and parameters used by the Scheduler to calculate a night's plan. It was eventually realised that this was due to the description of the circumstances under which this would be done and by whom being unclear. It was agreed that there is indeed a requirement for UKIRT management to change these, e.g. if after a semester of operation it was found that the "rules" agreed with the TAG did not produce the desired results in terms of programme completion, the weighting factor for this would need to be changed. However, it was also agreed that in "normal use" to draw up a plan for a night of observing the weightings given to such parameters would not be varied.
It was agreed that long-term scheduling tools which could be available at the TAG meetings would be extremely valuable, and it was noted again that this implies that the electronic database of Phase-1 proposals be available also - as a copy or via network access.
There was a long discussion of for "how far ahead", a detailed (nightly plan level) look at the likely schedule was required. It was agreed that a timescale of "about a week" would be appropriate for UKIRT - this would allow discussion at the weekly scheduling meetings of the likely programmes that could be done under different weather conditions. It was suggested that an efficient way to achieve this would be to allow the input to the Scheduling tool to be a "time-line" of predicted weather for the coming week (rather than a staff astronomer doing a few nights at a time in the "simulation" mode described in the documents). There could similarly be a time-line input for instrument availability. The user-interface for this simulation mode also needs to be able to highlight any high priority programmes that have not been scheduled for the coming week.
It was pointed out that as well as saving the night's plan and the parameters used to generate it, we needed to be sure that by the end of a nights observing sufficient information had been saved that questions about why a particular observation had NOT been scheduled could be answered easily. Possibly generate a nightly log of unscheduled observations with the criteria that caused them not to be scheduled - exact requirements here need some thought.
As well as distributing the Scheduling tool for classical observers to use, an additional requirement is for flexibly scheduled observers to use the tool to look at the schedulability of their observations.
The overall design and the issues relating to the choice of Scheduler Engine were presented. Most of the discussion centred around the detailed behaviour of various scheduling engines and the group's current understanding of their state of development. It was agreed that the plan in the document for selection of an appropriate engine was the right way to proceed and that in a few months time the options may be clearer in any case.
There was detailed discussion of the merits of adopting the HST "SPIKE" software as the scheduling engine. The ESO VLT have decided to use this. Remo reported that it was the intention of the HST to enable SPIKE to be used by other observatories as a "black box", and that they were willing to add additional "policies" to describe rules and parameters that other observatories wanted to use. It was however pointed out that, even with this assistance from the HST team, ESO had hired a fully time employee who was an expert in scheduling systems to work on their scheduler. It is apparently complicated to format the input for the Scheduler, but once this is done the functionality will more than meet the requirements.
There was also a lot of discussion of the use of "dispatch" schedulers, with some extensions to their functionality. In many ways it was thought that such a scheduler would come closest to meeting the requirements with the advantage of simplicity and transparency in how they work. It was noted that this is in essence how the Ames scheduler, which has been tested on UKIRT Service nights, works: it pre-sorts the programmes to select the ones for the required weather and instruments and then passes the result to a dispatch scheduler. The Gemini proto-type scheduler also implemented a sort followed by dispatch scheduling algorithm, and Phil Puxley's simulations showed that averaged over a semester it did well at ensuring high priority programmes were done, provided there were sufficient "backup" programmes to overload the schedule. Given the Liverpool Telescope plans in this area the option of an extended dispatch scheduler should be re-examined later.
There was a little discussion of the long term scheduler, since most of the questions had concerned the requirements and had already been discussed. The general flexibility between using science programmes and observing plans and blocks of time was liked.
It was noted that in the option for the observer to "re-calculate" the nights plan (perhaps because something had changed) had to mean an option for the observer to recalculate the nights plan starting from "soon", with the observer specifying when "soon" is, so that an observer can ask for a new queue that's suitable for starting after the current observation is finished.
It was noted that the JCMT methods for drawing up a queue were not so automated as this - they select in real time objects from one of a few much larger "queues" (i.e. the queues contain more observations than could be done in a night), rather than attempting to draw up a time ordered plan for a night. Partly this is in response to rapidly changing conditions, and often very short observation times. This difference in philosophy was discussed further at the workshop.
Discussion of the effort estimates included in this document was deferred until the end of the meeting.
The discussion began with identifying the fact that the requirements for authentication of users were not yet clear. The issue of whether UKIRT would issue authentication to just the PI or to co-I's or allow the PI to pass the information on was discussed and the general principle that there would be only one authentication per programme was agreed. It was also pointed out that UKIRT may have similar issues to JCMT, where there is a requirement to ensure that visiting observers carrying out flexibly scheduled runs must not be able to see/do their main competitors targets/plans. The JCMT solution is to flag such potential conflicts of interest and flexibly schedule those runs with only very low priority ones. Such issues may well arise at UKIRT, as like JCMT the flexibly scheduled nights may have to be carried out by observers. The UKIRT ops and ORAC teams will discuss further the details of how they wish to handle security issues as part of the process of working out the operations implications of flexible scheduling at UKIRT.
The other principal subject of discussion were the relative merits of relational and object-oriented databases for storing the Science programmes, etc. It was agreed that as set out in the document the decision on which to use should await further investigation of whether the hierarchy of the Science programmes can be stored and returned from Sybase and how complex the queries would really be. Many OO databases are very expensive, prohibiting their use or simple experiments with them. However, Gemini are seriously considering use of "PSE Pro" which costs about $500 and has free client licensing. It was noted that this would be suitable for "small" (in commercial terms) databases - to about 10Mbytes, but this would be more than adequate for the Observation Database.
The requirements for the database should include one on speed - since there is a requirement to add completion information in real time and also to recalculate Observing Plans at the telescope.
It was noted that although the database could be extended to include Phase-1 information, pointers to the data archive, thumbnails of data etc. the requirements for inclusion of these were not yet well developed. UKIRT would consider the archiving requirements as part of the process of examining the operations implications.
Remo pointed out that tools to automatically email the PI when data were obtained and generally assist with closing communication loops, while not really part of a Scheduler or Database would be extremely useful/ more or less essential for flexibly scheduled semesters. It was noted that because the possibility of flexible scheduling on UKIRT is currently in the very early stages of study, the expected processes at UKIRT and requirements from them are not yet fully understood. The JCMT experience in this area would be discussed further at the workshop.
There was some discussion of the estimates of required effort and the timescales on which it would become available. There was general agreement that the effort requirements were reasonable and that the error bars were consistent with the uncertainties in UKIRT's requirements, and the range of options still being explored for use of "other systems". For a UKIRT-only project it was agreed that significant effort was unlikely to be available at either UKIRT or the ATC until after ORAC Stage-1 and Michelle commissioning.
There was some discussion of the options for de-scoping ORAC Stage-2. It was noted that another option was to provide the database and the user interface for browsing it first. Searching the database with queries to find for example "all incomplete Michelle projects with an RA between 10 and 15 hours and requiring good 20mm conditions" is in effect a first stage towards drawing up a queue for a nights observing, and would enable a semi-manual mode of flexible observing to be carried out. Such database searches are similar to the pre-sort stage of a more sophisticated scheduler. It was agreed that providing the database was therefore a valid de-scope option for a UKIRT Scheduler.
No formal action list resulted from the review because this stage of the project is not yet funded. It was noted that the next stages were for UKIRT Board approval for flexible scheduling of UKIRT to be sought, and to explore the options for joint software development with JCMT and Gemini in the "workshop" following the review meeting. Nonetheless when the project restarts the comments from this review will be fed back into it.