The WFCAM science Archive must provide rapid and straightforward access to WFCAM data from both the UKIDSS surveys and generalised open-time usage. The WFCAM science archive should:
Top-level requirements:
T1: provide the maximum possible potential for capitalizing on the UKIDSS surveys.
Rationale: UKIDSS amounts to 80% of WFCAM time and rapid exploitation is crucial to the UK's future strategy. Since exploitation is through the science archive, the need for T1 is clear.
T2: contain data from both UKIDSS surveys and open-time usage
Rationale: WFCAM data are likely to be far larger than the average UK University site can hold. Standard pipeline products held centrally are the solution.
T3: be flexible and copes with alterations in the UKIDSS survey design over time
Rationale: UKIDSS has an initial two-year allocation followed by review. This review may result in changes in both priority and design of the surveys. It is imperative that the science archive should not be a blockage in this process.
T4: not preclude usage from the GRID and inclusion in Virtual Observatory
Rationale: obvious, and clearly relates to T1
T5: allow simple and complex queries with appropriate interfaces for both
Rationale: a single complex interface suitable for data federation would be excessive when all the user wants is a K frame around a given target.
T6: be simple to use for PR purposes
Rationale: UKIDSS is a truly world-beating undertaking. The science archive will be the obvious point from which to draw PR material and should not be a hindrance to this.
T7: allow access to survey data before all observations are complete, and must not be disrupted by regular ingest of new survey data.
Rationale: This is consistent with rapid exploitation (and was a PDR requirement). Access to interim releases of the data will clearly be needed.
T8: allow arithmetic operations and options from the advanced processing toolkit on pixel data to be specified in requests
Rationale: multicolour imaging is perfectly possible given standard pipeline output and should not require the data to be transferred to the users home site just to stitch three colours together.
T9: Must be scalable to VISTA data volumes
Rationale: WFCAM and VISTA represent the UK's deep infrared survey capability. To use the same science archive design for both will ensure maximum exploitation of both.
T10: Must be able to merge reduced frames taken in non-photometric conditions with other data from the same survey
Science archive contents and functions (minimum)
C1: Contains calibrated object catalogues resulting from the pipeline, for both UKIDSS and open-time observations
Rationale: obvious, basic science archive function.
C2: Ingests and stores pipeline output frames for later online processing, generates compressed pixel images on the fly for rapid web-based access, carries out immediate cross-referencing with existing UKIDSS survey data and produces consolidated UKIDSS catalogue in a given field
Rationale: again basic science archive functionality. Compression by at least a factor of order 10 is necessary in the early stages, whether or not the need disappears later due to enhanced network speeds. It is assumed (but not required) that this will be lossy compression.
C3: Is able to recalibrate a given field or fields in the event of revised calibration information (specifically, photometric and astrometric), and allow database queries on the recalibrated quantities
Rationale: changes in calibration information are frequently encountered in survey operations, and the science archive itself may lead to such changes.
C4: Is able to cross-calibrate photometric information using areas of overlap between processed frames, where available.
Rationale: this is not a sensible function of the CASU pipeline, which is required only to produce results on a night-by-night basis. The science archive will have all photometric information and calibrations for all superframes, and is where this should happen.
C5: Allows public access to subsets of survey data on a variety of different search criteria (specified below)
Rationale: basic science archive functionality.
C6: Allows rapid on-line cross-referencing of search results with other catalogues
Rationale: consistent with T1, this requirement is expanded on later.
C7: Allows for generation of finder charts via a web form
Rationale: simple to provide and useful when observing at a site remote from the UK
C8: Holds housekeeping information for all archived data
Detailed requirements
Consistent with the above, the Science archive:
D1: must allow searching individual (or all) UKIDSS surveys on at least
the following criteria (or combination of them):
Positional rectangle RA, Declination (J2000 coordinates)
Positional rectangle Galactic Coordinates (
)
Circular sky patch within a specified radius from a given RA, Dec
Circular sky patch within a specified radius from a given Galactic Coordinate
Circular sky patch within a specified radius of a resolvable source name
Source colour index (e.g. YJHK magnitudes and any linear combination
thereof) between any of the available survey colours for the particular survey
Source parameter ranges (using any of the available source parameters)
Rationale: These are all basic science archive functions
D2: must allow searching within open-time programme data on the same criteria (where possible), returning whatever data are available (may only be image frames)
Rationale: open-time data become UKIRT science archive data as normal, and in many instances will be as searchable as data from the UKIDSS survey.
D3: must allow similar queries to be repeated for all objects in a given user-supplied source catalogue, or those in any catalogue also stored locally to the UKIDSS science archive.
Rationale: the user may have their own catalogue generated at a different wavelength, and may wish to obtain UKIDSS catalogue or image data at this list of field centres.
D4: must allow combinations of queries on a number of different source catalogues
Rationale: this requirement anticipates compound queries such as ``return all UKIDSS image data around infrared sources with axial ratios greater than 1.5 and which have no detection in the SDSS''. An arbitrary number of source catalogues should be allowed, subject only to their local availability.
D5: must allow functions [methods] to be used in setting up complex queries (i.e. for colour index if not stored separately)
Rationale: for consistency with T9.
D6: must have a web-based user interface to the object catalogue database implemented in an SX-like manner, and pixel database access implemented with in-house software. Ideally a simple interface for very quick searching on a given object name or position would be provided separately.
Rationale: clearly a web interface is needed, at least in the short term, and the SX similarity requirement follows from the desire to maximise the power of matching with Sloan for the LAS. For very quick returns of UKIDSS data around a given object, either the basic web interface must be straightforward enough to make this very quick, or a separate form should be provided.
D7: must allow plotting of returned parameters direct from the user interface, in selected pairs (x-y plots) or histograms (in the case of a single parameter)
D8: must return pixel images, confidence maps and catalogue data (where reduction recipes generate it) to the user in (at least) gzipped FITS format
D9: must be able to return pixel data in any available passband, over a contiguous field up to [TBD but one array across seems a sensible number], together with matched object catalogue
D10: must be able to generate (on-the-fly) and return larger areas from survey data, blocked down as required
D11: must be able to generate and return stacked images using a variety of stacking algorithms, where survey data are available
D12: must be able to generate and return merged multi-colour multi-parameter catalogues with best available astrometric and photometric calibration from survey data
D13: must support database federation with other source catalogues (initially SDSS and Schmidt sky survey)
D14: must be able to generate and return meaningful optical/IR colours for all objects in the overlap with SDSS, whether or not detected in the Sloan data.
D15: must support ANDing of a given query with another, where both have already been executed
D16: must support the returning of only a subset of the entire possible array of object parameters
D17: must allow trial-and-error searches (e.g. return number of source hits rather than entire catalogue), for any valid query
D18: must allow repetition of queries using previous versions of astrometric and photometric calibration
D19: must be able to produce a finder chart for any region within which survey data exist, returning both object shapes and positions (ellipse map) and a colour pixel image if suvey data in more than one colour are available.
D20: must allow access to best or duplicate data for objects in overlapping survey data (e.g. to generate light curves and/or proper motions from duplicate observations of the same object).
D21: must allow general access to all housekeeping data - e.g. for a given survey area, what is currently available, how good it is, etc.
D22: must specify calibrated quantities through coefficient values and calibration scheme
D23: must allow a summary of data available to be generated for a given search region
Security
A1: Archived data must be accessible only by validated users
A2: Archived data must be uncorruptable by science archive users and managers
A3: Allows data protection on the basis of proprietary date (per frame)
A4: Science archive must be quickly recoverable in the event of corruption by hardware/software faults etc.
Rationale: Clear need to insure against data loss (e.g. backup on removable media and/or 100% redundant storage with data striping)