The question naturally arises as to how science exploitation of such large datasets will be undertaken. Prior to science exploitation, we require raw data reduction. Data volumes will simply be too large for users to download and keep their own copies. Raw data processing is likely to be complicated, while calibration procedures will evolve as cameras are better characterised and more calibration data are obtained. Reprocessing of substantial amounts of pixel data may be necessary in the light of improved algorithms or for specific `non-standard' science goals. Once data are reduced using standardised pipeline procedures, the establishment of a monolithic science archive offers the greatest potential for full science exploitation. Again, calibration procedures can be more easily developed and applied in a controlled manner to data in a central repository - it makes sense to solve data-specific reduction and calibration problems once, yielding an optimal solution. Early community access to well calibrated data will facilitate timely science exploitation. A well constructed science archive will enhance greatly the scope of research that can be done with the survey data; in fact, certain science applications will only be feasible via a sophisticated science archive. For example, much of the science that will be done with the UKIDSS LAS will rely on complementary data from the SDSS and other non-IR wavelength surveys. Given the volume of all of these datasets, some thought needs to go into the design of the archive to enable full exploitation; with reference to the natural WFCAM and VISTA phased development and the greater scope of the latter, any solution must be clearly scalable to increasing volumes of data.
In outline, what should the science archive provide? As an absolute bare minimum, a simple web interface to pixels and object catalogues along the lines of WFAU's SuperCOSMOS Sky Survey5(SSS) provides community access to the data. However, full science exploitation requires much more. A well-designed archive should:
Clearly, such functionality adds considerable value to the survey data by creating the potential for advanced science exploitation. Furthermore, in the era of `e-science' and `virtual observatories' we envision yet more advanced archive functionality:
In short, the UKIDSS and VISTA science programmes comprise surveys of unprecedented scope with data volumes that are hugely increased over previous experience in the UK. Moreover, user expectation as to the kinds of science that should be possible with these surveys is growing. Archive developments for these major new projects will benefit all UK astronomers over the next few decades and will continue the traditional leading role of the UK in survey astronomy which underpins much of the research done here.
The next sections in this part of the WFAU RG renewal describe how we have positioned ourselves (with our collaborators) to confront these issues and present our plan to solve these challenging problems.