Current status at WFAU

WFAU has a proven track record in survey astronomy. We have been routinely handling survey data at rates of several tens of Gbytes per day for some time now, and have much experience in delivering survey products to the international astronomical community - see, for example, details of the SuperCOSMOS Sky Survey (URL given previously). WFAU has formally undertaken science archiving for all WFCAM data (not just UKIDSS data).

During the past 3 yr period, WFAU has been instigating R&D projects and establishing collaborative links to address large database problems. At the basic level, we have gained significant experience in the production, curation and user support of the SSS archive (which is $\sim2$ Tbyte in volume). With an eye on more advanced database technology and the future requirements for large astronomical databases in the UK we have been following the development of the archive system for the multi-terabyte SDSS (see http://www.sdss.org/). This has produced a solution to the generic large astronomical database problem, and our proposed solution to the WSA development problem is based on this since we believe the SDSS system is unlikely to be bettered using available resources. WFAU has established links with the Johns Hopkins University group responsible for these developments, and has established a UK mirror for the SDSS-EDR. We are undertaking implementation of the SSS archive in SQLServer, and are collaborating with Jim Gray of Microsoft Research in the ingestion of this terabyte dataset to study the scaling issues. We intend to build on this, and our experience in creating the SSS and other similar products, to design and implement the WSA.

On the hardware side, we have employed monolithic RAID technology as a solution to mass storage and fast random access for the SSS. We have established an 8-node Beowulf cluster for our SDSS-EDR mirror, and have also acquired a 12-node rack-mounted Beowulf for experimentation with parallelisation techniques (both Beowulfs were funded through University and JREI sources). We have begun an analysis of the hardware requirements for the WFCAM science archive as part of an end-to-end data flow analysis and have made preliminary approaches to vendors, e.g. Eclipse Computing (providers of bespoke hardware solutions like our monolithic RAID system and Beowulf clusters) and Sun Microsystems. Finally, the Blackford Hill site is now connected to SuperJanet via two 1 Gbit s$^{-1}$ network links as part of a SRIF upgrade award.

Funding for WFAU is mainly via PPARC Rolling Grant. We are currently half way through the 4 year award period 2000-2004, and have applied for funding for the overlapping period 2002-2006. WSA development forms a major part of the work for this period (the relevant section of the RG application, on which this document is based in part, is online at http://www.roe.ac.uk/~nch/wfcam).