FITS formatting of SuperCOSMOS data

 

written by:-

N.C. Hambly

August 1998

(Last modified 19th August 1998)

 

SuperCOSMOS data files are very large, and are generally supplied in compact, unformatted binary files for speedy access and efficient storage (see the Beginner's Guide for more details). This data format is non-standard, and internal to the SuperCOSMOS project. Users may wish to port their data into any one of a large number of software packages that manipulate images and/or catalogues (eg. IRAF , GAIA , KAPPA , CURSA , FTOOLS ) and the Flexible Image Transport System (FITS) allows easy export of data once they are in the required FITS format. This document describes two utilities for reformatting pixel and/or image parameter data into FITS images and/or tables. These utilities adhere strictly to the FITS standard via William Pence's FITSIO routines and are written in standard Fortran 77. The code is available for you to ftp to your home institution to reformat any unformatted SuperCOSMOS data.

Retrieving and compiling the code

To retrieve the FITS reformatting code, use the indicated URL above or anonymous ftp as follows:
% ftp ftp.roe.ac.uk
username: anonymous
password: your email address
ftp> binary
ftp> cd pub/nch/code
ftp> get fits.tar.gz
ftp> bye
then uncompress and unpack using the following:
% gunzip fits.tar
% tar -xvf fits.tar
Users at sites where the STARLINK Software Collection is installed can then compile the code as follows:
% f77 iam2fits.f -L/star/lib `fitsio_link` `sla_link` -o iam2fits
%  "  pix   "         "            "            "        pix
Non-Starlink users can obtain the source code for the FITSIO and SLALIB positional astronomy libraries from those indicated URLs. Any comments/suggestions/queries/bug reports concerning the codes should be sent to N.Hambly (nch):
roe-dot-ac-dot-uk.

Running the codes

If you are reformatting data using the codes, be aware that for simplicity of use the routines assume the data are held in the standard filenames after a path-name (normally) indicating the plate measured, eg.
% ls /data1/nch/R12345
iam.srtrd
hkfile.lis
residual.corrns
mmdata
.
.
.
%
where the directory "mmdata" contains the individual lanes of pixel data (see the Beginner's and Introductory data processing guides for more details). If files have been renamed or moved, then they should be restored to their names and relative directories as detailed above (alternatively, you could alter the code to prompt for these filenames). Files supplied on EXABYTE or DAT tapes should unpack from the tape archive as above.

Reformatting Image Analysis Mode data

IAM data are reformatted using iam2fits. You may choose to reformat the entire data set (which will result in a new file around 5% larger than the unformatted binary file iam.srtrd) or you can specify a square region around a specific position of interest resulting in a subset of the whole which may be more convenient to handle for users with disk space limitations. Run the code as follows:
% ./iam2fits
Path-name of measurement:
/disk1/nch/UKR444
Size of square region to extract:
(arcmin; 0.0 to convert entire IAM file to FITS format)
0.0
Output fits file:
/disk1/nch/UKR444/iam.fits
No. of IAM records written to FITS file:      598523
%
Note that the code will always write a minimal FITS header detailing the columns of the output FITS table; in addition, an attempt will be made to write "housekeeping" information from the file hkfile.lis. This is not required by the FITS standard, and is not essential to further manipulation of the data; however it may be useful at some later stage in data reduction. If the required ASCII housekeeping file is not found, then a warning is issued but the routine continues execution.

Examples of FITS table IAM data access

Once your IAM data are in FITS table format, you can browse or manipulate them in a variety of packages. One particularly useful suite is CURSA which allows browsing, pairing, subsetting and many other features with a web-based, GUI-driven utility allowing user-friendly data interfacing with other catalogue data via the internet. Another method of manipulating FITS tables is available within the IRAF environment in the package tables. Read your FITS data in using tables.fitsio.strfits, then manipulate using the tasks in tables.ttools. The usual online help facility within IRAF describes the usage of the various routines.

Reformatting Mapping Mode data

MM data can be extracted in up to 4096x4096 pixel subsets about an input co-ordinate centre using pix2fits. The size limitation is machine and user-dependent: if you get errors as detailed in the source code comments then you may need to get your user quotas and/or machine swap space increased on your machine; alternatively reduce the size of the large array dimension in the source code before compilation. Run the code as follows:
% ./pix2fits
Path-name of measurement:
/disk1/nch/UKR444
Extent of scan (xmin, xmax, ymin, ymax):
16.22 338.78 16.22 338.78
Centre co-ordinates of region to be extracted:
13 34 59.04 -27 30 15.05
Equinox of co-ordinates (0=1900, 1=B1950, 2=J2000):
2
Size of square image to extract (arcmin):
5.0
Output pixel values in raw transmission (t) or intensity (i)?
t
Output FITS filename:
obj1.fits
Number of records read from IAM file:     598523
No. of IAM records written to FITS file:        375
%
Once again, a minimal (required) FITS header is written, but housekeeping information will also be written if available (note that the extent of the measurement is normally as detailed above, but can be obtained from the housekeeping file). Two other useful features of the code are as follows: World Co-ordinate System (WCS) FITS keywords will be written to the header based on the astrometric solution used in the IAM file. This is particularly useful if displaying the image using utilities such as SAOimage TNG or GAIA (see below) which then give a continuous display of the cursor position in celestial co-ordinates (note that old versions of SAOimage written before the Greisen and Calabretta FITS WCS proposal will not necessarily read and/or use all the WCS keywords correctly). The other useful feature is that the IAM data corresponding to the region specified (or a 5x5 arcminute region, whichever is the larger) will be written as a FITS table in an extension after the image. Moreover, a binary index file of pointers to those images is also created, which can be used as detailed below. Note that pix2fits provides an option to output pixel values in raw "transmission" units or in intensity units (for more information concerning photometry from photographic plates, see the Photometry data processing guide ). Be aware that the intensity scaling is somewhat arbitrary - a "gamma" is assumed by the code (and can be changed in the source code if you wish to expermient); in any case, for various reasons the dynamic range of the image is limited when compared to, for example, CCD data.

Example of the use of FITS image MM data

The image display utility GAIA (based on ESO's utility skycat ) is an extremely powerful tool for browsing and analysing image and catalogue data in a user-friendly GUI-based utility. Once you have a subset of MM data in FITS format, you can display the data, overplot celestial co-ordinate axes and overplot catalogued objects to investigate optical counterparts to spaced-based detections, for example. If you wish to overplot the IAM catalogued objects corresponding to the pixel data, then a "tab" formatted list file of the subset of images must first be written. The code gaialist.f does just that using the index pointer file created by pix2fits (note that GAIA does not currently read FITS table extensions in FITS files as written by iam2fits and pix2fits). This code is available via ftp at the address given above (simply get gaialist.f after issuing the command ascii when in ftp). The "tab" formatted file can then be interrogated as a local catalogue option in GAIA. The cursor can then be used to point at images and the corresponding image parameters will be highlighted in the catalogue list: this is a very powerful browsing utility.