Package invocations :: Package cu13 :: Module cu13 :: Class Cu13
[hide private]

Class Cu13

source code


Creates deep stacks and confidence images from intermediate stacks derived from MSB observations. The deep stacks are described in the RequiredStack table and have a UID comprising the programmeID and productID. The deep stacks are produced using the CASU fitsio_stack code. Various options allow the operator to choose which stacks to produce and how many intermediate stacks to use. The database release number (DR#) defines the directory where the mosaic will be found.

Nested Classes [hide private]
  Pointing
Pointing(index, ra, dec, filterID, dualID)
  ProductDetails
ProductDetails(productID, multiframeID, imageName, confName, catName)
    Nested Errors and Exceptions

Inherited from wsatools.DbConnect.CuSession.CuSession: CuError

Instance Methods [hide private]
 
_onRun(self)
Create products.
source code
 
getSoftVersion(self) source code
 
arePotDodgyDeeps(self)
Simple test - anything before Aug 2012 is redone.
source code
 
getProdMfID(self)
Get dictionary of productMfIDs
source code
 
areMalformedProducts(self)
Check to see if their are problems with OBs in wrong order in existing products
source code
 
getExpectedOrder(self, filterTable)
For each tile work out best OB order
source code
 
getOBList(self, productID, pwpProdID, pwpList)
Splits any deeps into components
source code
 
getTiles(self, pawprintComps)
Find OB tiles made from these components
source code
 
testProv(self, pawprintProducts, tileProdID, badOrder)
Get list of OBs that went into pawprint
source code
 
setComponentOrder(self, productID, components)
Creates an order for the stacking.
source code
 
createProducts(self, newProducts, productOfID, componentsOfProd)
Creates the products in the given list.
source code
 
calcProcessLayers(self, processType, products, productOfID, componentsOfProducts)
Organise products into processing layers.
source code
 
hasProcessBeenRun(self, products, productOfID, processType) source code
 
organiseProducts(self, memoryPerProduct, noProcessesPerProduct) source code
 
getImageList(self, compOfProducts) source code
 
compressProd(self, bestProduct)
Compressed all necessary files associated with the product.
source code
 
fixBadExtensions(self, bestProduct, zeroExtNums) source code
 
createCatalogue(self, bestProduct, components, dualImageProduct=None, isQuick=False)
Extracts the catalogue for the given product, using the extraction tool for the programme.
source code
list(int)
createStacks(self, newProducts, productOfID, componentsOfProd)
Creates the deep stack of given product ID in the output directory when supplied a list of component images (from queries.getStacks()).
source code
 
getOBLevel(self, allFiles, bestProduct) source code
 
runInitialCasuExtraction(self, newProducts, productOfID, componentsOfProd, stackInfoDict=None) source code
 
createTiles(self, newProducts, productOfID, componentsOfProd) source code
 
finishOffProducts(self, newProducts, productOfID, componentsOfProd)
Last parts.
source code
 
createMosaics(self, newProducts, productOfID, componentsOfProd)
Create mosaics
source code
 
fitWCS(self, newProducts, productOfID, componentsOfProd, inParallel=False, areZeroSizeExtsDict={})
Use wcsfit to do astrometry
source code
 
testFits(self, fileName, processName) source code
 
productProcessing(self, productIDList, productOfID, process=None, initialise=False, isComplete=False) source code
 
runThreads(self, threads)
Executes given list of thread processes.
source code
 
tidyUp(self) source code
 
createSoftLinks(self, bestProduct, tmpGroutDir) source code
list(str)
createMosaic(self, bestProduct, components)
Creates the deep mosaic of given product ID in the output directory when supplied a list of component images (from queries.getStacks()).
source code
 
modifyExternalMosaic(self, bestProduct, components, provInfoObj)
Sets up attributes to convert an external mosaic into the correct format: MEF, with the correct headers for CU3 to ingest into the science archive.
source code
 
convertMosaic(self, mosaicObj, outputFileName, isImage, provObj, normalFactor)
Converts a mosaic to MEF and adds the correct primary and extension header.
source code
 
findContents(self)
Checks ProductProcessing to see if there are partially processed products: sets directory date
source code
pyfits.header
getPrimaryHeader(self, bestProd, prihdr, originalHeader, isImage, provObj, addOrig=True)
This produces all the relevant info for the primary header.
source code
pyfits.header
getInitImageHeader(self, imghdr, originalHeader, provObj)
This produces all the relevant astrometric info for the second header.
source code
pyfits.header
getImageHeader(self, bestProd, imghdr, isImage, provObj)
This produces all the relevant info for the second header in mosaic.
source code
 
calcZP(self, bestProd, components, isGrt=False)
Calculate ZP of catalogue.
source code
list(tuple), list(tuple)
getArchivePhotometry(self, mfIDs)
Get archived photometry with latest calibration.
source code
list(tuple)
getCataloguePhotometry(self, components, detectorCounts, fTable)
Runs through list of catalogues that are created, but not ingested and gets the photometry.
source code
list(tuple(int, numpy.array))
getFitsCatData(self, bestProduct, isGrt=False, noMagLim=False, calcOnly=False)
Checks astrometry and gets photometry from a catalogue product.
source code
 
setUpProductsInfo(self)
Gets information for these products...
source code
float
calcDiskSpaceRequired(self, numProducts)
Disk space required for stack, confidence and catalogue.
source code
str
createProductDir(self, spaceRequired)
Creates the directory to store newly created deep stacks.
source code
str
getDualImageData(self, bestProduct)
Creates a Product class for the dual filter image if there is one.
source code
 
stackImages(self, images, bestProduct, expTimes)
Creates a deep stack image from a list of component images using external CASU tools.
source code
 
tileImages(self, inputImages, bestProduct, isFiltered=False) source code
 
filterImages(self, inputImage, filteredFileName) source code
 
checkProductStatus(self, reqProducts, reqInputs)
Determines which products require processing.
source code
 
testProduct(self, bestProduct, components, calcOnly=False)
Tests magnitude limit, seeing, classifications against expectations based on input files.
source code
 
checkQBFlags(self)
Check whether ppErrBitStatus is set correctly for each intermediate stack in list.
source code
 
classifyCat(self, catName, areZeroSizeExts)
Classify cat
source code
 
extractCat(self, bestProduct, areZeroSizeExts)
Run imcore
source code
 
groutCat(self, catName, finalCatName, srcPath)
Grout catalogue
source code
 
wcsfit(self, bestProduct, threadNo, areZeroSizeExts) source code
 
runCasuExtractor(self, bestProduct)
Run CASU Extractor.
source code
float
getRequiredMemory(self, files)
Calculates memory required to process an image batch.
source code

Inherited from wsatools.DbConnect.CuSession.CuSession: __del__, __init__, logBrokenFile, run

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]
int
getUsefulNumCPUs(numCPUs, progTable, stackOnly=False)
Takes the requested number of CPUs and returns the effective useful number of CPUs.
source code

Inherited from wsatools.DbConnect.CuSession.CuSession: logException

Class Variables [hide private]
  cuNum = 13
Curation task number.
  airmass = 1.0
Airmass value for FITS products (must be a float).
  externalZPAccuracy = 0.01
External zeropoint accuracy in mags
  aperture = defaultdict(<function <lambda> at 0x3b26578>, {'UDS...
Default aperture; 3 by default,
  extinct = 0.0
Extinction value for FITS products (must be a float).
  memoryLimit = 66148010.0
Maximum amount of memory that CU13 should consume in kilobytes.
  numObjsBin = 100
?
  largeDetTableLimit = 1000000000.0
DetectionTableLargeSizeLimit
  _autoCommit = True
Should this curation task auto-commit database transactions?
  _isPersistent = True
Should this curation task try to re-open broken database connections?
  _reqDbLock = False
Should only a single instance of this curation task run at a time?
  _useWorkDir = True
Create a temporary work directory.
  casuSoftDir = None
CASU software directory
  filterer = 'nebuliser'
Image filtering program, e.g.
  stacker = 'imstack'
Image stacker program, e.g.
  tiler = 'mosaic'
Image tiler program, e.g.
  bitWiseProcessingDict = {'calcZP': 24, 'classify': 20, 'comple...
  breakIntoParts = False
Stack with intermediate stages?
  dateRange = DateRange(begin=<mx.DateTime.DateTime object for '...
Nights to stack.
  dirDate = '20140908'
Output dir date.
  doQFlagCheck = True
Check that detection quality flags are set?
  extractor = None
Override database choice of catalogue extractor.
  isCompressed = True
Compress deep stack files?
  maxProdID = None
Maximum product ID of stack to be produced.
  minProdID = None
Minimum product ID of stack to be produced.
  notDepOnly = False
Use non-deprecated stacks only
  numCPUs = 64
Number of CPUs to utilise.
  numParts = None
Force stacking to uses this number of stages.
  numStack = None
Number of intermediate stacks to be used.
  redoHeader = False
Don't create stacks just update headers?
  releaseNum = None
Release number of stacks to create.
  reprocessTypes = None
List of product types that need reprocessing.
  restartDir = None
Rerun in this directory
  reqProdIDs = None
List of product IDs for stacks to be created.
  useDeep = False
Use previous stack as part of this stack?
  stackOnly = False
Only create stacks
  deepProductTable = None
A DataFactory.View of the contents of the database table Required{Product} for current programme and product type.
  existingProducts = None
List of details of products already ingested in the database.
  fileListForType = None
Dictionary of paths to the ingest file lists referenced by product type.
  prodInfo = None
The contents of the database table Required{Product} for current programme and product type.
  pType = None
Product type currently being curated.
  reprocess = False
Reprocess existing products of the current type?
  reqProdsOfType = None
Dictionary of lists of product IDs referenced by product type, giving the products required to be created.
  _tmpWorkDir = None
Common temporary working directory for all threads.
  isDetTableTooBig = False
Test whether detection table is too big to get OB cat data from archive.
  newProductsOfType = None
Dictionary of ProductDetails for each type of product successfully created.
  startedProductsOfType = None
Dictionary of ProductDetails for each type of product successfully created.
  prodDirectory = None
Dictionary of directory for each type of product successfully created.
  extractIgnoreMsgs = ['WCSFIT: Matching routine failed -- MATCH...
List of messages to ignore when extracting
  missExtensionErrorMsgs = ['Error running imcore_conf', 'imcore...
List of messages to ignore when extracting if one bad extension
  tileOBDict = None
Dictionary of OB tiles for each tile product...
  pptTileDict = None
Look up for tile mfID based on pawprint mfID
  obNonTiles = None
Look up for OB components without a tile
  prodMfIDDict = None
Lookup for mfID of product if single epoch
  deepProductTableDict = None
Dictionary of product tables - one for each productType

Inherited from wsatools.DbConnect.CuSession.CuSession: archive, comment, cuEventID, curator, eTypes, isDayStampedLog, onlyNonSurveys, onlySurveys, programme, programmeID, resultsFilePathName, shareFileID, sysc

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

_onRun(self)

source code 

Create products.

Overrides: wsatools.DbConnect.CuSession.CuSession._onRun

createStacks(self, newProducts, productOfID, componentsOfProd)

source code 

Creates the deep stack of given product ID in the output directory when supplied a list of component images (from queries.getStacks()). If the number of images being stacked requires more than the available amount of physical system memory then the stacking is broken up into parts, unless overridden by the Cu13.breakIntoParts option.

Returns: list(int)
List of broken extensions.

createMosaic(self, bestProduct, components)

source code 

Creates the deep mosaic of given product ID in the output directory when supplied a list of component images (from queries.getStacks()).

Returns: list(str)
List of new deep product files created.

getPrimaryHeader(self, bestProd, prihdr, originalHeader, isImage, provObj, addOrig=True)

source code 

This produces all the relevant info for the primary header.

Returns: pyfits.header
Primary header for mosaic

getInitImageHeader(self, imghdr, originalHeader, provObj)

source code 

This produces all the relevant astrometric info for the second header.

Returns: pyfits.header
Extension header for mosaic

getImageHeader(self, bestProd, imghdr, isImage, provObj)

source code 

This produces all the relevant info for the second header in mosaic.

Returns: pyfits.header
Extension header for mosaic

getArchivePhotometry(self, mfIDs)

source code 

Get archived photometry with latest calibration.

Returns: list(tuple), list(tuple)
List of good stellar like sources with photometry errors better than limit.

getCataloguePhotometry(self, components, detectorCounts, fTable)

source code 

Runs through list of catalogues that are created, but not ingested and gets the photometry.

Parameters:
  • components (list(ProductDetails)) - Must be an ordered list of ProductDetails for each component.
Returns: list(tuple)
List of photometric values

getFitsCatData(self, bestProduct, isGrt=False, noMagLim=False, calcOnly=False)

source code 

Checks astrometry and gets photometry from a catalogue product.

Returns: list(tuple(int, numpy.array))
List of pairs of extension numbers for detectors and their photometric measurements as an array of source measurements (ra, dec, aperMag, aperMagErr) in order of extension number.

calcDiskSpaceRequired(self, numProducts)

source code 

Disk space required for stack, confidence and catalogue.

Returns: float
Disk space required per deep stack.

createProductDir(self, spaceRequired)

source code 

Creates the directory to store newly created deep stacks.

Parameters:
  • spaceRequired (float) - Amount of disk space required.
Returns: str
Full path to directory where deep stacks will be created.

getDualImageData(self, bestProduct)

source code 

Creates a Product class for the dual filter image if there is one.

Parameters:
  • bestProduct (float) - Amount of disk space required.
Returns: str
Full path to directory where deep stacks will be created.

testProduct(self, bestProduct, components, calcOnly=False)

source code 

Tests magnitude limit, seeing, classifications against expectations based on input files. Doesn't work for products made of previous products. These have already been calculated ...

runCasuExtractor(self, bestProduct)

source code 

Run CASU Extractor.

Parameters:
  • bestProduct (ProductDefinition) - Details of the catalogue to extract.

To Do: Obsolete?

getRequiredMemory(self, files)

source code 

Calculates memory required to process an image batch.

Returns: float
Memory required in kilobytes.

getUsefulNumCPUs(numCPUs, progTable, stackOnly=False)
Static Method

source code 

Takes the requested number of CPUs and returns the effective useful number of CPUs.

Parameters:
  • numCPUs (int) - Requested number of CPUs.
  • progTable (df.ProgrammeTable) - Programme table for current database with the current row set to that of the programme being curated.
Returns: int
Effective useful number of CPUs for this script.

Class Variable Details [hide private]

aperture

Default aperture; 3 by default,

Value:
defaultdict(<function <lambda> at 0x3b26578>, {'UDS': 7})

filterer

Image filtering program, e.g. nebuliser.

Value:
'nebuliser'

stacker

Image stacker program, e.g. fitsio_stack.

Value:
'imstack'

tiler

Image tiler program, e.g. mosaic

Value:
'mosaic'

bitWiseProcessingDict

Value:
{'calcZP': 24,
 'classify': 20,
 'complete': 30,
 'filtmos': 5,
 'grout': 19,
 'imageproc': 6,
 'imcore': 14,
 'imstack': 0,
...

dateRange

Nights to stack.

Value:
DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 00:00:00.\
00' at 318a930>, end=<mx.DateTime.DateTime object for '9999-12-31 00:0\
0:00.00' at 318af60>)

fileListForType

Dictionary of paths to the ingest file lists referenced by product type. This may be preset by the calling task or else defaults will be used.

Value:
None

reqProdsOfType

Dictionary of lists of product IDs referenced by product type, giving the products required to be created. This may be preset by the calling task or else is automatically determined from the database.

Value:
None

extractIgnoreMsgs

List of messages to ignore when extracting

Value:
['WCSFIT: Matching routine failed -- MATCHSTDS: No rows in object cata\
logue',
 'CLASSIFY: missing header info',
 'CLASSIFY: can\'t update header',
 '']

missExtensionErrorMsgs

List of messages to ignore when extracting if one bad extension

Value:
['Error running imcore_conf',
 'imcore_backstats: all values appear to be zero',
 'Error running classify']