Package wsatools :: Package DbConnect :: Module CommonQueries
[hide private]

Module CommonQueries

source code

Common database queries used by many CUs.


Note: Please put all newly added functions in alphabetical order for ease of maintenance.

Author: N.C. Hambly & I.A. Bond

Organization: WFAU, IfA, University of Edinburgh

Contributors: J. Bryant, R.S. Collins, N.J.G. Cross, E. Sutorius

Classes [hide private]
  ProgrammeNotSetUpError
Exception to denote that programme set up has not been successfully.
Functions [hide private]
 
deprecateOldDetections(db, tableName)
Deprecate old detections from a given programme's detection table.
source code
dict(str: list(int))
findRequiredDeepProducts(programme, minProdID=None, maxProdID=None, reqProdIDs=None, fieldIDs=None)
Find required products of all types
source code
list(str)
getAllDBs(serverName, includeTestDBs=False, archive='ALL')
Retrieves the list of all released databases on a given server.
source code
set(int)
getAllFieldObs(db, ra, dec, dlRaDec, filterID, programmeID, frameType, dateRange=DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 0..., nustep=None, offsetPos=-1, deprecated='deprecated IN (0, 50, 51)', noDeeps=False, onlyTileComps=False, getConfToo=False)
Selects a list of multiframeIDs of given frame type in a certain direction, through a certain filter and in a certain programmeID.
source code
set(int)
getAllFieldObsProd(db, productList, programmeID, frameType, dateRange=DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 0..., deprecated='deprecated IN (0, 50, 51)', noDeeps=False, onlyTileComps=False, getConfToo=False)
Selects a list of multiframeIDs of given frame type in a certain direction, through a certain filter and in a certain programmeID.
source code
str
getBBFilter(db, nbFilterName, programmeID)
Retrieves broad band filter that overlaps the given narrow band filter.
source code
 
getFieldID(db, frameSetID, progTable)
Find fieldID of given frame set.
source code
 
getFrameSetID(db, fieldID, progTable)
Find fieldID of given frame set.
source code
tuple(str, list(str))
getFrameSelection(imageType, noDeeps=False, deepOnly=False, alias='', selType='%stack')
Returns: Frame type selections for image types.
source code
 
getFrameSetGroups(programme, frameSets, otherDB=None)
Group frame sets into those that share the same files.
source code
 
getGroupPositionCentres(programme, otherDB=None)
This function uses the frameSets in MergeLog to derive grouping positions for a survey.
source code
set(int)
getIntStacks(db, progID, reqProdIDs, prodType, dateRange, allNuStep=True, intFrameType='stack', returnAsProdIDDict=False, getConfToo=False)
Retrieves multiframe IDs for the intermediate stacks that will form the given range of deep products over the given time frame.
source code
SelectSQL
getMultiframes(programme, dateRange, fieldIDs=None, deprecated='deprecated IN (0, 50, 51)', prodType='all', useDeeps=True, otherDB=None)
Returns SQL query to select multiframe IDs of images of a particular type from a given programme, with the option of restricting to certain field IDs.
source code
SelectSQL
getFrameSetIDs(programme, fieldIDs=None, otherDB=None)
Returns SQL query to select all frame set IDs of a given programme, with the option of restricting to certain field IDs.
source code
SelectSQL
getSourceIDs(programme, fieldIDs, otherDB=None)
Returns SQL query to select all source IDs of a given programme, with the option of restricting to certain field IDs.
source code
SelectSQL
getSynFrameSetIDs(programme, dateRange, fieldIDs=None, otherDB=None)
Returns SQL query to select all synoptic frame set IDs of a given programme, with the option of restricting to certain field IDs.
source code
 
getProductInfo(db, programmeID, productType, reqProdIDs='')
?
source code
tuple(set, set)
getComponentFrames(db, multiframeID, componentType='normal')
Selects provenence multiframeIDs.
source code
tuple(set, set)
getProvenence(db, deepMfID)
Selects provenence multiframeIDs.
source code
dict(str: bool)
getRedoStatus(db, programmeID, redoProducts)
Works out the redo status, based on the list of products to be redone and their descendants.
source code
list(int)
getRequiredProductIDs(programme, minProdID=None, maxProdID=None, reqProdIDs=None, fieldIDs=None, productType=None, whereStr='', otherDB=None)
Find required products of given type.
source code
set(int) or tuple(set(tuple), set(int))
getStacks(db, programmeID, inputImages, prodID, onlyMfIDs=False, number=None, nonDepOnly=True, productMfID=None)
Queries database for combined stacks for all the given normal images.
source code
Variables [hide private]
  __package__ = 'wsatools.DbConnect'
Function Details [hide private]

deprecateOldDetections(db, tableName)

source code 

Deprecate old detections from a given programme's detection table.

Parameters:
  • db (DbSession) - Connection to the database where detections are to be deprecated.
  • tableName (str) - The name of the detection table in which to perform the operation.

To Do: Make this a general propagate deprecations function like the one used in the Automator? That way we don't get the double deprecation of WFAU reprocessed detections - first they are deprecated by CU4 with deprecation+128, then they are set to 255 by Automator.

findRequiredDeepProducts(programme, minProdID=None, maxProdID=None, reqProdIDs=None, fieldIDs=None)

source code 

Find required products of all types

Parameters:
  • programme (DataFactory.ProgrammeTable) - Programme table for current database with the current row set to that of the programme to be curated.
  • minProdID (int) - Restrict to product IDs with this minimum value.
  • maxProdID (int) - Restrict to product IDs with this maximum value.
  • reqProdIDs (sequence(int)) - Restrict to product IDs in this list.
  • fieldIDs (list(str)) - List of fieldIDs to curate.
Returns: dict(str: list(int))
A dictionary of product ID lists required to be created for this programme referenced by product type.

getAllDBs(serverName, includeTestDBs=False, archive='ALL')

source code 

Retrieves the list of all released databases on a given server.

Parameters:
  • serverName (str) - Catalogue server to query.
  • includeTestDBs (str) - Include test databases as well.
  • archive (str) - The archive for which databases are looked up.
Returns: list(str)
List of names for all released databases found on given server.

To Do: Move into DbSession module as an independent function, and rename to getReleaseDbNames().

getAllFieldObs(db, ra, dec, dlRaDec, filterID, programmeID, frameType, dateRange=DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 0..., nustep=None, offsetPos=-1, deprecated='deprecated IN (0, 50, 51)', noDeeps=False, onlyTileComps=False, getConfToo=False)

source code 

Selects a list of multiframeIDs of given frame type in a certain direction, through a certain filter and in a certain programmeID.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • ra (float) - The expected right ascension of the product in degrees.
  • dec (float) - The expected declination of the product in degrees.
  • dlRaDec (float) - The matching radius in arcmin
  • filterID (int) - Only observations with this filter ID.
  • programmeID (int) - The programmeID.
  • frameType (str) - Include these frame types.
  • dateRange (DateRange) - Include data observed between these nights only.
  • nustep (int) - Select level of microstepping.
  • offsetPos (int) - Position of pawprint in the tile
  • deprecated (str) - Deprecation string
  • noDeeps (bool) - Exclude deep products?
  • onlyTileComps (bool) - Only use stacks that are components of tiles
Returns: set(int)
Set of multiframe IDs.

To Do: Make a method of a Stacker class that CU13 and CU6 derive from?

getAllFieldObsProd(db, productList, programmeID, frameType, dateRange=DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 0..., deprecated='deprecated IN (0, 50, 51)', noDeeps=False, onlyTileComps=False, getConfToo=False)

source code 

Selects a list of multiframeIDs of given frame type in a certain direction, through a certain filter and in a certain programmeID.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • productList (list(int)) - A list of products
  • programmeID (int) - The programmeID.
  • frameType (str) - Include these frame types.
  • dateRange (DateRange) - Include data observed between these nights only.
  • nustep (int) - Select level of microstepping.
  • offsetPos (int) - Position of pawprint in the tile
  • deprecated (str) - Deprecation string
  • noDeeps (bool) - Exclude deep products?
  • onlyTileComps (bool) - Only use stacks that are components of tiles
Returns: set(int)
Set of multiframe IDs.

To Do: Make a method of a Stacker class that CU13 and CU6 derive from?

getBBFilter(db, nbFilterName, programmeID)

source code 

Retrieves broad band filter that overlaps the given narrow band filter.

Parameters:
  • db (DbSession) - Connection to database to query.
  • nbFilterName (str) - Short name of the narrow band filter.
  • programmeID (int) - UID of the programme that uses this filter.
Returns: str
Short name of the broad band filter that matches the wavelength range of this narrow band filter.

getFrameSelection(imageType, noDeeps=False, deepOnly=False, alias='', selType='%stack')

source code 
Returns: tuple(str, list(str))
Frame type selections for image types.

getGroupPositionCentres(programme, otherDB=None)

source code 

This function uses the frameSets in MergeLog to derive grouping positions for a survey. This is better than using RequiredStack, since RequiredStack only has deep stack positions and does not work in shallow surveys.

To Do: Is this function being used? CU9?

getIntStacks(db, progID, reqProdIDs, prodType, dateRange, allNuStep=True, intFrameType='stack', returnAsProdIDDict=False, getConfToo=False)

source code 

Retrieves multiframe IDs for the intermediate stacks that will form the given range of deep products over the given time frame.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • progID (int) - Select just deep products with this programme ID.
  • reqProdIDs (list(int)) - The list of productIDs
  • prodType (str) - Product type
  • dateRange (ObsCalendar.DateRange) - Include data observed between these nights only.
  • allNuStep (bool) - If True, select all micro step values.
  • intFrameType (str) - Data type of intermediate frames
  • returnAsProdIDDict (boolean) - Return as dictionary of multiframeIDs for each productID
Returns: set(int)
The set of intermediate stack multiframe IDs that will go into the given set of deep products.

getMultiframes(programme, dateRange, fieldIDs=None, deprecated='deprecated IN (0, 50, 51)', prodType='all', useDeeps=True, otherDB=None)

source code 

Returns SQL query to select multiframe IDs of images of a particular type from a given programme, with the option of restricting to certain field IDs.

Parameters:
  • programme (DataFactory.ProgrammeTable) - Programme to query (object set to required programme).
  • dateRange (ObsCalendar.DateRange) - Include data observed between these nights only.
  • fieldIDs (str) - Optional number range string of unique field IDs used to restrict selection to certain pointings of the source product type.
  • deprecated (str) - Deprecation clause to use in SQL statement.
  • prodType (str) - Type of frames to be selected.
  • useDeeps (bool) - If True, include deep images.
  • otherDB - Database connection to use if not the programme class @type otherDB DbSession()
Returns: SelectSQL
SQL query that provides the list of multiframeIDs.

getFrameSetIDs(programme, fieldIDs=None, otherDB=None)

source code 

Returns SQL query to select all frame set IDs of a given programme, with the option of restricting to certain field IDs.

Parameters:
  • programme (DataFactory.ProgrammeTable) - Programme to query (object set to required programme).
  • fieldIDs (str) - Optional number range string of unique field IDs used to restrict selection to certain pointings only.
Returns: SelectSQL
SQL query that provides the list of frameSetIDs.

getSourceIDs(programme, fieldIDs, otherDB=None)

source code 

Returns SQL query to select all source IDs of a given programme, with the option of restricting to certain field IDs.

Parameters:
  • programme (DataFactory.ProgrammeTable) - Programme to query (object set to required programme).
  • fieldIDs (str) - Optional number range string of unique field IDs used to restrict selection to certain pointings only.
Returns: SelectSQL
SQL query that provides the list of sourceIDs.

getSynFrameSetIDs(programme, dateRange, fieldIDs=None, otherDB=None)

source code 

Returns SQL query to select all synoptic frame set IDs of a given programme, with the option of restricting to certain field IDs.

Parameters:
  • programme (DataFactory.ProgrammeTable) - Programme to query (object set to required programme).
  • dateRange (ObsCalendar.DateRange) - Include data observed between these nights only.
  • fieldIDs (str) - Optional number range string of unique field IDs used to restrict selection to certain pointings only.
Returns: SelectSQL
SQL query that provides the list of synoptic frame set IDs.

getProductInfo(db, programmeID, productType, reqProdIDs='')

source code 

?

Parameters:
  • db (DbSession) - A connection to the database to query.
  • reqProdIDs (sequence(int)) - A sequence of the required productIDs.

getComponentFrames(db, multiframeID, componentType='normal')

source code 

Selects provenence multiframeIDs.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • multiframeID (int) - Multiframe ID to be checked for provenance.
  • componentType (str) - type of basic frame
Returns: tuple(set, set)
A tuple containing the set of all files that combine to give the input image and the set of those which are deprecated

getProvenence(db, deepMfID)

source code 

Selects provenence multiframeIDs.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • deepMfID (int) - Multiframe ID to be checked for provenance.
Returns: tuple(set, set)
A tuple containing the set of all files that combine to give the input image and the set of those which are deprecated

getRedoStatus(db, programmeID, redoProducts)

source code 

Works out the redo status, based on the list of products to be redone and their descendants.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • programmeID (int) - Query the programme with this unique programmeID.
  • redoProducts (list(str)) - List of product types to redo.
Returns: dict(str: bool)
A dictionary giving the redo status for a given product type.

getRequiredProductIDs(programme, minProdID=None, maxProdID=None, reqProdIDs=None, fieldIDs=None, productType=None, whereStr='', otherDB=None)

source code 

Find required products of given type. If no type given then just the products that contribute towards the source table are returned.

Parameters:
  • programme (DataFactory.ProgrammeTable) - Programme table for current database with the current row set to that of the programme to be curated.
  • minProdID (int) - Restrict to product IDs with this minimum value.
  • maxProdID (int) - Restrict to product IDs with this maximum value.
  • reqProdIDs (sequence(int)) - Restrict to product IDs in this list.
  • fieldIDs (list(str)) - List of fieldIDs to curate.
  • productType (str) - Optional product type for which to return product IDs, defaults to programme's sourceProdType.
  • whereStr (str) - Optional additional selection criteria.
Returns: list(int)
A list of product IDs to create of this product type for this programme.

getStacks(db, programmeID, inputImages, prodID, onlyMfIDs=False, number=None, nonDepOnly=True, productMfID=None)

source code 

Queries database for combined stacks for all the given normal images.

Parameters:
  • db (DbSession) - A connection to the database to query.
  • programmeID (int) - The programmeID that contains the product
  • inputImages (set(int)) - The input images that match the selection criteria
  • prodID (int) - ProductID of deep stack. If None, do not use
  • onlyMfIDs (bool) - If True, only return the multiframe IDs of the stacks.
  • number (int) - Optionally restrict the number of intermediate stacks per deep stack to a specified number.
  • nonDepOnly (bool) - If True, only select stacks where all extensions are non-deprecated.
  • doNotUpdate (bool) - If True, do not update the database
Returns: set(int) or tuple(set(tuple), set(int))
Stack multiframe IDs and optionally also the ID of the confidence images, the file names, the frame types and the totalExpTime, together with a set of multiframe IDs that were not found.