Package wsatools :: Module Automator

Module Automator

Automatic curation routines. Determines what needs to run on a given programme and automatically calls those curation tasks.

Author: N.J.G. Cross

Organization: WFAU, IfA, University of Edinburgh

Contributors: R.S. Collins

Functions

[hide private]

bool

autoCurate(cli, progID, releaseNum=0, dateRange=DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 0..., redoSources=False, redoProducts=None, redoRecal=False, redoNeigh=False, doCu6=False, skipMosaicCheck=False, minProdID=None, maxProdID=None, fileListForType=None, isRerelease=False, numCPUs=None, fields=None)
Automatically runs the necessary curation tasks to prepare a programme's data for release. source code

prepareCu(cuName, db, cli, comment='A descriptive comment')
Must be run before each CuSession in the batch script. source code

deprecateMissingDeepFrames(db, programmeID)
Deprecate deep stack detector frames with no components.

source code

deprecateDodgyDeeps(db, programmeID)
Deprecates deeps where some components have been deprecated

source code

deprecateProducts(releaseNum, db, programme, prodIDs=None, depOld=False, deleteDeeps=False)
Deprecates products of given release numbers for the current programme.

source code

dict(int: Cu13.ProductDetails)

findCreatedProducts(db, programmeID, releaseNum, reqProdsOfType, ingestedIDs, prodType)
Searches file system to find existing products from the list provided of given file type and release number for the given programme.

source code

tuple(set(int), int)

getIngestedProducts(db, programmeID, releaseNum, prodIDs, productType)
Searches database for all existing products ingested in the database for the given programme and date range.

source code

set(int)

getMergedMFIDs(db, programme, isSynoptic=False)
Retrieves all multiframe IDs listed in the merge log (synoptic if requested) of the given programme.

source code

int

getReleaseNum(programme)
Queries database to find the next available release number for a given survey.

source code

bool

ingestDeepStacks(db, fileLists, releaseNum)
Ingests into given database the deep stack products listed in the given file lists.

source code

getProductType(filePath)
Checks type of product being ingested

source code

bool

isCuTaskIncomplete(programme, cuNum, refCuNum=13)
Tests whether a programme has had the given CU task run for the current release.

source code

bool

isUncalibrated(programme)
Tests a deep programme to see if CU8 needs to be applied to it.

source code

propagateDeprecations(programme)
Propagate deprecations from MultiframeDetector to the detection table of the given programme.

source code

Variables

[hide private]

__package__ = 'wsatools'

Function Details

[hide private]

autoCurate(cli, progID, releaseNum=0, dateRange=DateRange(begin=<mx.DateTime.DateTime object for '1753-01-01 0`...`, redoSources=False, redoProducts=None, redoRecal=False, redoNeigh=False, doCu6=False, skipMosaicCheck=False, minProdID=None, maxProdID=None, fileListForType=None, isRerelease=False, numCPUs=None, fields=None)

source code

Automatically runs the necessary curation tasks to prepare a programme's data for release.

Parameters:

cli (CLI) - Standard command-line options for CuSessions.
progID (str) - Unique ID for programme, or acronym.
releaseNum (int) - Curate products of this specific WFAU-product version number.
dateRange (DateRange) - Curate data observed between these nights only.
redoSources (bool) - If True, force re-creation of source table.
redoProducts (list(str)) - List of specified product types to recreate along with any descendants.
redoRecal (bool) - If True, force recalibration of deep survey detections from variability analysis.
redoNeigh (bool) - If True, force recreation of neighbour tables.
doCu6 (bool) - If True, also run CU6 with default options at the end.
skipMosaicCheck (bool) - If True, force curation, even if mosaic provenance is incomplete.
minProdID (str) - Minimum product ID to curate for deep stacks.
maxProdID (str) - Maximum product ID to curate for deep stacks.
fileListForType (str) - Append newly created image and catalogue products to file lists written to the file paths given in this dictionary referenced by product type.
isRerelease (bool) - If True, deprecate existing products of the current release number before proceeding with curation.
numCPUs (int) - Optionally limit number of CPUs to this number.
fields (list(str)) - List of fieldIDs to curate.

Returns: bool

True, if ready for release.

deprecateMissingDeepFrames(db, programmeID)

source code

Deprecate deep stack detector frames with no components.

Parameters:

db (DbSession) - Connection to database where deprecations are to be made.
programmeID (int) - Only deprecrate frames belonging to this unique programme ID.

deprecateDodgyDeeps(db, programmeID)

source code

Deprecates deeps where some components have been deprecated

Parameters:

db (DbSession) - Connection to database where deprecations are to be made.
programmeID (int) - UID of current programme

deprecateProducts(releaseNum, db, programme, prodIDs=None, depOld=False, deleteDeeps=False)

source code

Deprecates products of given release numbers for the current programme. Optionally supply a specific list of product IDs to deprecate.

Parameters:

releaseNum (int) - Release number currently being curated.
db (DbSession) - Connection to database where deprecations are to be made.
programme (df.ProgrammeTable) - Programme table for current database with the current row set to that of the programme for which the products will be deprecated.
prodIDs (tuple(prodType, list(int))) - List of product IDs to deprecate with prodType.
depOld (bool) - If True, deprecate files with release numbers older than the current release, else just deprecate files with the current release number.

findCreatedProducts(db, programmeID, releaseNum, reqProdsOfType, ingestedIDs, prodType)

source code

Searches file system to find existing products from the list provided of given file type and release number for the given programme.

Parameters:

db (DbSession) - Connection to database containing the observations.
programmeID (int) - Unique ID of desired programme's products.
releaseNum (int) - Release number of desired products.
reqProdsOfType (list(int)) - List of required productIDs for each product type.
ingestedIDs (list(int)) - List of already ingested productIDs for product type.
prodType (str) - Product type.

Returns: dict(int: Cu13.ProductDetails)

Details of products already created reference by productID.

getIngestedProducts(db, programmeID, releaseNum, prodIDs, productType)

source code

Searches database for all existing products ingested in the database for the given programme and date range. If deep product IDs provided then just these existing products of the given release number are retrieved, otherwise all non-deprecated observations with catalogues of the given type are retrieved.

Parameters:

db (DbSession) - Connection to database containing the observations.
programmeID (int) - Unique ID of desired programme's observations.
releaseNum (int) - Release number of data to retrieve.
prodIDs (list(int)) - List of product IDs for all required deep products.
productType (str) - Frame type string pattern of required product.

Returns: tuple(set(int), int)

Set of product IDs already assigned in the database and the total number of multiframes with assigned product IDs.

getMergedMFIDs(db, programme, isSynoptic=False)

source code

Retrieves all multiframe IDs listed in the merge log (synoptic if requested) of the given programme.

Parameters:

db (DbSession) - Connection to database containing desired merge log.
programme (df.ProgrammeTable) - Programme table for current database with the current row set to the desired programme.
isSynoptic (bool) - If True, retrieve multiframe IDs from the synoptic merge log.

Returns: set(int)

All multiframe IDs listed in the programme's merge log table.

getReleaseNum(programme)

source code

Queries database to find the next available release number for a given survey.

Parameters:

programme (DataFactory.ProgrammeTable) - Database Programme table information with the row set to the current programme.

Returns: int

Next available release number.

ingestDeepStacks(db, fileLists, releaseNum)

source code

Ingests into given database the deep stack products listed in the given file lists.

Parameters:

db (DbSession) - Connection to ingest database.
fileLists (list(str)) - List of paths to files containing the list of deep stack product files to ingest.
releaseNum (int) - Release number of data to ingest.

Returns: bool

True, if stacks and catalogues are ingested.

isCuTaskIncomplete(programme, cuNum, refCuNum=13)

source code

Tests whether a programme has had the given CU task run for the current release.

Parameters:

programme (df.ProgrammeTable) - Programme table for current database with the current row set to the programme to test.
cuNum (int) - Unique CU task ID for the CU to test for incompleteness.
refCuNum (int) - Unique CU task ID for the CU that should always occur sometime prior to the CU being tested on a given release run.

Returns: bool

True, if given programme has had a successful CU run for the current release.

isUncalibrated(programme)

source code

Tests a deep programme to see if CU8 needs to be applied to it.

Parameters:

programme (df.ProgrammeTable) - Programme table for current database with the current row set to the programme to test.

Returns: bool

True, if given programme has not had a successful CU8 run following a deep catalogue ingest.

propagateDeprecations(programme)

source code

Propagate deprecations from MultiframeDetector to the detection table of the given programme.

Parameters:

programme (df.ProgrammeTable) - Programme table for current database with the current row set to that of the programme where the deprecations require propagating.