Package wsatools :: Module FitsToDb :: Class ProvenanceFiller
[hide private]

Class ProvenanceFiller

source code


Updates the Provenance table. Retrieves a list of non-deprecated combiframe IDs that are new or have incomplete or indeterminable (if the -q/--quick option is not selected) provenance records to be updated. The header of the FITS file for each combiframe is read to determine the list of provenance files, which are converted to multiframe IDs (to insert into the Provenance table) via a look-up table. This table lists all frames that are not deprecated due to reprocessing, referrenced by file name (which is combined with the night to distinguish between common file names across nights for confidence images etc.) and so should provide a unique list of the latest versions of each of the component files. If this list is not unique an error is thrown, as this suggests a database inconsistency. Since only non-deprecated combiframes have their provenance updated, we only need to consider the latest reprocessed version.

The final statistics report the number of broken combiframe files that list no provenance, as well as the number of combiframe files where a component file is missing from our database. A re-run in -q/-quick mode will recheck these files as well as files that are broken due to providing incomplete provenance lists, due to e.g. an interrupted observation run (these will normally become deprecated by quality control later on, so will not be rechecked on subsequent runs).


Notes:
Nested Classes [hide private]
    Nested Errors and Exceptions

Inherited from DbConnect.CuSession.CuSession: CuError

Instance Methods [hide private]
 
_onRun(self)
Update db: Prepare file list, and read the FITS files.
source code
 
_prepare(self)
Retrieve list of records to be updated and prepare the mfid<->filename look-up table.
source code
list(tuple(str, int))
_getRecordsList(self)
Retrieve a list of records that need updating in the database.
source code
 
updateDb(self, mfid, fitsFile, filePathName)
Inserts provenance info for given combiframe into table Provenance if its FITS file contains the PROV000x keywords.
source code
str
_frameTypeLike(self, frameTypes)
Returns: A SQL WHERE string of selected frameTypes.
source code

Inherited from DbUpdater: getErrorLog, getFileList

Inherited from DbUpdater (private): _stripServerName

Inherited from DbConnect.CuSession.CuSession: __del__, __init__, logBrokenFile, run

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]

Inherited from DbConnect.CuSession.CuSession: logException

Class Variables [hide private]
  cuNum = 23
Curation task number.
  combiFrameTypes = {'2': ['diff', 'diffconf'], 'nJitter': ['sta...
A list of frameTypes that are combiframes (i.e.
  isQuickRun = False
Don't re-check for missing provenance of frames that are already in the Provenance table, where we don't know how many components they should have (i.e.
  recheckDate = None
Only re-check stackconf and deep frames observed after this date.
  _guessDict = None
Dictionary of multiframeIDs referenced just by file name.
  _mfidDict = None
Dictionary of multiframeIDs referenced by file name and night.

Inherited from DbConnect.CuSession.CuSession: archive, comment, cuEventID, curator, eTypes, isDayStampedLog, onlyNonSurveys, onlySurveys, programme, programmeID, resultsFilePathName, shareFileID, sysc

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

_onRun(self)

source code 

Update db: Prepare file list, and read the FITS files.

Overrides: DbConnect.CuSession.CuSession._onRun

_prepare(self)

source code 

Retrieve list of records to be updated and prepare the mfid<->filename look-up table.

Overrides: DbUpdater._prepare

_getRecordsList(self)

source code 

Retrieve a list of records that need updating in the database. As DbUpdater but with self._recordSet modified to just select frames with provenance information that is not already in the Provenance table.

Returns: list(tuple(str, int))
List of records that need updating in the database, in the form (filePathName, multiframeID), with the server name stripped from the database fileName attribute.
Overrides: DbUpdater._getRecordsList

updateDb(self, mfid, fitsFile, filePathName)

source code 

Inserts provenance info for given combiframe into table Provenance if its FITS file contains the PROV000x keywords.

Parameters:
  • mfid (int) - MultiframeID of combiframe for which provenance info is to be inserted into table Provenance.
  • fitsFile (pyfits.HDUList) - Open FITS file containing the data for this mfid.
  • filePathName (str) - Full path to the FITS file.
Overrides: DbUpdater.updateDb

_frameTypeLike(self, frameTypes)

source code 
Parameters:
  • frameTypes (list(str)) - A list of frameTypes to be included.
Returns: str
A SQL WHERE string of selected frameTypes.

Class Variable Details [hide private]

combiFrameTypes

A list of frameTypes that are combiframes (i.e. have provenance info) and the parameter that defines the number of component frames.

Value:
{'2': ['diff', 'diffconf'],
 'nJitter': ['stack',
             'leavstack',
             'leavstackconf',
             'filtleavstack',
             'filtleavstackconf'],
 'nOffsets': ['tile%'],
 'nuStep': ['leav', 'leavconf'],
...

isQuickRun

Don't re-check for missing provenance of frames that are already in the Provenance table, where we don't know how many components they should have (i.e. just stackconf, %deep%, dark, sky).

Value:
False

recheckDate

Only re-check stackconf and deep frames observed after this date. Expects string, e.g. '2005-05-05' or '20050505' or '05A'. For semesters it takes the first night of the semester.

Value:
None