Home       Trees       Indices       Help   
WSA
Package invocations :: Package cu3 :: Module cu3 :: Class Cu3
[hide private]

Class Cu3

source code

Invokes frame metadata ingestion for a given log of files transferred from the data processing centre.

method: Establish locked DB connectivity; initiate curation objects; cycle though the transfer log, accumulating individual table metadata in CSV files. The database bulk ingest from those CSV files and the update of the curation logs will be done at a later stage.

Nested Classes [hide private]
  NoFilesError
A curation error instead of a programming error.

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: IngCuError

Instance Methods [hide private]
 
__init__(self, curator='rsc', database='WSA', csvSharePath=None, programmeList=False, noMfID=False, ReDo=False, onlyMfID=False, excludeFiles=False, isTrialRun=False, keepWorkDir=False, forceComp=False, isWfauProduct=False, xferlog=None, comment='A useful comment')
Initialises data members for all database state flags and details for this curation session.		 source code
 
_initFromDB(self)
initialize constants from the DB		 source code
 
_onRun(self)
Do CU3.		 source code
 
checkForSatData(self, commonPath) source code
 
parseSatData(self, satList, fitsList) source code
 
updateTableDict(self) source code
 
signum(self, integ) source code
 
addNonSurveyProgramme(self, newProgs)
Insert new programme ID entries into the Programme table.		 source code
 
createProgrammeCsv(self, progIDofFile=None)
Query Programme table and write programmes.csv file for C++ code.		 source code
 
getProgIDs(self)
Reads FITS files to determine the set of programme IDs for the metadata that will be ingested in this curation event.		 source code
 
selectFilesByProg(self)
Select files by programme list.		 source code
 
prepareMfIDlookup(self)
Get Multiframe IDs for processed dates.		 source code
 
setTableVariables(self)
set the variables dependend of the tables used in this cu		 source code
 
splitFileList(self, update=False)
Split the production frames from the general filelist.		 source code
 
writeMfIDs(self)
Write the multiframeIDs into the files.		 source code

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: __del__, attributesFromArguments, createFileList, createLogFileName, createMonthlyDetSchema, createWorkingDir, getDetTable, getErrorFileName, getMultiframeIDs, getProcessOrder, getProgramIDs, makeSysCmd, removeWorkingDir, run, runSysCmd, setupDetectionTable, testForProgs, translateProgID, writeErrorFiles, writeIngestList

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _connectToDb, _createProgrammeTranslation, _disconnectFromDb, _getNextCuEventID, _getNextID, _historyCommentPrefix, _onException, _prepareDbLock, _updateACHComment, _updateArchiveCurationHistory, _updateProgrammeCurationHistory

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: createFilePrefix

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _parseCuNums

Class Variables [hide private]
  cuedFileExists = False
hash(x)

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: beginDateDef, endDateDef, keepWorkDir, reqWorkDir, sysc

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _defComment, _fileList, _isDbLocked, _numSessions

Instance Variables [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _dateList

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, curator='rsc', database='WSA', csvSharePath=None, programmeList=False, noMfID=False, ReDo=False, onlyMfID=False, excludeFiles=False, isTrialRun=False, keepWorkDir=False, forceComp=False, isWfauProduct=False, xferlog=None, comment='A useful comment')
(Constructor)

source code 

Initialises data members for all database state flags and details for this curation session. Writes a database lock and opens a connection. Also initiates a Programme table object.

Parameters:
  • curator (str) - Name of curator.
  • comment (str) - Descriptive comment as to why curation task is being performed.
  • csvSharePath (str) - Directory where data is written to.
  • database (str) - Name of the database to connect to.
  • excludeFiles (list(str)) - Excluded files from processing.
  • forceComp (bool) - Compress uncompressed FITS files during MfID writing.
  • isTrialRun (bool) - If True, do not perform database modifications.
  • isWfauProduct (bool) - If True, files (e20*) are in a products subdir.
  • keepWorkDir (bool) - Don't remove working directory.
  • noMfID (bool) - If True, don't write new MfID into FITS file.
  • onlyMfID (bool) - If True, only write new MfID into FITS file.
  • programmeList (list(str)) - Only process data for given programmes (accepts keywords 'all', 'ns' (non-survey), 'ukidss' (all 5 main surveys)).
  • ReDo (bool) - If True, overwrite existing MfID.
  • xferlog (str) - Logfile containing files to be ingested.
Overrides: object.__init__

_onRun(self)

source code 

Do CU3.

Overrides: wsatools.DbConnect.IngCuSession.IngCuSession._onRun

addNonSurveyProgramme(self, newProgs)

source code 

Insert new programme ID entries into the Programme table.

Parameters:
  • newProgs (set(str)) - A set of new project DFS ID strings, for which programme IDs must be assigned.

createProgrammeCsv(self, progIDofFile=None)

source code 

Query Programme table and write programmes.csv file for C++ code.

To Do: It might make more sense to write the contents of self._progIDofName to the programmes.csv, which includes the merging of different types of programmes. This would save some code duplication in exmeta.

getProgIDs(self)

source code 

Reads FITS files to determine the set of programme IDs for the metadata that will be ingested in this curation event. Any new programmes are identified and the Programme table is updated.

splitFileList(self, update=False)

source code 

Split the production frames from the general filelist. If update is True, re-write the general file list as well.


   Home       Trees       Indices       Help   
WSA
Generated by Epydoc 3.0.1 on Mon Sep 8 15:46:48 2014 http://epydoc.sourceforge.net