Home       Trees       Indices       Help   
WSA
Package wsatools :: Package DbConnect :: Module IngIngester :: Class CuIngest
[hide private]

Class CuIngest

source code

Ingests products of parallel CU runs by DataBuilder.

Nested Classes [hide private]

Inherited from IngCuSession.IngCuSession: IngCuError

Instance Methods [hide private]
 
__init__(self, curator='rsc', database='WSA', isTrialRun=False, logFileName=False, ingestOneLog=False, cuNums=False, forceIngest=False, autoCommit=False, excludedTables=False, omitObjIDUpdate=False, isParallelRun=False, comment='A useful comment')
Initialises data members for all database state flags and details for this curation session.		 source code
 
__del__(self)
Tidy-up IngCuSession.		 source code
 
_onRun(self)
Ingest data.		 source code
File.File
_createLock(self)
Create a lock, writing info to sysLog.		 source code
 
ingestCU2(self, ingLogFile, histInfo, _ingestOrder, _fileInfo)
Ingest CU2 data.		 source code
 
ingestCU3(self, ingLogFile, histInfo, ingestOrder, fileInfo)
Ingest CU3 data.		 source code
 
ingestCU4(self, ingLogFile, histInfo, ingestOrder, fileInfo)
Ingest CU4 data.		 source code
 
checkCUEDFile(self, ingLogFile, histInfo)
Check flag file to say that the data are already ingested.		 source code
 
tryIngest(self, ingestCU, ingLogFile, histInfo, ingestOrder=None, fileInfo=None, enableRollback=True)
General ingest function for any CU.		 source code
 
updateCUEDFile(self, ingLogFile, histInfo)
Update CUed file.		 source code
 
ingestData(self, ingLogFile, ingestOrder, fileInfo, histInfo)
Ingest all files described in given log file.		 source code
 
parseIngest(self, logFile=None)
Do the appropriate ingest for the different CUs.		 source code
 
deprecateOldFrames(self)
Deprecate old versions of frames that are reingested in Multiframe.		 source code
int, int
setUnfilteredIDs(self, deprCode)
Set unfilteredID entries.		 source code
 
ingest(self, ingestOrder, ingestHost, fileInfo, ingestDate, isCu3=True, cuEventID=0)
Ingest the data file.		 source code
 
dropVVVConstraints(self, tableName)
Drop the constraints on monthly VVV tables.		 source code
 
checkVVVIndices(self)
Checks if the SQL for all objID indices exist for monthly tables.		 source code
list(str)
inspectInDir(self)
Returns: Ingest logs in dbShare that need ingesting.		 source code
 
readIngLog(self, logFile)
Read the given ingest log file and print its content to the screen.		 source code
 
reAllocateDXSSVFrames(self)
Re-allocate DXS_SV frames to the main programme.		 source code
 
removeProductProcessingLines(self)
Use newlyIngested flag to select newly ingested filenames Delete entry in ProductProcessing and GroutingLinks		 source code
set(str)
selectFiles(self, subDir, beginDate, endDate)
Selects all FITS files in given sub-directory of all disks where the date is within the given range.		 source code
 
setNewlyIngested(self, cuEventID)
Reset all the 'newlyIngested' flags to zero.		 source code
 
spaceOnFileShare(self, server) source code
 
updateHistory(self, historyUpdate, status, rolledBack, comment='')
Update Archive and Programme Curation History with data from the logfile.		 source code
 
updateObjIDs(self, tableName, cuEventID)
Since the objectIDs couldn't be calculated beforehand they have to be updated separately based on their value and the maximal value found in the database.		 source code

Inherited from IngCuSession.IngCuSession: attributesFromArguments, createFileList, createLogFileName, createMonthlyDetSchema, createWorkingDir, getDetTable, getErrorFileName, getMultiframeIDs, getProcessOrder, getProgramIDs, makeSysCmd, removeWorkingDir, run, runSysCmd, setupDetectionTable, testForProgs, translateProgID, writeErrorFiles, writeIngestList

Inherited from IngCuSession.IngCuSession (private): _connectToDb, _createProgrammeTranslation, _disconnectFromDb, _getNextCuEventID, _getNextID, _historyCommentPrefix, _onException, _prepareDbLock, _updateACHComment, _updateArchiveCurationHistory, _updateProgrammeCurationHistory

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]

Inherited from IngCuSession.IngCuSession: createFilePrefix

Inherited from IngCuSession.IngCuSession (private): _parseCuNums

Class Variables [hide private]

Inherited from IngCuSession.IngCuSession: beginDateDef, endDateDef, keepWorkDir, reqWorkDir, sysc

Inherited from IngCuSession.IngCuSession (private): _defComment, _fileList, _isDbLocked, _numSessions

Instance Variables [hide private]

Inherited from IngCuSession.IngCuSession (private): _dateList

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, curator='rsc', database='WSA', isTrialRun=False, logFileName=False, ingestOneLog=False, cuNums=False, forceIngest=False, autoCommit=False, excludedTables=False, omitObjIDUpdate=False, isParallelRun=False, comment='A useful comment')
(Constructor)

source code 

Initialises data members for all database state flags and details for this curation session. Writes a database lock and opens a connection. Also initiates a Programme table object.

Parameters:
  • curator (str) - Name of curator.
  • comment (str) - Descriptive comment as to why curation task is being performed.
  • database (str) - Name of the database to connect to.
  • forceIngest (bool) - If True and ingestOneLog is True, ingest log file.
  • logFileName (str) - Full path name of a log file to be read. Excludes ingest.
  • ingestOneLog (bool) - If True, ingest log file given by logFileName.
  • cuNums (list[int]) - Curation task numbers.
  • autoCommit (bool) - If True, do not auto-commit transactions. This speeds up transactions through minimal logging.
  • excludedTables (list(str)) - Exclude these tables from ingest.
  • omitObjIDUpdate (bool) - If True, don't update objIDs.
  • isTrialRun (bool) - If True, do not perform database modifications.
Overrides: object.__init__

__del__(self)
(Destructor)

source code 

Tidy-up IngCuSession.

Overrides: IngCuSession.IngCuSession.__del__
(inherited documentation)

_onRun(self)

source code 

Ingest data.

Overrides: IngCuSession.IngCuSession._onRun

_createLock(self)

source code 

Create a lock, writing info to sysLog. If lock exists, abort.

Returns: File.File
Lock file object.

deprecateOldFrames(self)

source code 

Deprecate old versions of frames that are reingested in Multiframe.

To Do: This function will eventually contain the contents of the SQL fragment, as a DB API update method.

setUnfilteredIDs(self, deprCode)

source code 

Set unfilteredID entries.

Parameters:
  • deprCode (str) - Deprecation code.
Returns: int, int
Number of updated entries in Multiframe and MultiframeDetector.

ingest(self, ingestOrder, ingestHost, fileInfo, ingestDate, isCu3=True, cuEventID=0)

source code 

Ingest the data file.

Parameters:
  • ingestOrder (list) - List of table names, giving order in which ingest occurs.
  • fileInfo (dict(str: dict(str: str))) - Dictionary containing schema and ingest file names.
  • ingestDate (str) - Date of ingested data.
  • isCu3 (bool) - If True, expects a CU3 ingest of a CSV file, else expects a CU4 binary file ingest, where indices must remain in place for objID updates.
  • cuEventID (int) - The cuEventID of the ingested data.

inspectInDir(self)

source code 
Returns: list(str)
Ingest logs in dbShare that need ingesting.

readIngLog(self, logFile)

source code 

Read the given ingest log file and print its content to the screen. No ingest takes place.

Parameters:
  • logFile (File) - IngestLogFile object

selectFiles(self, subDir, beginDate, endDate)

source code 

Selects all FITS files in given sub-directory of all disks where the date is within the given range.

Parameters:
  • subDir (str) - Common RAID system sub-directory name to search in for each disk.
  • beginDate (int) - First night to include.
  • endDate (int) - Last night to include.
Returns: set(str)
Set of FITS file paths.

To Do: Useful for FitsUtils?

updateHistory(self, historyUpdate, status, rolledBack, comment='')

source code 

Update Archive and Programme Curation History with data from the logfile.

Parameters:
  • historyUpdate (dict) - Dictionary containing the updates for the Archive and the Programme CurationHistory.
  • status (int) - Ingest status for the ProgrammeCurationHistory.
  • rolledBack (int) - Flag to indicate if the transaction has been rolled back.

updateObjIDs(self, tableName, cuEventID)

source code 

Since the objectIDs couldn't be calculated beforehand they have to be updated separately based on their value and the maximal value found in the database.

Parameters:
  • tableName (str) - Name of the table to update.
  • cuEventID (int) - The cuEventID of the ingested data.

   Home       Trees       Indices       Help   
WSA
Generated by Epydoc 3.0.1 on Mon Sep 8 15:46:52 2014 http://epydoc.sourceforge.net