Home       Trees       Indices       Help   
WSA
Package invocations :: Package cu0 :: Module DataBuilder :: Class DataBuilder
[hide private]

Class DataBuilder

source code

Executes CUs in parallel-mode, preparing ingest files.

Nested Classes [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: IngCuError

Instance Methods [hide private]
 
__init__(self, cuNums=False, curator='rsc', database='WSA', beginDate=False, endDate=False, versionStr=False, outPath=False, noMfID=False, ReDo=False, programmeList=False, progOrder=False, isTrialRun=False, xferlog=False, timeCheck=False, numThreads=False, writeMfID=False, subDir=False, janet=False, forceXfer=False, excludeData=False, deprecation=False, reproMode=False, forceMosaic=False, detectionSubset=False, keepWorkDir=False, ffluOnly=False, getHeader=False, gesOptions=False, comment='A useful comment')
Initialises data members for all database state flags and details for this curation session.		 source code
 
_onRun(self)
Run each CU requested.		 source code
 
getGesMJDs(self, startMJD, endMJD)
Get the list of MJDs for the GES transfer.		 source code
 
splitDateList(self, dateList)
Split the dateList into multiple similar sized threads		 source code
FitsList
createDailyLists(self, xferLog=None)
Create the daily filelists.		 source code
FitsList
deprecateJpgs(self, deprFitsList)
Deprecate jpgs of deprecated FITS files.		 source code
 
parseXferList(self)
Parse the transfer list and check for files.		 source code
 
updateComment(self, dateVersStr, cuNum)
Update the ArchiveCurationHistory comment.		 source code
 
updateFlatFileLookUp(self, csvFileName)
Update the FlatFileLookUp table.		 source code
 
updateMfIDs(self, dailyFileListDict)
Write multiframe IDs into FlatFileLookUp.		 source code

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: __del__, attributesFromArguments, createFileList, createLogFileName, createMonthlyDetSchema, createWorkingDir, getDetTable, getErrorFileName, getMultiframeIDs, getProcessOrder, getProgramIDs, makeSysCmd, removeWorkingDir, run, runSysCmd, setupDetectionTable, testForProgs, translateProgID, writeErrorFiles, writeIngestList

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _connectToDb, _createProgrammeTranslation, _disconnectFromDb, _getNextCuEventID, _getNextID, _historyCommentPrefix, _onException, _prepareDbLock, _updateACHComment, _updateArchiveCurationHistory, _updateProgrammeCurationHistory

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]
 
chunks(l, n)
Yield successive n-sized chunks from l.		 source code
 
getCasuSizeDict(dateList, UKLight)
Get a rated dictionary of CASU date dirs ordered by size.		 source code

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: createFilePrefix

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _parseCuNums

Class Variables [hide private]
  diskList = None
hash(x)
  casuDisk = None
hash(x)

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: beginDateDef, endDateDef, keepWorkDir, reqWorkDir, sysc

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _defComment, _fileList, _isDbLocked, _numSessions

Instance Variables [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _dateList

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, cuNums=False, curator='rsc', database='WSA', beginDate=False, endDate=False, versionStr=False, outPath=False, noMfID=False, ReDo=False, programmeList=False, progOrder=False, isTrialRun=False, xferlog=False, timeCheck=False, numThreads=False, writeMfID=False, subDir=False, janet=False, forceXfer=False, excludeData=False, deprecation=False, reproMode=False, forceMosaic=False, detectionSubset=False, keepWorkDir=False, ffluOnly=False, getHeader=False, gesOptions=False, comment='A useful comment')
(Constructor)

source code 

Initialises data members for all database state flags and details for this curation session. Writes a database lock and opens a connection. Also initiates a Programme table object.

Parameters:
  • beginDate (str) - First date to process, eg. 20050101.
  • comment (str) - Descriptive comment as to why curation task is being performed.
  • cuNums (list[int]) - Curation task numbers.
  • curator (str) - Name of curator.
  • database (str) - Name of the database to connect to.
  • detectionSubset (list(str)) - Process subset of [Astrometry table, Photometry table, Raw table].
  • endDate (str) - Last date to process, eg. 20050131.
  • excludeData (list(str)) - Exclude given files from processing (CU4).
  • ffluOnly (bool) - Don't transfer but update FlatfileLookup table.
  • forceXfer (bool) - Force the data transfer.
  • gesOptions (str) - Options for GES transfers.
  • getHeader (bool) - Only transfer FITS headers.
  • isTrialRun (bool) - If True, do not perform database modifications.
  • janet (bool) - Use JANET instead of UKLight.
  • forceMosaic (bool) - Create jpg for mosaic.
  • keepWorkDir (bool) - Don't remove working directory.
  • noMfID (bool) - If True, don't write new MfID into FITS file.
  • numThreads (int) - Number of scp transfer or processing threads.
  • outPath (str) - Directory where data is written to.
  • programmeList (list(str)) - Only process data for given programmes (accepts keywords 'all', 'ns' (non-survey), 'ukidss' (all 5 main surveys)).
  • progOrder (list(str)) - The order of processing of the programmes, all programmes not explicitely named can be put anywhere in the list as 'others'.
  • ReDo (bool) - If True, overwrite existing MfID (CU1/3) or jpg (CU2).
  • reproMode (str) - Mode for reprocessed data: otm (one-to-many dirs) or datedir suffix.
  • subDir (str) - The subdirectory containing FITS files.
  • timeCheck (bool) - Determines if existing files should be checked for their timestamp against the same file at CASU;
  • versionStr (str) - Version number of the data.
  • writeMfID (bool) - If True, only write MfIDs into FITS files.
  • xferlog (str) - Logfile containing files to be ingested.
Overrides: object.__init__

_onRun(self)

source code 

Run each CU requested.

Overrides: wsatools.DbConnect.IngCuSession.IngCuSession._onRun

createDailyLists(self, xferLog=None)

source code 

Create the daily filelists.

Parameters:
  • xferLog (str) - Logfile containing files to be ingested.
Returns: FitsList
A FitsList object holding daily file information.

deprecateJpgs(self, deprFitsList)

source code 

Deprecate jpgs of deprecated FITS files.

Parameters:
  • deprFitsList (list) - List of deprecated FITS files.
Returns: FitsList
A FitsList object holding daily file information.

updateComment(self, dateVersStr, cuNum)

source code 

Update the ArchiveCurationHistory comment.

Parameters:
  • dateVersStr (str) - The date directory name.
  • cuNum (int) - Curation task number.

updateFlatFileLookUp(self, csvFileName)

source code 

Update the FlatFileLookUp table.

Parameters:
  • csvFileName (str) - CSV file where data is written to.

updateMfIDs(self, dailyFileListDict)

source code 

Write multiframe IDs into FlatFileLookUp.

Parameters:
  • dailyFileListDict (dict) - Dictionary containing daily file lists.

   Home       Trees       Indices       Help   
WSA
Generated by Epydoc 3.0.1 on Mon Sep 8 15:46:45 2014 http://epydoc.sourceforge.net