Home       Trees       Indices       Help   
WSA
Package wsatools :: Package DbConnect :: Module IngCuSession :: Class IngCuSession
[hide private]

Class IngCuSession

source code

A parallel curation session for transfer/ingest CUs 1 to 4.

Nested Classes [hide private]
  IngCuError
A curation error instead of a programming error.
Instance Methods [hide private]
 
__init__(self, cuNum=-9999, curator='rsc', comment='A useful comment', reqWorkDir=False, keepWorkDir=False, database='WSA', userName='ldservro', autoCommit=False, isTrialRun=False)
Initialises data members for all database state flags and details for this curation session.		 source code
 
__del__(self)
Tidy-up IngCuSession.		 source code
 
_connectToDb(self, maxTries=100)
Connect to the database.		 source code
 
_createProgrammeTranslation(self)
Create the translation dictionary of programme identification strings to programme ID numbers for all programmes.		 source code
 
_disconnectFromDb(self)
Disconnect from the database.		 source code
 
_getNextCuEventID(self)
Obtain the next available cuEventID.		 source code
int
_getNextID(self, attrName, tableName, where='')
Obtain next available ID number in the db for a given ID attribute.		 source code
str
_historyCommentPrefix(self, success, rolledBack)
Prefix for the comment in the ArchiveCurationHistory table.		 source code
 
_onException(self)
Tidy up run tasks specific to this CU following an exception.		 source code
 
_onRun(self)
Performs all the tasks that define this curation session.		 source code
 
_prepareDbLock(self, maxTries=100)
Writes database lock.		 source code
 
_updateArchiveCurationHistory(self, row)
Adds an entry for this curation session to the ArchiveCurationHistory database table.		 source code
 
_updateACHComment(self, newComment, cuEventID)
Updates the comment of an existing entry in ArchiveCurationHistory.		 source code
 
_updateProgrammeCurationHistory(self, row)
Adds an entry for this curation session to the ProgrammeCurationHistory database table.		 source code
 
attributesFromArguments(self, argumentNames, argDict, prefix='', types=None)
Initialize automatically instance variables from __init__ arguments.		 source code
 
createFileList(self, fileListPath=None)
Create a list of FITS files to be ingest into the database from the list supplied in the file at the given path.		 source code
str
createLogFileName(self, cuEventIDs=None)
Create the standard log file name for the current cuEventID or the given list of CU event IDs.		 source code
 
createMonthlyDetSchema(self, schemaName, detTableList, month, ReDo=False)
Copy original schema to monthly schema.		 source code
 
getDetTable(self, detTableList, line) source code
str
createWorkingDir(self)
Create a temporary working directory.		 source code
dict(str:int), dict(str:str)
getProgramIDs(self, fileListPath)
Determines the full set of programmes for a given list of FITS files.		 source code
str
getErrorFileName(self, name)
Creates a error file name from the given name, including the details of this CU run, e.g.		 source code
 
getMultiframeIDs(self, dateVersStr)
Get the multiframeIDs from FlatFileLookUp for a given dateVersStr.		 source code
Utilities.Ratings
getProcessOrder(self, progIDs, include=['all'], progOrder=[])
Process programmes in given order, append/include all others from the list of all programme IDs.		 source code
 
makeSysCmd(self, serverName, cmnd) source code
 
removeWorkingDir(self)
Remove the temporary working directory.		 source code
 
run(self)
Run the curation session.		 source code
list(str)
runSysCmd(self, cmd, showErrors=True, isVerbose=False)
Run a system command.		 source code
 
setupDetectionTable(self, surveyName, monthSet, ReDo) source code
 
testForProgs(self, obsName)
Test if programme is one of the main VISTA surveys.		 source code
int
translateProgID(self, pIDkeys, progTranslation, knownTranslations={})
Translates a programme ID string into a programme ID number.		 source code
 
writeErrorFiles(self, notIngList)
Writes the list of not ingestable files into a list in the corrupt directory for further investigation.		 source code
 
writeIngestList(self)
Writes the list of FITS files to ingest to file "filelist.dat" in the work directory in a 'imagefile catfile num' format for the C++ code.		 source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]
list
_parseCuNums(cus)
Parse the given list of CU strings into a list of cuNum, cuEventID and serverName.		 source code
 
createFilePrefix(cuNum, cuEventID, hostName, database)
Create a file prefix used for the ingest log and the data files.		 source code
Class Variables [hide private]
  beginDateDef = 10000101
  endDateDef = 99991231
  reqWorkDir = None
hash(x)
  keepWorkDir = False
hash(x)
  _defComment = 'A useful comment'
Default comment placeholder.
  _fileList = None
File list.
  _isDbLocked = False
Does a database lock file exist?
  _numSessions = 0
Counter to track number of concurrent sessions.
  sysc = IngCuSession.sysc
System constants for this CuSession.
Instance Variables [hide private]
  _dateList
Set of full paths to date directories of files being ingested.
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, cuNum=-9999, curator='rsc', comment='A useful comment', reqWorkDir=False, keepWorkDir=False, database='WSA', userName='ldservro', autoCommit=False, isTrialRun=False)
(Constructor)

source code 

Initialises data members for all database state flags and details for this curation session. Writes a database lock and opens a connection. Also initiates a Programme table object.

Parameters:
  • cuNum (int) - Curation task number.
  • curator (str) - Name of curator.
  • comment (str) - Descriptive comment as to why curation task is being performed.
  • reqWorkDir (bool) - If True, a working directory will be created.
  • keepWorkDir (bool) - Don't remove working directory.
  • database (str) - Name of the database to connect to.
  • userName (str) - Optionally override default database username.
  • autoCommit (bool) - If True, do not auto-commit transactions. This speeds up transactions through minimal logging.
  • isTrialRun (bool) - If True, do not perform database modifications, just print the SQL statement to the terminal.
Overrides: object.__init__

_createProgrammeTranslation(self)

source code 

Create the translation dictionary of programme identification strings to programme ID numbers for all programmes. It contains associations between data flow system (DFS) programme identification strings (ie. as supplied in ingest FITS files) and the WSA programme unique identification numbers.

To Do: This method wouldn't be needed if a ProgrammeTable object always existed and had an equivalent method to peform this translation. Hence, the ProgrammeTable object would replace the dictionary here and there wouldn't be a need to query the database again.

_getNextCuEventID(self)

source code 

Obtain the next available cuEventID. Whilst putting a placeholder row into the ArchiveCurationHistory table to reserve this cuEventID, preventing it from being hijacked by a parallel process.

_getNextID(self, attrName, tableName, where='')

source code 

Obtain next available ID number in the db for a given ID attribute.

Parameters:
  • attrName (str) - Name of ID attribute.
  • tableName (str) - Name of the table where attribute resides.
  • where (str) - Optional SQL WHERE clause.
Returns: int
Next available ID number.

_historyCommentPrefix(self, success, rolledBack)

source code 

Prefix for the comment in the ArchiveCurationHistory table.

Parameters:
  • success (int) - The success status of the CU.
  • rolledBack (int) - Flag to indicate ingest roll back.
Returns: str
Prefix for the comment in the ArchiveCurationHistory table.

_parseCuNums(cus)
Static Method

source code 

Parse the given list of CU strings into a list of cuNum, cuEventID and serverName.

Returns: list
List of tuples characterising the CU events to be ingested.

_prepareDbLock(self, maxTries=100)

source code 

Writes database lock. If lock already present then pause and wait for database to unlock.

_updateArchiveCurationHistory(self, row)

source code 

Adds an entry for this curation session to the ArchiveCurationHistory database table.

Parameters:
  • row (list) - List of entries for the ArchiveCurationHistory table.

_updateACHComment(self, newComment, cuEventID)

source code 

Updates the comment of an existing entry in ArchiveCurationHistory.

Parameters:
  • newComment (str) - The new comment.
  • cuEventID (str) - The cuEventID of the event which is to be updated.

_updateProgrammeCurationHistory(self, row)

source code 

Adds an entry for this curation session to the ProgrammeCurationHistory database table.

Parameters:
  • row (list) - List of entries for the ProgrammeCurationHistory table.

attributesFromArguments(self, argumentNames, argDict, prefix='', types=None)

source code 

Initialize automatically instance variables from __init__ arguments. If types dictionary is given, translate commandline str arguments into correct types.

Parameters:
  • argumentNames (list[str]) - List of arguments of the class to be initialized.
  • argDict (dict{arg:value}) - Dictionary of arguments and their values.
  • prefix (str) - Prefix to be given to the argument names, eg. '_'
  • types (dict{arg:type}) - Dictionary of arguments and their types.

createFileList(self, fileListPath=None)

source code 

Create a list of FITS files to be ingest into the database from the list supplied in the file at the given path. The list is stored in the _fileList member variable as a pair of image and catalogue file paths and also written to a file by writeIngestList() for the C++ code.

Parameters:
  • fileListPath (str) - Full path to the file containing the list of FITS files to ingest, as an optional alternative to the command-line supplied option, xferLog.

createFilePrefix(cuNum, cuEventID, hostName, database)
Static Method

source code 

Create a file prefix used for the ingest log and the data files.

Parameters:
  • cuNum (int) - The number of the CU.
  • cuEventID (int) - The cuEventID of this CU run.
  • hostName (str) - The host name where the CU is running.
  • database (str) - The database where the data is ingested to.

createLogFileName(self, cuEventIDs=None)

source code 

Create the standard log file name for the current cuEventID or the given list of CU event IDs.

Parameters:
  • cuEventIDs (list(int)) - Optional list of CU eventIDs to record.
Returns: str
A filename for the log.

createWorkingDir(self)

source code 

Create a temporary working directory. Directories will have a 4 digit extension number for multiple tmpdirs created by one top level script calling different CUs.

Returns: str
Path to the working directory.

getProgramIDs(self, fileListPath)

source code 

Determines the full set of programmes for a given list of FITS files.

Parameters:
  • fileListPath (str) - Full path to the file listing all FITS files for which their programmes should be determined.
Returns: dict(str:int), dict(str:str)
A dictionary of programme IDs referenced by FITS file name, and a dictionary of new project DFS ID strings referenced by FITS file name.

getErrorFileName(self, name)

source code 

Creates a error file name from the given name, including the details of this CU run, e.g. database, cuEventID.

Parameters:
  • name (str) - Unique name for this type of error file.
Returns: str
Full file name for error file.

getProcessOrder(self, progIDs, include=['all'], progOrder=[])

source code 

Process programmes in given order, append/include all others from the list of all programme IDs.

Parameters:
  • progIDs (list(int)) - List of all (used) programme IDs.
  • include (list(str)) - List of project names to be processed.
  • progOrder (list(str)) - Ordered list of programme names, eg. [las,gps] or [las,others,gps].
Returns: Utilities.Ratings
Rating enhanced dictionary.

To Do: Will become part of DataFactory.ProgrammeTable when programme translation object moved there.

runSysCmd(self, cmd, showErrors=True, isVerbose=False)

source code 

Run a system command.

Parameters:
  • cmd (str) - The command to run.
  • isVerbose (bool) - If True, print error messages.
Returns: list(str)
List of lines returned.

translateProgID(self, pIDkeys, progTranslation, knownTranslations={})

source code 

Translates a programme ID string into a programme ID number.

Parameters:
  • pIDkeys (list) - List of Programme ID strings. For VISTA the order is: OBS PROG ID, OBS NAME, DPR CATG
  • progTranslation (dict(str:int)) - A dictionary of upper-case programme identification string keys and programme IDs values.
  • knownTranslations (dict(str:int)) - A dictionary of programme identification keys and dfsIDStrings.
Returns: int
Programme ID number. Default int value if unknown programme.

To Do: Tie this up with DataFactory.ProgrammeTable somehow, so that progTranslation isn't required.

writeErrorFiles(self, notIngList)

source code 

Writes the list of not ingestable files into a list in the corrupt directory for further investigation.

Parameters:
  • notIngList (list(str)) - List of files that cannot be ingested.

   Home       Trees       Indices       Help   
WSA
Generated by Epydoc 3.0.1 on Mon Sep 8 15:46:52 2014 http://epydoc.sourceforge.net