Home       Trees       Indices       Help   
WSA
Package invocations :: Package cu0 :: Module DataDaemon :: Class DataDaemon
[hide private]

Class DataDaemon

source code

Top level function to invoke ingests.

Nested Classes [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: IngCuError

Instance Methods [hide private]
 
__init__(self, info=False, curator='rsc', database='WSA', serverNames=set(['ini', 'menkaure', 'neferefre', 'sahure', 'shepseskaf', '..., janet=False, cuNums=False, casuDisks=False, dateRegExpStr=False, comment='A useful comment')
Initialises data members for all database state flags and details for this curation session.		 source code
 
getCron(self, serverList)
Get the crontab information from all servers.		 source code
dict(int:set(str))
getCUs(self)
Get the CUed information from all fits directories.		 source code
 
getDBOverview(self, dbDict, archive=None)
Print a list of all DBs on given server group (dbload or dbpub).		 source code
list(str)
getDateList(self, begin, end, version)
Create a dateVersStr list.		 source code
dict(str:list(str))
getPendingIngestDates(self, cuNum)
Get the DateVersStr from the ingest log files.		 source code
list((st, str))
getPsStats(self, serverName, searchTerms)
Get stats from ps command on given server.		 source code
dict(str:dict(str:list))
getServerJobs(self, serverDict)
Get jobs ran by scos on given servers.		 source code
dict(str:float)
getServerLoad(self)
Get the mean load average of all servers.		 source code
list(str)
makeDateVersStrList(self, dirList, versDict=None)
Make a list of dateVersStrs from a list of date directories.		 source code
list(str)
readCasuSr(self)
List successfully_read dirs at CASU.		 source code
list(str)
readCasuOtc(self)
List OK_TO_COPY dirs at CASU.		 source code
dict(str:str)
readCasuVers(self)
List version numbers at CASU.		 source code
 
runContinuousIngests(self, cuNums, curator, comment) source code
 
purgeShare(self, mode, archive, isIngest=False)
Cleanup the samba file shares.		 source code
 
deleteFiles(self, serverName, filesPath, isTest, archive, verbose=False)
Remove files from given server and path.		 source code
 
runMonitor(self, semester, excludedCUs, remStats, versions)
Run the createCalendar script to update the monitor pages.		 source code
 
svnupdate(self, updateDir, serverList)
Update sandboxes on servers given in the serverList.		 source code
 
assertComp(self, sandbox, serverList)
Assert that the compilation of sandboxes on servers is up-to-date.		 source code
 
compileServer(self, branch, serverList)
Compile code on servers given in the serverList.		 source code
list(str), list(str)
createCU0Run(self, dataBuilderJobs)
Create a list of CU0 runs, ie.		 source code
list(str), list(str)
createCU1Run(self, dataBuilderJobs)
Create a list of CU1 runs, ie.		 source code
list(str), list(str)
createCU2Run(self, cuedDirs, dataBuilderJobs)
Create a list of CU2 runs, ie.		 source code
list(str), list(str)
createCU3Run(self, cuedDirs, dataBuilderJobs)
Create a list of CU3 runs, ie.		 source code
list(str), list(str)
createCU4Run(self, cuedDirs, dataBuilderJobs)
Create a list of CU4 runs, ie.		 source code
 
runCUs(self, cuRunDict)
Print out commands to run the CUs on the servers.		 source code
 
run(self)
Do all the stuff needed.		 source code
 
removeFileSuffixes(self, suffix, fileTags) source code

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: __del__, attributesFromArguments, createFileList, createLogFileName, createMonthlyDetSchema, createWorkingDir, getDetTable, getErrorFileName, getMultiframeIDs, getProcessOrder, getProgramIDs, makeSysCmd, removeWorkingDir, runSysCmd, setupDetectionTable, testForProgs, translateProgID, writeErrorFiles, writeIngestList

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _connectToDb, _createProgrammeTranslation, _disconnectFromDb, _getNextCuEventID, _getNextID, _historyCommentPrefix, _onException, _onRun, _prepareDbLock, _updateACHComment, _updateArchiveCurationHistory, _updateProgrammeCurationHistory

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Static Methods [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: createFilePrefix

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _parseCuNums

Class Variables [hide private]
  _defServerNames = set(['ini', 'menkaure', 'neferefre', 'sahure...
  _defCuNums = [0, 1, 2, 3, 4]
  _defDateRegExpStr = '20[01][90123].*'
  builderloc = 'DataBuilder.py'
  ingesterloc = ['IngestCUFiles.py', 'DataDaemon.py -I']
  otherculoc = 'cu'
  othercodeloc = ['AutoCurate.py', 'FinaliseFlatFileIngest.py', ...
  casuDisk = None
hash(x)
  maxLoad = 3.0
  fullCommand = False
hash(x)
  reqWorkDir = False
hash(x)
  waitMins = 10
  force = False
hash(x)
  inclTestDb = False
hash(x)
  omitObjIDUpdate = False
hash(x)

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession: beginDateDef, endDateDef, keepWorkDir, sysc

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _defComment, _fileList, _isDbLocked, _numSessions

Instance Variables [hide private]

Inherited from wsatools.DbConnect.IngCuSession.IngCuSession (private): _dateList

Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, info=False, curator='rsc', database='WSA', serverNames=set(['ini', 'menkaure', 'neferefre', 'sahure', 'shepseskaf', '..., janet=False, cuNums=False, casuDisks=False, dateRegExpStr=False, comment='A useful comment')
(Constructor)

source code 

Initialises data members for all database state flags and details for this curation session. Writes a database lock and opens a connection. Also initiates a Programme table object.

Parameters:
  • casuDisks (str) - Disk at CASU where the data is stored.
  • comment (str) - Descriptive comment as to why curation task is being performed.
  • cuNums (list[int]) - Curation task numbers.
  • curator (str) - Name of curator.
  • database (str) - Name of the database to connect to.
  • info (bool) - Only print status of servers, don't calculate usage.
  • janet (bool) - Use JANET instead of UKLight.
  • serverNames (set(str)) - List of servers to check, or exclude server with preceding '-'.
  • dateRegExpStr (str) - Regular expression for the dates.
Overrides: object.__init__

To Do: This class constructor does not respect its parent class's constructor, so the class hierarchy needs reworking.

getCron(self, serverList)

source code 

Get the crontab information from all servers.

Parameters:
  • serverList (list(str)) - List containing servers.

getCUs(self)

source code 

Get the CUed information from all fits directories.

Returns: dict(int:set(str))
Dictionary of CU numbers and associated fits dirs.

getDBOverview(self, dbDict, archive=None)

source code 

Print a list of all DBs on given server group (dbload or dbpub).

Parameters:
  • dbDict (dict(str:list(str))) - Dictionary of DB informations.

getDateList(self, begin, end, version)

source code 

Create a dateVersStr list.

Parameters:
  • begin (int) - Begin date.
  • end (int) - End date.
  • version (str) - Version number.
Returns: list(str)
List of dateVersStr.

getPendingIngestDates(self, cuNum)

source code 

Get the DateVersStr from the ingest log files.

Parameters:
  • cuNum (int) - CU number.
Returns: dict(str:list(str))
Dictionary containing dates per cuID.

getPsStats(self, serverName, searchTerms)

source code 

Get stats from ps command on given server.

Parameters:
  • searchTerms (list(str)) - List of jobs to search for.
Returns: list((st, str))
List of (job, etime) pairs.

getServerJobs(self, serverDict)

source code 

Get jobs ran by scos on given servers.

Parameters:
  • serverDict (dict) - Rated dictionary containing servers.
Returns: dict(str:dict(str:list))
Dictionary of dictionaries of CU statistics on each server.

getServerLoad(self)

source code 

Get the mean load average of all servers.

Returns: dict(str:float)
Rated dictionary of servers by their laod average.

makeDateVersStrList(self, dirList, versDict=None)

source code 

Make a list of dateVersStrs from a list of date directories.

Parameters:
  • dirList (list(str)) - List containig directory paths.
  • versDict (Dict(str:str)) - Dictionary of dates and their version number.
Returns: list(str)
List of dateVersStrs.

readCasuSr(self)

source code 

List successfully_read dirs at CASU.

Returns: list(str)
List of directories that are successfully read.

readCasuOtc(self)

source code 

List OK_TO_COPY dirs at CASU.

Returns: list(str)
List of directories that are OK to copy.

readCasuVers(self)

source code 

List version numbers at CASU.

Returns: dict(str:str)
Dictionary of directories and their version number.

purgeShare(self, mode, archive, isIngest=False)

source code 

Cleanup the samba file shares.

Parameters:
  • mode (str) - Either check or delete ingested files.
  • isIngest (bool) - Is this a check during continuous ingests?

deleteFiles(self, serverName, filesPath, isTest, archive, verbose=False)

source code 

Remove files from given server and path.

Parameters:
  • archive (str) - The archive for which the files are deleted.
  • filesPath (str) - Path to checked dir.
  • isTest (bool) - If true, don't delete files.
  • serverName (str) - Name of server to be checked.
  • verbose (bool) - If true add additional log messages.

runMonitor(self, semester, excludedCUs, remStats, versions)

source code 

Run the createCalendar script to update the monitor pages.

Parameters:
  • semester (str) - Semester to be monitored.
  • excludedCUs (list(str)) - List of excluded CUs.
  • remStats (str) - Remove stats for these CUs
  • versions (str) - Versions to be monitored.

svnupdate(self, updateDir, serverList)

source code 

Update sandboxes on servers given in the serverList.

Parameters:
  • updateDir (str) - Directory to update.
  • serverList (list(str)) - List of servers.

assertComp(self, sandbox, serverList)

source code 

Assert that the compilation of sandboxes on servers is up-to-date.

Parameters:
  • sandbox (str) - Sandbox to check.
  • serverList (list(str)) - List of servers.

compileServer(self, branch, serverList)

source code 

Compile code on servers given in the serverList.

Parameters:
  • branch (str) - The branch to be compiled.
  • serverList (list(str)) - List of servers.

createCU0Run(self, dataBuilderJobs)

source code 

Create a list of CU0 runs, ie. ingests.

Parameters:
  • dataBuilderJobs (disct(str:str)) - Dictionary of jobs already running.
Returns: list(str), list(str)
List of ingest runs, list of cuNum/cueventIDs to ingest.

createCU1Run(self, dataBuilderJobs)

source code 

Create a list of CU1 runs, ie. transfers.

Parameters:
  • dataBuilderJobs (disct(str:str)) - Dictionary of jobs already running.
Returns: list(str), list(str)
List of ingest runs, list of dates to process.

createCU2Run(self, cuedDirs, dataBuilderJobs)

source code 

Create a list of CU2 runs, ie. JPG creation.

Parameters:
  • cuedDirs (dict(int:list(str))) - Dictionary of already CUed directories.
  • dataBuilderJobs (disct(str:str)) - Dictionary of jobs already running.
Returns: list(str), list(str)
List of ingest runs, list of dates to process.

createCU3Run(self, cuedDirs, dataBuilderJobs)

source code 

Create a list of CU3 runs, ie. metadata ingest.

Parameters:
  • cuedDirs (dict(int:list(str))) - Dictionary of already CUed directories.
  • dataBuilderJobs (disct(str:str)) - Dictionary of jobs already running.
Returns: list(str), list(str)
List of ingest runs, list of dates to process.

createCU4Run(self, cuedDirs, dataBuilderJobs)

source code 

Create a list of CU4 runs, ie. catalogue data ingest.

Parameters:
  • cuedDirs (dict(int:list(str))) - Dictionary of already CUed directories.
  • dataBuilderJobs (disct(str:str)) - Dictionary of jobs already running.
Returns: list(str), list(str)
List of ingest runs, list of dates to process.

runCUs(self, cuRunDict)

source code 

Print out commands to run the CUs on the servers.

Parameters:
  • cuRunDict (dict(str:str)) - Dictionary containing commands for CUs.

run(self)

source code 

Do all the stuff needed.

Overrides: wsatools.DbConnect.IngCuSession.IngCuSession.run

Class Variable Details [hide private]

_defServerNames

Value:
set(['ini',
     'menkaure',
     'neferefre',
     'sahure',
     'shepseskaf',
     'thoth',
     'unas',
     'userkaf'])

othercodeloc

Value:
['AutoCurate.py',
 'FinaliseFlatFileIngest.py',
 'DataDaemon.py -M',
 'NonSurveyRelease.py',
 'UpdateTilePawPrintTables.py']

   Home       Trees       Indices       Help   
WSA
Generated by Epydoc 3.0.1 on Mon Sep 8 15:46:45 2014 http://epydoc.sourceforge.net