Home       Trees       Indices       Help   
WSA
Package invocations :: Package cu1 :: Module cu1Transfer :: Class CASUTransfer
[hide private]

Class CASUTransfer

source code

Contains methods and functions related to CU1 transfers.

Instance Methods [hide private]
 
__init__(self, scpMainDir=None, scpThreads=1, checkTimeFlag=False, checkMd5Flag=False, UKLight=True, forceTransfer=False, deprecationMode='mv', database=None, outPrefix=None, reproMethod='', reproVersStr=None, ffluOnly=False, onlyHeaderTransfer=False, isTestRun=False)
x.__init__(...) initializes x; see help(type(x)) for signature		 source code
 
initTransfer(self, runDate)
Initialise the transfer.		 source code
tuple(str, int, StringIO, list)
transfer(self, runDate, xferList=[])
Main script to transfer data from CASU.		 source code
dict
getListings(self, runDate, xferList=[], checkSR=True)
Get the remote and local listing (remote first to create same dirs here).		 source code
int
transferFiles(self, fileList, totalDLSize, casuStats, hereStats, casuDict, outDirs)
Transfer the files in the given list.		 source code
StringIO
_writeLoggerMessages(self)
Write log messages to logText.		 source code
 
_cleanUp(self, rmDirs=[])
Removes the lock file, closes the download log file.		 source code
dict
_dictFromList(self, list)
Build dict from file list.		 source code
tuple(int, float)
_getTimeDiff(self, dateTimeTpl)
Returns: Time difference tuple: (days, seconds)		 source code
dict(str:list), dict(str: list(int, int, int))
_readDirList(self, aListFile, lsNames=['', '', ''], lsOpt='')
Read directory listing from file and transform it into internal list.		 source code
list
_cleanupTransferList(self, tflin)
Scans through the transfer list to see which files match the delete conditions and deletes them.		 source code
fileObj
_createCasuListFile(self, copyDict)
Create a listing of given directories into a file at CASU and copy the file into the sys dir.		 source code
dict(str: list(str, str))
_getTransferableDirs(self, runDate, checkSR=True)
Create a dictionary of directories which are ready to copy and not yet successfully downloaded.		 source code
list, int, dict
_listRemoteDir(self, copyDict, lsNames, lsOpt, xferList=[])
List remote directory 'scpMainDir'.		 source code
list(list(str))
_buildLoadList(self, fileList, totalSize)
Build a list of #-of-threads sublists.		 source code
list, int
_checkLocalExist(self, casuDict, hereDict, casuStats, hereStats)
Check if remote file is in local dict.		 source code
dict(str: str)
_checkNewDirs(self, copyDict, casuSizes)
Check if there is enough space to transfer the whole dir, otherwise change outdir to a free disk.		 source code
int
_getDirsize(self, dirPath)
Get the size of a directory.		 source code
dict(str: str)
_getLatestAccess(self, diskList, subDir='ingest/fits')
Create a rating dict for the given disks containing date directories rated by last access time.		 source code
list, int, dict
_listHomeDir(self, copyDict, lsNames, xferList=[])
List home directory 'fitsDir'.		 source code
 
_mvDeprecatedFiles(self, fileList, deprMode='mv')
Move files in fileList to the directory for deprecated files.		 source code
 
_calcTransferRate(self, startTime, endTime, totalSize)
Calculate the transfer rate.		 source code
dict
_downloadFiles(self, fileList, outDirDict, totalSize)
Download files.		 source code
list(str)
_missingFileCheck(self, fileList)
Check for not transfered files.		 source code
 
_removeScpScripts(self)
Remove the previously made scp scripts.		 source code
 
_threadScpScripts(self)
Run the scp scripts simultaneously, but wait until each has finished.		 source code
dict(str: str)
_writeScpScripts(self, fileList, outDirs)
Write shell scp script based on the file list comparison.		 source code
 
_writeRemSuccess(self, dirStats)
Write successful_downdload file to Cambridge dirs.		 source code
 
_writeRemSuccessToCasu(self, dirStats)
Write successful_downdload file to Cambridge dirs.		 source code
 
_mvToFin(self, fileList)
Move files in fileList to final directory.		 source code
list
_writeLocSuccess(self, dirStats, outDirs)
Write successful_downdload file to local dirs and create transfer list.		 source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Class Variables [hide private]
  maxRetries = 10
  remLsOpt = '.fit'
  remExclPattern = ['dqc', 'xsky', 'chan', 'bpm', 'tmp', 'orig',...
  transferMethod = 'scp'
  sysc = SystemConstants()
  verbose = 0
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, scpMainDir=None, scpThreads=1, checkTimeFlag=False, checkMd5Flag=False, UKLight=True, forceTransfer=False, deprecationMode='mv', database=None, outPrefix=None, reproMethod='', reproVersStr=None, ffluOnly=False, onlyHeaderTransfer=False, isTestRun=False)
(Constructor)

source code 

x.__init__(...) initializes x; see help(type(x)) for signature

Parameters:
  • scpMainDir (str) - Disk at CASU where the data is stored.
  • scpThreads (int) - Number of scp transfer threads.
  • checkTimeFlag (bool) - Determines if the timestamp of existing files should be checked against the file at CASU.
  • checkMd5Flag (bool) - Check the md5 checksums at CASU and WFAU.
  • UKLight (bool) - Use UKLight instead of JANET.
  • ffluOnly (bool) - Don't transfer but update FlatfileLookup table.
  • forceTransfer (bool) - Force the data transfer.
  • deprecationMode (str) - Modus for deprecating files.
  • database (str) - The database.
  • onlyHeaderTransfer (bool) - Transfer only FITS headers.
  • outPrefix (str) - Prefix of output file(s).
  • reproMethod (str) - Method for reprocessed data: s::suffix (CASU datedir suffix) or v::version.
  • reproVersStr (str) - Version number of reprocessed data.
  • isTestRun (bool) - If True, do not perform database modifications.
Overrides: object.__init__

transfer(self, runDate, xferList=[])

source code 

Main script to transfer data from CASU.

Parameters:
  • runDate (int) - The date for the transfer.
  • xferList (list) - Optional list containing files to be downloaded.
Returns: tuple(str, int, StringIO, list)
xferlogfile name, download status, log text, list of transferred dates.

getListings(self, runDate, xferList=[], checkSR=True)

source code 

Get the remote and local listing (remote first to create same dirs here).

Parameters:
  • runDate (int) - The date for the transfer.
  • xferList (list) - Optional list containing files to be downloaded.
  • checkSR (bool) - Check if the SUCCESSFULLY_READ tag is set.
Returns: dict
dictionaries containing data and stats for directories at CASU and WFAU.

transferFiles(self, fileList, totalDLSize, casuStats, hereStats, casuDict, outDirs)

source code 

Transfer the files in the given list.

Parameters:
  • fileList (list) - The list of files to transfer.
  • totalDLSize (int) - The estimated total size to transfer.
  • casuStats (int) - Statistics of CASU directories.
  • hereStats (int) - Statistics of WFAU directories.
  • casuDict (dict) - Dictionary of CASU data.
  • outDirs (dict(str: str)) - Dictionary of date-version strings and their download directories.
Returns: int
transfer status

_writeLoggerMessages(self)

source code 

Write log messages to logText.

Returns: StringIO
Log text string object.

_cleanUp(self, rmDirs=[])

source code 

Removes the lock file, closes the download log file.

Parameters:
  • rmDirs (list(str)) - List of directories that should be removed.

_dictFromList(self, list)

source code 

Build dict from file list.

Parameters:
  • list (list) - File name list (fileName, path, values).
Returns: dict
Dictionary of full file paths and values.

_getTimeDiff(self, dateTimeTpl)

source code 
Parameters:
  • dateTimeTpl ((str, str)) - Date time tuple: ('yyyy-mm-dd', 'hh:mm:ss(.ss)')
Returns: tuple(int, float)
Time difference tuple: (days, seconds)

_readDirList(self, aListFile, lsNames=['', '', ''], lsOpt='')

source code 

Read directory listing from file and transform it into internal list.

Parameters:
  • aListFile (File class object) - File containing the file system listings.
  • lsNames (list(str, str, str)) - Names of the locked and the successfully read file.
  • lsOpt (str) - listing option (regexp for name.find).
Returns: dict(str:list), dict(str: list(int, int, int))
Dict of directories, and a dictionary of directory statistics, i.e. [locked,used,success].

_cleanupTransferList(self, tflin)

source code 

Scans through the transfer list to see which files match the delete conditions and deletes them.

Parameters:
  • tflin (list) - Transfer file list.
Returns: list
Updated list of transferable files.

_createCasuListFile(self, copyDict)

source code 

Create a listing of given directories into a file at CASU and copy the file into the sys dir.

Parameters:
  • copyDict (dict) - Dictionary of transferable dates and CASU paths.
Returns: fileObj
File listing the directories at CASU.

_getTransferableDirs(self, runDate, checkSR=True)

source code 

Create a dictionary of directories which are ready to copy and not yet successfully downloaded. It also includes the version number as a combination of <date>_v<#>.

Parameters:
  • runDate (int) - Transfer date.
  • checkSR (bool) - Check if the SUCCESSFULLY_READ tag is set.
Returns: dict(str: list(str, str))
Dictionary of CASU path and date-version number strings for each date.

_listRemoteDir(self, copyDict, lsNames, lsOpt, xferList=[])

source code 

List remote directory 'scpMainDir'.

Parameters:
  • copyDict (dict) - Dictionary of transferable directories.
  • lsNames (list(str, str, str)) - Names of the locked and the successfully read file.
  • lsOpt (str) - listing option (regexp for name.find).
  • xferList (list) - Optional list containing files to be downloaded.
Returns: list, int, dict
List of directories at WFAU, size of directories at WFAU, and a dictionary containing directory statistics.

_buildLoadList(self, fileList, totalSize)

source code 

Build a list of #-of-threads sublists.

The filelist is split into #-of-threads roughly equal bytesized sublists.

Parameters:
  • fileList (list(str)) - List of filenames.
  • totalSize (int) - Total size of files in filelist.
Returns: list(list(str))
List of sublists of files to download in different threads.

_checkLocalExist(self, casuDict, hereDict, casuStats, hereStats)

source code 

Check if remote file is in local dict. If so, deprecate.

Parameters:
  • casuDict (dict) - Dictionary of CASU data.
  • hereDict (dict) - Dictionary of WFAU data.
  • casuStats (int) - Statistics of CASU directories.
  • hereStats (int) - Statistics of WFAU directories.
Returns: list, int
file list of transferrable files and their total size.

_checkNewDirs(self, copyDict, casuSizes)

source code 

Check if there is enough space to transfer the whole dir, otherwise change outdir to a free disk.

Parameters:
  • copyDict (dict(str:list(str, str))) - Dictionary containing the CASU path and the date-version number string for every date.
  • casuSizes (dict(str: int)) - Dictionary of directory names and their sizes.
Returns: dict(str: str)
Dictionary of date-version strings and their download directory.

_getDirsize(self, dirPath)

source code 

Get the size of a directory.

Parameters:
  • dirPath (str) - Full path to a directory.
Returns: int
Size in kilobytes.

_getLatestAccess(self, diskList, subDir='ingest/fits')

source code 

Create a rating dict for the given disks containing date directories rated by last access time.

Parameters:
  • diskList (list(str)) - List of raid file system disks.
  • subDir (str) - Subdirectory to check.
Returns: dict(str: str)
Ratings dictionary of disks and their latest access time.

_listHomeDir(self, copyDict, lsNames, xferList=[])

source code 

List home directory 'fitsDir'.

Parameters:
  • copyDict (dict) - Dictionary of transferable directories.
  • lsNames (list(str, str, str)) - Names of the locked and the successfully read file.
Returns: list, int, dict
Dict of directories at WFAU, their size, and a dictionary containing directory statistics.

_mvDeprecatedFiles(self, fileList, deprMode='mv')

source code 

Move files in fileList to the directory for deprecated files.

Parameters:
  • fileList (list(str)) - list of filenames(file, datedir, path(incl. datedir))
  • deprMode (str) - Mode for handling the deprecated files: move ('mv'), remove ('rm'), leave in place ('ip')

_calcTransferRate(self, startTime, endTime, totalSize)

source code 

Calculate the transfer rate.

Parameters:
  • startTime (time object) - Begin of download.
  • endTime (time object) - End of download.
  • totalSize (int) - Total size from download scripts.

_downloadFiles(self, fileList, outDirDict, totalSize)

source code 

Download files.

Parameters:
  • fileList (list) - File list of transferable files.
  • outDirDict (dict(str: str)) - Dictionary of date-version strings and their download directories.
  • totalSize (int) - Total size of transfer.
Returns: dict
Dictionary of transferred files.

_missingFileCheck(self, fileList)

source code 

Check for not transfered files.

Parameters:
  • fileList (list) - list of filenames(file, datedir, path(incl. datedir))
Returns: list(str)
List of missing files.

_writeScpScripts(self, fileList, outDirs)

source code 

Write shell scp script based on the file list comparison.

Parameters:
  • fileList (list(str)) - List of transferable file names.
  • outDirs (dict(str: str)) - Dictionary of date-version strings and their download directories.
Returns: dict(str: str)
Dictionary of downloaded file names and their paths.

_writeRemSuccess(self, dirStats)

source code 

Write successful_downdload file to Cambridge dirs.

Parameters:
  • dirStats (dict(str:list(int))) - Directory statistics.

_writeRemSuccessToCasu(self, dirStats)

source code 

Write successful_downdload file to Cambridge dirs.

Parameters:
  • dirStats (dict(str:list(int))) - Directory statistics.

_mvToFin(self, fileList)

source code 

Move files in fileList to final directory.

Parameters:
  • fileList (list(str)) - List of full path file names

_writeLocSuccess(self, dirStats, outDirs)

source code 

Write successful_downdload file to local dirs and create transfer list.

Parameters:
  • dirStats (dict) - Directory statistics.
  • outDirs (dict(str: str)) - Dictionary of date-version strings and their download directories.
Returns: list
List of successfully transferred files.

Class Variable Details [hide private]

remExclPattern

Value:
['dqc',
 'xsky',
 'chan',
 'bpm',
 'tmp',
 'orig',
 'jrf',
 'mos',
...

   Home       Trees       Indices       Help   
WSA
Generated by Epydoc 3.0.1 on Mon Sep 8 15:46:45 2014 http://epydoc.sourceforge.net