To Do List

To Do in helpers.EsoRelease.EsoRelease.getTilePosDict Position angle?

To Do in helpers.EsoRelease.EsoRelease.outgestSources (priOrSec = 0 || priOrSec=frameSetID) as primary_source VVV - simpler for DR2? Be careful. Just primary sources selected?

To Do in helpers.FilterTableCalculator.FilterCalculator make more sophisticated. Transmission curves read from FITS files with instructions on how to use encapsulated in header values.

To Do in helpers.FilterTableCalculator.FilterCalculator.rebinFilterData smoothing, lsq

To Do in helpers.SyncDb Due to the way indices are defined this may be faster if we were able to make the primary keys the real primary keys, so instead of just querying MFD on multiframeID, do it on the seemingly unnecessary multiframeID and extNum. However, not sure how to do within a NOT IN (SELECT attr FROM...) context. Optimise date range updates.

To Do in helpers.TransposeCsv Make use of utils.extractColumns() when then gets CSV support.

To Do in helpers.UpdateDetectionTables The calcNParts() code uses common methods and member variables that should be encapsulated in their own class, that both the DeblendParamFixer and AProf1Fixer classes should derive from separately from the generic DetectionUpdater class?

To Do in helpers.UpdateTableData Needs to be upgraded to become a CuSession to save curation history.

To Do in invocations.cu0.DataDaemon.DataDaemon.__init__ This class constructor does not respect its parent class's constructor, so the class hierarchy needs reworking.

To Do in invocations.cu0.FinaliseFlatFileIngest.FileHistoryDataFiller.updateFits when is this used

To Do in invocations.cu13.cu13.Cu13.runCasuExtractor Obsolete?

To Do in invocations.cu13.cu13.ProductDefinition.getDualProductID once CU9 is finished, get rid of this

To Do in invocations.cu16.cu16.NeighboursIngester This should no longer be a subclass of Ingester. Instead these functions should become methods of the Cu16 class and call the parent Ingester class directly.

To Do in invocations.cu19.cu19 Resolve complication over whether PTS/CAL should be considered non-surveys or not - it may well be easier to create survey like DBs for them (though there is the consideration of deprecation selections). Create a Cu19._parseSchemas() sub-routine to tidy the code up a bit. Provide Info message about largest table size. Automatic determination of initial file-group file sizes. See trac ticket:89 for implementation details.

To Do in invocations.cu21.NonSurveyRelease Have an option to prepare the release of just shallow or deep surveys (i.e. first stage run). However, this requires the script to do the ProgrammeBuilder setup first to determine whether it is shallow or deep, then the ProgrammeTable needs to test and skip if not the right type (without failure).

To Do in invocations.cu21.NonSurveyRelease.NonSurveyCurator This class borrows heavily on the CuSession design, so need to derive both CuSession and this class from a class that represents the common super set.

To Do in invocations.cu21.NonSurveyRelease.NonSurveyCurator.run This should become part of abstracted class at a level above CuSession.

To Do in invocations.cu3.cu3.Cu3.createProgrammeCsv It might make more sense to write the contents of self._progIDofName to the programmes.csv, which includes the merging of different types of programmes. This would save some code duplication in exmeta.

To Do in invocations.cu4.CreatePixCatList Make the programme list both database and user driven, obviating the need for a cu4programmes.list file.

To Do in invocations.cu6.cu6 The CU6 class or at least the class it uses should be polymorphic between the correlated and non-correlated synoptic cases spatial/time variations - fit function f(x,y) to each intermediate Recalibration changes. Best match - just nearest object. If motion fitted then come back and fix up BM later. Only a few objects. This way can add in new MFs to BM without reprocessing whole table.

To Do in invocations.cu8.cu8 CU8 *still* has issues with interrupts and not closing down properly may be sensible to give up with the multi-threading idea... Order FITS updates in disk order?? Reduce code replication, break up into functions further, document, improve calib table parser. Tidy up calibrateZeroPoint. Separate out FITS Writing code etc. into a FitsUtils.Writer class, for CU6 etc. Implement global recalibration solution.

To Do in invocations.cu9.cu9 Update to newer CU7 style way of working (e.g. enable ingest, logging, dropping indicies, updating tables, tiding up files etc.)

To Do in invocations.cu9.cu9.Cu9.createIlluminationTableList Move CU4 version to Astrometry?? and link to that.

To Do in invocations.cu9.cu9.Cu9.filterCat Do this in C++ code -- more efficient.

To Do in invocations.cu9.cu9.Cu9.runLDExtractor Parallelise - like CU13 Describe inputs, e.g. args. Add in ZP etc...

To Do in wsatools.Astrometry.convPixelToCartSky Would be more efficient if PyWCS performed multiple co-ordinate conversions for a given WCS value set, using ctypes.

To Do in wsatools.Astrometry.convPixelToSky Would be more efficient if PyWCS performed multiple co-ordinate conversions for a given WCS value set, using ctypes.

To Do in wsatools.Astrometry.convSkyToPixel Would be more efficient if PyWCS performed multiple co-ordinate conversions for a given WCS value set, using ctypes.

To Do in wsatools.Astrometry.pairObjects Make completely schema driven for both data types and units, basically the caller gets raIndex/raSize/raUnit from schema file and also passes raType, e.g. 'd' in this case. Can be made even more Pythonic and faster if we include the merge table in with this function, avoiding the need for index look-ups and reading/writing to binary files... assuming memory req. are OK.

To Do in wsatools.CLI Update design/documentation of classes in this module. Phase out use of getOptDef / getArgDef. This will result in a clear separation between classes that accept Python objects as arguments, and CLI objects return strings that are converted into Python objects by scripts. Would be nice if each argument could have a convertor as well as an isValOK method to perform the conversion on get. Tidy up indentation of option descriptions in help screen. Bring in an argument Type class heirachy (see trac ticket 42) to allow this module to perform automatic type conversions to Python types. Make consistent with optparse. Johann wants multiple examples on a help screen - this can be done but may lead to the development API becoming even more confusing! An alternative is to just initialise a CLI object with an additional text string containing whatever notes Johann wishes to add - only problem is it won't automatically be updated when arguments change which is the nice feature of the CLI module's help screen right now. Perhaps this is a good reason to leave these notes on a TWiki page?

To Do in wsatools.CSV Support dialects on write. Needs a substantial change to the current interface.

To Do in wsatools.DataFactory Move Table and View into a general Database module / package. Replace with metadata classes? Where a record is a metadata class whose attributes are created on the fly, rather than using a dictionary for referencing attributes. Usage documentation. Complement getAttr() with [], but beware of conflicts of use.

To Do in wsatools.DbConnect.CommonQueries.deprecateOldDetections Make this a general propagate deprecations function like the one used in the Automator? That way we don't get the double deprecation of WFAU reprocessed detections - first they are deprecated by CU4 with deprecation+128, then they are set to 255 by Automator.

To Do in wsatools.DbConnect.CommonQueries.getAllDBs Move into DbSession module as an independent function, and rename to getReleaseDbNames().

To Do in wsatools.DbConnect.CommonQueries.getAllFieldObs Make a method of a Stacker class that CU13 and CU6 derive from?

To Do in wsatools.DbConnect.CommonQueries.getAllFieldObsProd Make a method of a Stacker class that CU13 and CU6 derive from?

To Do in wsatools.DbConnect.CommonQueries.getGroupPositionCentres Is this function being used? CU9?

To Do in wsatools.DbConnect.CuSession Should the success flag be in ACH, and the PCH not updated at all if rolledback = True, in all cases?

To Do in wsatools.DbConnect.CuSession.CuSession.run Replace init and del with a context manager for database connections and locks, implemented as a with-block in this function.

To Do in wsatools.DbConnect.DbSession Make method keyword arguments consistent. The query() interface should be attrs, table, where; and this naming convention should be stuck to as much as possible in other methods. Should update() use attrs or entry instead of entryList? Cut down code duplication between updateEntries and deleteRows + possibly between updateEntries + update, ditto for deleteRows Retrospectively upgrade code using queryNumRows() to make use of new features if necessary. Upgrade code to make use of join feature, and upgrade documentation of query*() methods to reflect this. Upgrade code to make use of query() returning namedtuples. addColumn / dropColumn should be merged with add/drop constraint, and so be designed like createObject/dropObject. Password option?

To Do in wsatools.DbConnect.DbSession.Database.release Merge with mirror somehow.

To Do in wsatools.DbConnect.DbSession.DbSession.alterColumn Should become more schema driven together with addColumn().

To Do in wsatools.DbConnect.DbSession.DbSession.query Either replace arguments with one SelectSQL argument or: replicate SelectSQL.__init__() interface: selectStr becomes select, fromStr becomes table, whereStr becomes where - then remove code duplication by creating an SelectSQL object here.

To Do in wsatools.DbConnect.DbSession.DbSession.sharePath Deprecate!

To Do in wsatools.DbConnect.DbSession.DbSession.truncate Deprecate in favour of delete() without where clause? Always call truncate over delete from if not where clause, with the addition of a drop and add constraints?

To Do in wsatools.DbConnect.DbSession.Ingester Always parse indices on initialisation of Ingester. Can't drop indices at this stage as it should occur too soon. Then give ingestTable() a default option of dropIndices=True. Ingester can tell from the schema whether it should parse non-survey indices or not. Ingester itself could have a dropIndices option at initialisation with default of False, for IngIngester to overrule.

To Do in wsatools.DbConnect.DbSession.Ingester.ingestTable If not isOrdered and overWrite=True then we don't apply primary key until after ingest. OR even better - if not isOrdered try to create table without any constraints, if succeed, i.e. there is no existing table or overWrite=True, then apply primary key after ingest other don't but do check foreign keys. May make parts of NeighbourTableIngester redundant.

To Do in wsatools.DbConnect.DbSession.bulkCopy Merge Outgester and Ingester classes into one BulkTransfer class with individual outgest, ingest and copy functions? Then we can handle the tableSchema initialisation a bit more neatly, and put fileTag into the initialiser. Then codes can choose whether or not to pre-initialise.

To Do in wsatools.DbConnect.IngCuSession.IngCuSession.getProcessOrder Will become part of DataFactory.ProgrammeTable when programme translation object moved there.

To Do in wsatools.DbConnect.IngCuSession.IngCuSession.translateProgID Tie this up with DataFactory.ProgrammeTable somehow, so that progTranslation isn't required.

To Do in wsatools.DbConnect.IngIngester.CuIngest.deprecateOldFrames This function will eventually contain the contents of the SQL fragment, as a DB API update method.

To Do in wsatools.DbConnect.IngIngester.CuIngest.selectFiles Useful for FitsUtils?

To Do in wsatools.DbConnect.IngIngester.IngestLogger This class will become obsolete, probably by giving Logger a writeMode argument to the constructor, with a default of 'w'. Need to be sure that the log reset is OK too. If class stays it will be renamed and moved into Logger.py.

To Do in wsatools.DbConnect.Schema Reduce code replication in all parse*() objects Better exception handling on parsing - return file and line number, as well as line where problem occurred. Also handling missing files better? Rename Attribute Column? Give Attributes create/dropSQL() methods so that add/drop column can work like createObject/dropObject.

To Do in wsatools.ExternalProcess If we really want to be pedantic we should enclose the Popen call in a with-block, e.g. "with Popen() as proc:".

To Do in wsatools.ExternalProductConverter.ExternalProduct May have to update for OSA mosaics

To Do in wsatools.FitsToDb DbUpdater probably should update ProgrammeCurationHistory. Member variables and documentation need to be updated a bit for FitsReader and associated classes.

To Do in wsatools.FitsToDb.FitsReader Test on more than one task at a time.

To Do in wsatools.Logger Define the verbose CLI option here - need to be careful though to make sure that the scripts that inherit it don't already have -v options - also would be nice to make sure they all use it!

To Do in wsatools.Logger.ForLoopMonitor Optionally supply separation of markers.

To Do in wsatools.Logger.Logger Replace setEchoOn()/Off() with a public echoOn member variable. Give the constructor a writeMode argument, with a default of 'w', so that parallel CUs can supply 'a' instead (this avoids the need for an IngestLogger class). Might be better though to have this as a class variable because when in 'a'ppend mode we need to ignore the isVerbose option otherwise old logs are overwritten and the last log doubled up.

To Do in wsatools.Logger.Logger.addExceptionDetails This function is deprecated. Use addExceptionMessage() with the includeTrace flagged turned on instead.

To Do in wsatools.ProgrammeBuilder Make sure new pointingString and old pointingString refer to same positions Need to replace some hard wired constants with database constants parsed from WSA_InitiateArchive.sql. Therefore, may need to add more constants to that SQL script.

To Do in wsatools.ProgrammeBuilder.ProgrammeBuilder.createSchema Combine surveyparser.py functionality into this function, which will allow surveyparser.py to be just an interface to this class and improve consistency between survey and non-survey schema.

To Do in wsatools.QualityControl Make bit flags module constants. Seriously consider changing cross-talk to avoid need for detections of TWOMASS stars to exist in the WFCAM field for MosaicCrossTalkFlag at least - if not others too. Would need to use min/maxRa/Dec from CurrentAstrometry table and then convert TWOMASS RA, Decs to x, y using pywcs. Add statistics of percentage flagged with each quality issue. Make use of DbSession.update() accepting Join objects as the fromTables argument. Additional quality flags (trac tickets: 13, 31).

To Do in wsatools.QualityControl.CrossTalkFlagger Complete reliability tests (trac ticket:32)

To Do in wsatools.QualityControl.QualityFlagUpdater.updateDitherFlag Boundary flag for UDS mosaic? Probably not worthwhile because there is only one field with detections consistent in each band, so users could just make their own x/y cuts. Alternatively implement a flag based on avConf.

To Do in wsatools.SExtractorInterface Calculate seeing,avStellEll, go deeper with classification import self from CU13 - use isTrialRun, redo ... use SExtractor 286 - no longer need to break out extensions.

To Do in wsatools.SourceMerger Implement View, FrameSetView, and FrameView objects. Remove sort on outgest during reseaming and alter _analyseSeams() to remove requirement that pointers are sorted by declination, which will also speed up _getFrameSetSources() query as no sort by dec will be required. Make SourceMerger a base class, that CU6,7,9 have like an Outgester class.

To Do in wsatools.Utilities.WordWrapper Instead of not wrapping pre-formatted lines, handle the case of a new-line character when calculating the overlap.

To Do in wsatools.Utilities.arbSort See if defining my own compare function that calls index() is faster/simpler than the Decorate-Sort-Undecorate method used here.

To Do in wsatools.Utilities.extractColumn Replace with extractColumns()? Could leave this method here for speed and simplicity. Though normally we want more than one column anyway! So, may just extractColumns(file, 3)[0] isn't so bad.

To Do in wsatools.Utilities.extractColumns Possibly alter to make use of the CSV module's abilities to handle different dialects? Would simplify code a bit, and make more useful.

To Do in wsatools.Utilities.getNextDisk Instead of taking a SystemConstants object, why not make this a method of SystemConstants?