Participants:
David Bacon, Gary Bernstein, Michael L. Brown, Håkon Dahle, Meghan Gray, Marco Hetterscheidt, Catherine Heymans, Henk Hoekstra, Mike Jarvis, Vera Margoniner, Richard Massey, Alexandre Refregier, Jason Rhodes, Mischa Schirmer, Tim Schrabback, Ludovic Van Waerbeke.

Unable to attend:
Emmanuel Bertin, Sarah Bridle, Douglas Clowe, Thomas Erben, Konrad Kuijken, Yannick Mellier, Reiko Nakajima, Molly Peeples, David Wittman

Short summary:

STEP paper 1: Condensed version of report including SkyMaker simulations (and shapelet simulations?). Simple mistakes found and corrected in pipelines allow for a re-analysis to be included, once agreed by whole group. First draft 15th Feb.

STEP paper 2: Blind analysis of large set of shapelet simulations, made available for analysis by 1st Feb. Modifications and improvements to pipelines encouraged using lessons learnt from SkyMaker simulations.

STEP paper 3: Space-based simulations analysed by space subgroup.

Agenda items:

1) Are people happy to publish some form of their results?

By e-mail, Sarah not yet sure, but is keen to have a benchmark paper to compare future results with.
Everyone else says yes but there are some issues about PR.
Alex & Richard concerned that the results as they are do not present lensing in a good light when compared to the accuracy being quoted by other w-measuring-tools e.g SZ.
Henk - these quoted figures do not include systematics
Ludo - STEP results show a snapshot of how well we can do now and yes we need to reduce the errors by several orders of magnitude to measure w, but these results provide a benchmark upon which we can improve
Vera - important to publish these results as a benchmark, different groups can move forward from this project to improve upon their methods with the lessons learnt
Catherine & Alex - important to word the paper carefully with PR in mind

2) For those methods which have not yet been used in published cosmic shear results, are modifications allowed?

Vera - this was a blind test and the results are stronger if its kept as a blind test. Modifications should be included in a follow up paper
Richard - this test was entered into by many as a learning exercise. For new methods it may have been used as a way to find bugs. Therefore any simple bugs found should be corrected and the simulations re-analysed
Alex & Gary - knowing the input simulation shear, its very easy to correct things until you get the right answer, and not correct things that don't give you the right answer
Catherine - If people want to use what they've learnt from the simulations to find bugs and correct simple mistakes thats fine, provided the corrections do not include dubious selection criteria that somehow give the right answer. We want to present the best results possible.
Vera - if people find mistakes they should e-mail the group with their proposed correction which can then be discussed
Catherine - If everyone is agreed that the correction is minor/simple, simulations should be re-analysed with the corrected pipeline by 1st Feb.

3) How much time can people dedicate?

Ludo - analysis of simulations is automated and so analysis is predominantly computer time.
Vera - the downloading of data was difficult and time consuming - DVDs of future simulations would be much better

4) Bearing in mind (3) do we want to publish a condensed version of the report or do we need to do more work to understand the different biases.

Catherine - lots of great ideas from Konrad, Gary and Doug to condense the information in the report. E-mail if you have any more suggestions. The COMBO-17 team is trying to address issues with different implementations of KSB. This forthcoming paper may be sufficient to address the inevitable question of the referee, why does different implementations of KSB give different results? If you haven't already, please send details about your KSB analysis.
Gary - there is a precedent set by a group of photo-zers who all ran their code on a set of data and presented the results. Interpretation of why the differences occur was not given and it was still a very good paper.
Catherine - encourage people to look at the input catalogues to try and understand where calibration biases and selection biases come from - include a 1-2 paragraphs for each method with the findings.
Ludo - if we cross correlate the different catalogues we remove different peoples selection bias and the dispersion then directly tells us about calibration bias.
Gary - proposes 2 papers. The first a condensed version of the report. The second, an analysis of a large set of shapelet simulations where the lessons learnt from the SkyMaker simulations can be included in pipelines to improve them. Details about the improvements and modifications can be included in the second paper.
Everyone agreed

5) Shapelet simulations - to understand the differences between the two types of simulations we need shapelet simulations with the same area and number density of stars and galaxies. Do people have time to analyse these simulations?

Everyone agreed that analysing more simulations was a good idea. Richard can make them available by 1st Feb hopefully with DVD option, please e-mail him with suggestions of what you would like included in the simulations.
Gary - to reduce shape noise you can introduce galaxy pairs rotated by 90 degrees to each other
Catherine - suggests 3 months timescale to analyse new simulations - submission by 1st May. Big changes to the code that improve SkyMaker results now allowed. The improvements can then be blindly tested with the shapelet simulations.
Alex - for the shapelet method this test isn't so good as shapelets should do very well with shapelet based simulations.
Ludo - can make more SkyMaker simulations for blind analysis if required
Catherine - Should all authors provide results for the SkyMaker simulations for continuity?
Meghan - owing to time/space constraints SkyMaker simulations were not analysed - but can be if required
Jason - only analysed shapelet simulations as RRG is optimised for space-based data so choose to analyse the simulations with the lowest seeing
Catherine - extra SkyMaker analysis is not required (input shear is now known), but if Meghan and Jason want to include SkyMaker results in the paper they can by 1st Feb.
Catherine - do we include the results from the first set of shapelet simulations? Personal preference is no, as the noise makes it difficult to interpret why we find different results with the shapelet and SkyMaker simulations.
Jason - would like them included.
Henk - some problems with the shapelet simulations would prefer to wait for the analysis of the large set of simulations
Catherine - e-mail me with your opinion on whether you'd like the first set of shapelet simulations included in the first paper or not.

6) Space-based simulations

Henk - one method should work for all types of data - no more space-based data for the next 5 yrs - is this test important?
Catherine - different methods may be optimal for ground or space based data as the data properties are so different - the test is important to interpret the differences between published space-based cosmic shear results
Alex - COSMOS, GEMS, ACS parallel data currently being analysed so this test is important
Ludo - can make SkyMaker simulations with space-based PSF
Richard - can make shapelet simulations with space-based PSF
Catherine - please e-mail if you would like to be part of a space testing sub-group

7) Simulations that include cosmic shear

Catherine - the biases with magnitude and size seen in SkyMaker results makes it difficult to interpret how the calibration biases found effect our cosmic shear results - do we need simulations with a real cosmic shear signal?
Alex & Ludo- can't correct &sigma ₈ estimates by factors derived by STEP, but we can compare the dispersion in the STEP results and the dispersion in the &sigma ₈ results.
Henk - mention this as a smoking gun, something that needs to be worked on in the coming years

8) Deadlines

Date for final approved SkyMaker analyses : 1st Feb
Date new shapelet simulations available: 1st Feb
Date for first draft of STEP paper 1: 15th Feb
Date for completion of shapelet simulations analysis : 1st May

9) Any other business

Many thanks to all those involved and MPIA for funding telecon.

---

17th December 2004.
STEP pages maintained by Catherine Heymans: heymans[at]mpia.de