STEP 2: Results
STEP home Simulations Data Mailing List

Shear TEsting Programme 2: The accuracy of weak lensing analysis

Richard Massey, Catherine Heymans, David Bacon, Joel Berge, Gary Bernstein, Sarah Bridle, Douglas Clowe, Thomas Erben, Marco Hetterscheidt, Will High, Christopher Hirata, Henk Hoekstra, Mike Jarvis, David Johnston, Konrad Kuijken, Rachel Mandelbaum, Yannick Mellier, Reiko Nakajima, Stephane Paulin-Henriksson, Molly Peeples, Alexandre Refregier, Jason Rhodes, Chris Roat, Mischa Schirmer, Tim Schrabback, Uros Seljak, Elisabetta Sembolini, Ludovic Van Waerbeke.

The second STEP project assesses the impact of complex galaxy morphologies on weak lensing analyses using two large sets of Shapelet simulations, that have been designed to remove the effect of shape noise in the analysis. Here we present the results of the blind analysis. The minutes of the April STEP2 telecon, where these results were discussed, can be found here along with a copy of some of the e-mail discussions that followed. The input shears and catalogues can also now be downloaded.

Quick links

Summary of results
In summary, we find that:
  • the introduction of complex galaxy morphologies does not dramatically change shear measurement accuracy in the methods that were tested
  • the most accurate methods with better than ~3% level calibration errors for most of the PSFs tested were the Mike Jarvis implementation of BJ02, the Henk Hoekstra and Tim Schrabback implementation of KSB+, the Konrad Kuijken implementation of shapelets and the Rachel Mandelbaum implementation of REGLENS.
  • the method that was most efficient at removing all the different strengths of PSF anisotropy to better than 0.2% accuracy was BJ02 (Mike Jarvis and Reiko Nakajima)
  • all those who took part in STEP1 now achieve better than ~5% shear measurement accuracy
  • both the shear calibration errors (multiplicative) and PSF anisotropic errors (additive) are found to be dependent on the PSF model
  • shear calibration errors are also somewhat dependent on which shear component is measured, where the e1 component is typically measured to a better accuracy than the e2 component.
  • for most methods, the shear measurement accuracy is dependent on both galaxy magnitude and size. Typically the shear measured from bright galaxies is overestimated and the shear measured from faint galaxies is underestimated by up to 10 percent. The PSF correction is typically the poorest for the smallest galaxies.
Shear measurement methods and authors
The following people took part in the full STEP2 analysis. The identity key is used in all of the results which follow. See Heymans et al. 2006 for descriptions of most of the methods tested in STEP2. For the fast reader, columns 4-7 contain the multiplicative and additive % errors on shear measurement (m and c multiplied by 100) for each shear component as measured from the PSF A simulations, where PSF A is a typical Subaru PSF. The results are explained in more detail below, and are dependent on PSF type.

ID AuthorMethodPSF A m1 PSF A c1PSF A m2PSF A c2
JB Joel BergeShapelets (Refregier 03) 2.5 +/- 3.0 -0.1 +/- 0.1 1.7 +/- 2.8 0.0 +/- 0.1
MH Marco HetterscheidtKSB+ 3.3 +/- 1.3 0.1 +/- 0.0 3.7 +/- 1.3 0.1 +/- 0.0
HH Henk HoekstraKSB+ -0.6 +/- 0.8 0.0 +/- 0.0 -1.9 +/- 0.8 -0.1 +/- 0.0
MJ Mike JarvisBJ02 (Bernstein & Jarvis 2002) -1.6 +/- 1.2 -0.1 +/- 0.0 -5.0 +/- 1.1 0.0 +/- 0.0
J2 Mike JarvisBJ02 + new weighting scheme -1.8 +/- 1.0 -0.1 +/- 0.0 -3.6 +/- 0.9 0.0 +/- 0.0
KK Konrad KuijkenShapelets (Kuijken 06) -0.5 +/- 1.1 0.0 +/- 0.0 -2.3 +/- 1.0 0.0 +/- 0.0
RM Rachel MandelbaumREGLENS (Hirata & Seljak 03 +) -1.9 +/- 1.0 0.2 +/- 0.0 -3.1 +/- 0.9 -0.1 +/- 0.0
RN Reiko NakajimaBJ02 (Deconvolution method) -1.8 +/- 1.4 -0.1 +/- 0.0 -7.2 +/- 1.3 0.0 +/- 0.0
SP Stephane Paulin-HenrikssonKSB+ -10.7 +/- 1.2 0.0 +/- 0.0 -4.2 +/- 1.3 0.0 +/- 0.0
MS Mischa SchirmerKSB+ (scalar) -14.9 +/- 1.1 -0.1 +/- 0.0 -15.3 +/- 1.1 0.1 +/- 0.0
S2 Mischa SchirmerKSB+ (tensor) -3.3 +/- 1.8 -0.1 +/- 0.0 -2.9 +/- 1.8 0.2 +/- 0.0
TS Tim SchrabbackKSB+ -1.4 +/- 1.5 0.0 +/- 0.0 -1.2 +/- 1.4 0.0 +/- 0.0
ES Elisabetta SemboliniKSB+ -21.8 +/- 1.3 0.0 +/- 0.0 -21.1 +/- 1.3 -0.1 +/- 0.0

STEP2 accuracy analysis; multiplicative and additive errors
For a full description of how we analyse the STEP2 catalogues to remove the effects of shape noise download this. For each author and PSF type, following the first STEP project, we determine, from the range of applied sheared images, the best-fit parameters to,

where gamma^true_i is the external shear applied to each image. In the absence of shear calibration bias, we would expect the mulitplicative error m=0. In the absence of anisotropic PSF systematics, we would expect the additive error c=0. The STEP2 PSF models are all very different. We therefore do not average the results over the different PSFs, but present the shear calibration error (m) and PSF additive error (c) for different authors and different PSFs here.

STEP2 accuracy as a function of galaxy magnitude and size
With the removal of shape noise in the STEP2 analysis, we are now able to effectively probe the accuracy of shear measurement as a function of galaxy magnitude and galaxy size. You can download plots of the shear calibration error (m) and PSF additive error (c) as a function of galaxy magnitude and galaxy size here. Note that the magnitude is relative at the moment as it might be still be subject to a zeropoint correction.

The variation in the intrinsic ellipticity distribution as a function of STEP2 galaxy magnitude and size (see the e-mail discussions on this matter) will impact on the responsivity correction of the BJ02-esque methods. If we take this variation into account in the BJ02 shear measurement calculation, the BJ02 measurement accuracy as a function of galaxy magnitude becomes more constant, as shown in these updated plots.

Number counts and selection bias
Authors use different criteria to select galaxies from the simulations. You can download a data table that compares the number density of objects used in each analysis and the mean magnitude of those objects (which is again might be subject to a zeropoint correction). Also listed are the number of false detections and stellar contamintion, which is <1% for most authors.

The STEP2 analysis relies on matching the shear catalogues from the original and rotated sets of images. This introduces an effective signal-to-noise cut on the original catalogues, reducing the number density of galaxies typically by ~10% and decreasing the mean galaxy magnitude by ~1%, as detailed in the data table.

The STEP2 analysis method does not appear to introduce any selection bias in the average selected galaxy shapes. The data table lists the average selected source ellipticity in the original catalogue and the matched catalogue. These measurements are still subject to shape noise error, and are, in most cases, consistent with zero selection bias.

We thank Caltech for hosting the STEP2 simulations, and for providing the extensive computing resources required to produce them. We also thank the Jet Propulsion Laboratory for financial and administrative support of the STEP workshop, and NSERC and CITA for financial support of STEP teleconferencing. We also thank Richard Ellis for his ever helpful comments and discussion.

Last modified 12th May.
STEP pages maintained by Catherine Heymans: heymans[at]