Processing (PDR3)

Updates since PDR2

As described in the PDR3 paper, there are a number of updates in the processing pipeline.  Details can be found in the paper, but some of the most important changes include:
    • We used to use meas_mosaic for internal astrometric and photometric calibrations, but due to excessive memory usage in the UltraDeep fields, where we have more than 100 visits, we switch to jointcal for astrometry and FGCM for photometry.
    • We have improved the global sky subtraction method by changing the algorithms to generate sky frames as well as to subtract the sky.  However, we subtract the sky on a small scale (128 pixel) just before we detect and measure sources.  This means that there are two types of coadds; one with the global sky subtraction (patch images under deepCoadd/) and the other with the local sky subtraction (calexp images under deepCoadd-results/).  Those interested in extended wings of bright objects can use the former.
    • We have two versions of the r and i-band filters; r vs. r2 and i vs. i2.  The transmission functions of these filters are similar but not identical.  We combine them in the coaddition phase, but an afterburner analysis has been performed to translate all the r/i-band photometry into r2/2 using the effective response curve available for each object (generated in the same way as coaddPsf).  The corrections are available in the database.  See this page for detail.


Processing for HSC SSP data is done using hscPipe, a custom version of the LSST Data Management codebase, specialized and enhanced for Hyper-Suprime Cam processing by software teams at NAOJ, IPMU, and Princeton. Most of the pipeline code was written from scratch for LSST and HSC, with significant algorithmic heritage from the SDSS Photo and Pan-STARRS Image Processing Pipelines. Details of the pipeline are described in Bosch et al. 2018 and Bosch et al. 2019 with updates summarized in the PDR2 and PDR3 papers.

There are five major processing stages:

  1. CCD Processing
  2. Global Sky Subtraction
  3. Internal Calibration (jointcal and FGCM)
  4. Image Coaddition
  5. Detection and Measurement

Photometry and astrometry are calibrated against an external source catalog, which is PS1 DR2 astrometrically calibrated against Gaia DR1.  This reference catalog is not without problems and the user is referred to the QA section of the PDR3 paper.

CCD Processing

For each CCD, we –

  • Perform low-level detrending: subtract bias, dark, fringe (only in red filters), apply flat-field corrections, mask and interpolate defects
  • Estimate and subtract the sky background
  • Detect and measure bright sources (basic centroiding, aperture photometry, and moments-based shapes)
  • Select isolated stars and model the point-spread function (PSF), using a modified version of PSFEx
  • Find and interpolate cosmic rays using morphology
  • Fit an initial World Coordinate System (WCS) and approximate photometric zero-point by matching to the reference catalog
  • Detect, deblend, and measure all sources

All of the above steps are run separately on each CCD in parallel.

The five images show a CCD pixel area of raw (unprocessed), bias-subtracted, flat-fielded, estimated sky-background, and sky-subtracted images, from left to right.

Global Sky Subtraction

In PDR1, we over-subtracted the sky around bright or extended sources. In PDR2, we mitigated this problem by introducing a new global sky subtraction approach. We further improve it in this release.  First, the background subtracted in the CCD processing is put back in. We then subtract a large-scale empirical background model across the field-of-view for each exposure image using 8kx8k superpixel, followed by subtraction of the sky frame, a static pattern that has a spatial variation in a smaller scale than the large-scale model. The static patterns are generated for each filter and for each observing run. Finally, relatively small superpixels of 256 pix are used to estimate the local sky residual by carefully masking objects.  The resultant image preserves the extended wings much better than without this algorithm enabled.  The following images compare coadds from PDR1 and PDR3.

We note, however, we choose to perform the local sky subtraction intentionally removing extended wings of bright objects for detection and measurements.  Thus, all of our coadd measurements are performed on the locally subtracted images.  The online image browser, hscMap, allows you to compare local vs. global images (go to ‘Dataset’ pull-down menu).

On the left is PDR1 and the right is this release (PDR3).

Internal Calibration

Using measurements of the same stars from multiple visits, we determine accurate positions and angles of the CCDs. This is done by solving for a final WCS solution with higher-order distortion terms and a spatially-varying photometric zero-point of each CCD, by requiring self-consistent fluxes and centroids for each star.  This procedure is carried out for each tract separately using the jointcal package  from LSST.  Joint photometric calibration is performed separately and is done by using all filters and all visits.  FGCM forward models the atmosphere and system response and their variability and thus more data generally result in better calibrations.

Image Coaddition

All CCD images in a tract are resampled onto a rectangular coordinate system centered on that tract, and then averaged together to build a combined image, called a “coadd”. This is done separately for each band. Instead of matching the PSFs of the input images by degrading images with better seeing, we combine the images as they are, and then construct a PSF model by resampling and combining images of the per-exposure PSF models at the position of every source on the coadd.

To avoid including artifacts (e.g. satellite trails, cosmic rays) or asteroids in the coadd, we generate a list of artifact candidates and exclude them on each exposure before coadding. First, we resample individual CCD images onto the target coordinate system to make per-exposure images  (called warp). We smooth each warp image to a common PSF size. We also generate a reference coadd by taking clipped mean. By differencing the PSF-matched warps and the reference coadd, we can identify these artifacts directly in the original exposures and reject them from the final coadd.  This procedure allows us to reasonably preserve PSFs on the final coadd, while it helps to detect and reject artifacts better than a simple per-pixel outlier-rejection procedure.  In PDR2, the config params for this artifact rejection were not optimal and resulted in poor PSF photometry over a small area, but we have update the configs in PDR3 to eliminate the photometry problem.

When we warp individual CCD images, we used to use Lanczos3 kernal.  But, there is a non-negligible PSF size difference between the model PSF and observed PSF for weak-lensing cosmology.  We discover that increasing the order to the 5th kernel reduces the size residual and we adopt the 5th order kernel as default in PDR3.

Image coaddition of multiple exposures. Blue areas are overlaid on detected sources, and green contours correspond to individual objects. We detect fainter sources in the coadd.

Detection and Measurement

We detect sources in each band separately.  Before deblending and measuring the sources we first merge them across bands. Above-threshold regions (called “footprints”) from different bands that overlap are combined; peaks within these regions that are sufficiently close together are merged into a single peak. This yields footprints and peaks that are consistent across all bands. We consider each peak to represent a source, and treat all peaks within the same footprint as blends.  There was a bug in PDR2 that prevented the logical flow here in very rare cases and that has been fixed.

We then divide (“deblend”) the peaks within the footprints separately in each band. This assigns a fraction of the flux in each pixel to every source in the blend, allowing them to be measured separately. We again run a suite of centroid, aperture flux, and shape measurement algorithms on these deblended pixel values, as well as PSF and galaxy model fluxes and shear estimation codes for weak gravitational lensing.

We first measure the coadd for each band independently.  This is called the unforced measurement and the results are stored in the meas tables at the database.  We then select a “reference band” for each source. For most sources, this is the i-band; for sources not detected (or low signal-to-noise) in i, we use r, then z, y, g, and narrow bands. We then perform “forced” measurement in the coadds in all bands, in which positions and shapes are held fixed at their values from the reference band and only amplitudes (i.e. fluxes) are allowed to vary. This consistent photometry across bands produces our best estimates of colors of sources.  The output is stored in the forced tables.