Overview
Processing for HSC SSP data is done using the HSC pipeline, a custom version of the LSST Data Management codebase, specialized and enhanced for Hyper-Suprime Cam processing by software teams at NAOJ, IPMU, and Princeton. Most of the pipeline code was written from scratch for LSST and HSC, with significant algorithmic heritage from the SDSS Photo and Pan-STARRS Image Processing Pipelines. The HSC pipeline can also be used for general-observer HSC or custom processing of SSP data; see documentation for basic installation and use instructions.
There are five major processing stages:
- CCD Processing
- Global Sky Subtraction
- Internal Calibration
- Image Coaddition
- Detection and Measurement
CCD Processing
For each CCD in an exposure (“visit”), we:
- Perform low-level detrending: subtract bias, dark, fringe (only in red filters), apply flat-field corrections, mask and interpolate defects
- Estimate and subtract the sky background
- Detect and measure bright sources (basic centroiding, aperture photometry, and moments-based shapes)
- Select isolated stars and model the point-spread function (PSF), using a modified version of PSFEx
- Find and interpolate cosmic rays using morphology
- Fit an initial World Coordinate System (WCS) and approximate photometric zero-point by matching to a Pan-STARRS (PS1) DR1 reference catalog
- Detect, deblend, and measure all sources
All of the above steps are run separately on each CCD in parallel.
Global Sky Subtraction
There has been a known issue around bright or extended sources where the sky background is over-subtracted. In this release, we mitigate this problem by introducing a new approach. First, the background subtracted in the CCD processing is put back in. We then subtract a large-scale empirical background modelacross the field-of-view for each exposure image, using super-pixels of 1024 x 1024 pixels for estimating backgrounds. Finally, we subtract the sky frame, a static pattern that has a spatial variation in a smaller scale than the large-scale model. The static patterns are generated for each filter and for each observing run. The resultant image preserves the extended wings much better than without this algorithm enabled.
Internal Calibration
Using measurements of the same stars from multiple visits, we determine accurate positions and angles of the CCDs and their flux scales. This is done by solving for a final WCS solution with higher-order distortion terms and a spatially-varying photometric zero-point of each CCD, by requiring self-consistent fluxes and centroids for each star. This procedure is carried out in approximately 1.7×1.7 degree pre-defined sky regions. These regions are called “tracts” that tile the sky with small overlaps.
Image Coaddition
All CCD images in a tract are resampled onto a rectangular coordinate system centered on that tract, and then averaged together to build a combined image, called a “coadd”. This is done separately for each band. Instead of matching the PSFs of the input images by degrading images with better seeing, we combine the images as they are, and then construct a PSF model by resampling and combining images of the per-exposure PSF models at the position of every source on the coadd.
To avoid including artifacts (e.g. satellite trails, cosmic rays) or asteroids in the coadd, we generate a list of artifact candidates and exclude them on each exposure before coadding. First, we resample individual CCD images onto the target coordinate system to make per-exposure images (called warp). We smooth each warp image to a common PSF size. We also generate a reference coadd with a simple median of them. By differencing the PSF-matched warps and the reference coadd, we can identify these artifacts directly in the original exposures and reject them from final coadd with a simple weighted mean. This procedure allows us to reasonably preserve PSFs on the final coadd, while it helps to detect and reject artifacts better than a simple per-pixel outlier-rejection procedure.
Detection and Measurement
We detect sources in each band separately. Before deblending and measuring the sources we first merge them across bands. Above-threshold regions (called “footprints”) from different bands that overlap are combined; peaks within these regions that are sufficiently close together are merged into a single peak. This yields footprints and peaks that are consistent across all bands. We consider each peak to represent a source, and treat all peaks within the same footprint as blends.
We then divide (“deblend”) the peaks within the footprints separately in each band. This assigns a fraction of the flux in each pixel to every source in the blend, allowing them to be measured separately. We again run a suite of centroid, aperture flux, and shape measurement algorithms on these deblended pixel values, as well as PSF and galaxy model fluxes and shear estimation codes for weak gravitational lensing.
After measuring the coadd for each band independently, we then select a “reference band” for each source. For most sources, this is the i-band; for sources not detected (or low signal-to-noise) in i, we use r, then z, y, g, and narrow bands. We then perform “forced” measurement in the coadds in all bands, in which positions and shapes are held fixed at their values from the reference band and only amplitudes (i.e. fluxes) are allowed to vary. This consistent photometry across bands produces our best estimates of colors of sources.