Random Points

The random points are drawn with a density of 100 points per sq. arcmin, for each patch and each filter that have been processed by the pipeline. We draw the points and probe the mask information at each random position using a python module running within the HSC pipeline environment. The source code can be accessed on gitHub here: https://github.com/jcoupon/maskUtils.

The random points can be used for several purposes;  (1) a random sample for clustering analysis, (2) to identify problematic areas, (3) to compute the survey area and the fraction of masked areas, etc.  Available information carried by the random points, per filter and per patch, is:

  • patch and tract number
  • patch and tract geometry (non-overlapping regions: patch_inner and tract_inner)
  • pixel mask bits such as flags_pixel_interpolated_any
  • local PSF size: recorded as the second moment matrix (see FAQ: how to measure the size of an object, to convert into FWHM)
  • sky_mean: mean of sky value in 2″ diameter apertures (merge.footprint.sky == True objects). Constant within patch.
  • sky_std: standard deviation of sky value in 2″ diameter apertures (merge.footprint.sky == True objects). Constant within patch. To compute the limiting magnitude at 5 sigma, do: lim_mag = -2.5 log10(5*sky_std) + 27.0
  • adjust_density parameter: a random number between 0 and 1 to adjust the sky density. A cut like adjust_density < 0.5 will decrease the sky density by a factor of two
  • number of visits contributed to coadds (countinputs)
  • pix_variance : pixel variance at random point position (note that the variance given for the object is the sum of the pixel variance over the source footprint).
  • Milky Way extinction value (Schlegel map).

Known issues

  • centroid_sdss_flags and flags_badcentroid flags must be ignored because they are meaningless for randoms. If set to False, 85% of randoms are lost.
  • The bright-star masks are too large for a few very bright stars and include about 10% of bright galaxies.  See also the Known Problems page.
  • Some functions of the object database are not available in the randoms database.


Query the database

-- This is a simple sql query to get all columns from the random database -- in one single patch SELECT * FROM pdr1_wide.random AS ran WHERE tract=8523 AND patch_s='3,5' AND iflags_pixel_bright_object_center is not True AND iflags_pixel_edge is not True

Do a joint object/random query

-- The object version. SELECT object_id, ra, dec, imag_kron, imag_kron_err FROM pdr1_wide.forced WHERE boxSearch(coord, 34.0, 36.0, -5.0, -4.5) AND NOT iflags_pixel_bright_object_center AND NOT iflags_pixel_edge AND icountinputs > 4 -- The random version. -- Note that the boxSearch function is not available in -- the random database. -- For the randoms, the "NOT iflags_pixel_edge" is mandatory -- near the border of the survey. SELECT object_id, ra, dec FROM pdr1_wide.random WHERE ra BETWEEN 34.0 AND 36.0 AND dec BETWEEN -5.0 AND -4.5 AND NOT iflags_pixel_bright_object_center AND NOT iflags_pixel_edge AND icountinputs > 4

Add a cut on PSF size

Note that in the random database, the moments are in pixel, whereas in the object data, it's in arcsec (hence the 0.17 factor for the random). The factor 2sqrt(2) is to convert from sigma to FWHM, assuming the PSF is gaussian
random: shape_detradius(array[ishape_sdss_psf_11, ishape_sdss_psf_22, ishape_sdss_psf_12])*2.0*sqrt(2.0)*0.17 < 1.0 object: shape_detradius(array[ishape_sdss_psf_11, ishape_sdss_psf_22, ishape_sdss_psf_12])*2.0*sqrt(2.0) < 1.0

Compute the survey area

The random point density is 100 per arcmin2. Thus the area in deg2 is simply the number of random points given a selection:
SELECT COUNT(*)/100.0/3600.0 FROM pdr1_udeep.random WHERE icountinputs > 10;

Adjust the random local density

The "adjust_density" is a random number in the interval [0:1]. For example, to divide the random point density by a factor of two (to get 50 /arcmin2), set:
iflag_adjust_density < 0.5

Compute the patch mean sky and sky variance

The mean and standard deviation of the sky object in 2" diameter aperture per patch is stored as sky_mean and sky_std. To get these quantities per patch and compute the limiting magnitude, e.g. at 5 sigma, one may do this directly using the database
-- This query gets the patch mean sky and variance SELECT ran.isky_mean, ran.isky_std, -2.5*LOG(5.0*ran.isky_std)+27.0 as imag_limit FROM pdr1_wide.random AS ran WHERE ran.tract=8523 AND ran.patch_s='3,5' LIMIT 1

Compute the masked fraction

-- This query returns the masked fraction in bright object masks for tract 8523. SELECT 1.0 - CAST(sum( CASE WHEN iflags_pixel_edge is not True AND iflags_pixel_bright_object_center is not True THEN 1 ELSE 0 END ) AS DECIMAL ) / CAST(sum( CASE WHEN iflags_pixel_edge is not True THEN 1 ELSE 0 END ) AS DECIMAL) as masked_fraction FROM pdr1_wide.random AS ran WHERE tract=8523

Identify areas with large seeing

-- This query gets 1% of random objects in the entire Wide and record the position as well as the local seeing FWHM. SELECT ran.ra, ran.dec, shape_detradius(array[ran.gshape_sdss_psf_11, ran.gshape_sdss_psf_22, ran.gshape_sdss_psf_12])*2.0*sqrt(2.0)*0.17 as gPSF, shape_detradius(array[ran.rshape_sdss_psf_11, ran.rshape_sdss_psf_22, ran.rshape_sdss_psf_12])*2.0*sqrt(2.0)*0.17 as rPSF, shape_detradius(array[ran.ishape_sdss_psf_11, ran.ishape_sdss_psf_22, ran.ishape_sdss_psf_12])*2.0*sqrt(2.0)*0.17 as iPSF, shape_detradius(array[ran.zshape_sdss_psf_11, ran.zshape_sdss_psf_22, ran.zshape_sdss_psf_12])*2.0*sqrt(2.0)*0.17 as zPSF, shape_detradius(array[ran.yshape_sdss_psf_11, ran.yshape_sdss_psf_22, ran.yshape_sdss_psf_12])*2.0*sqrt(2.0)*0.17 as yPSF from pdr1_wide.random AS ran where iadjust_density < 0.01