Bolocat v1-v2 comparison with new calibration

I re-examined the Bolocat data on l351 after re-running the pipeline with the new calibration curve. The change wasn't all that great. See this post for a brief description of the procedure.

In the data below, I've fit the residuals as a function of v1.0.2 flux density in an aperture (source mask) with a line. The slope of the line should ideally be zero - that would indicate a multiplicative offset is an acceptable correction.

Nicely, in the 40" aperture case, I see no reason to exclude the m=0 case. For the most reliable data - the 13pca - the slope is rather small and the "correction factor" is disturbingly close to what we recommended (1.5). We have no right to be that lucky...

...and so perhaps wer are not. The source mask includes more area and therefore is more sensitive to extended flux recovery. The slopes are not consistent with 0 - just look at the data above and below 2 Jy to see that there is a difference. The multiplicative correction of 1.5 is decent for a pretty wide range of flux densities, but is inadequate for the brightest sources. This is somewhat interesting... it implies that the brightest sources also lie on the highest backgrounds.

You might note that the brightest source has a smaller correction factor in both apertures. It's not clear why that is the case, but I don't think it's enough to call it a trend yet - wait for the full 8000-source comparison first.


Why is there so much scatter? Not entirely clear, but the scatter is primarily at low S/N.

Here are the same for all of the data reduced up to this point:

Bolocat v1-v2 comparison

For this experiment, I ran Bolocat on all of the v1 and v2 images using the source masks Erik derived for v1. I then compared the derived fluxes using both aperture photometry (as defined in bolocat) and the mask sums.

First, note that the object_photometry code Erik wrote does NOT do background subtraction - this is important for understanding non-multiplicative offsets between v1 and v2.

I fit a 2-parameter model (a line) to the v2 vs v1 plots using two methods. The simple linear-least-squares method is one we're all familiar with, and it gives semi-reasonable results, but is NOT statistically robust when there are errors on both axes. The other fit method used was Total Least Squares, which is approximately equivalent to Principle Component Analysis with 2 vectors. It uses components of the singular-value-decomposition to determine the best fit. The fits returned by TLS should be more robust, though the additive offsets need not be.

Conclusion: Our factor of 1.5 looks like it was pretty accurate. For 40" apertures, the best fit is ~1.56, which is easily within the error bars. Luckily, for 40" apertures, there is no apparent need for an additive offset (I'm pretty sure the uncertainty on the measured offset is larger than the offset, though statistically that is not the case), which would complicate things.

However, for the source mask, there is a greater scaling factor and a very substantial offset. This implies that the peaks in v2 are brighter by a large factor, but the backgrounds in v2 are actually lower than in v1 (though please do check my reasoning here). I'm really not sure what to make of the difference between source mask and aperture yet, though... there's probably something significant in the source mask being forced to pick positive pixels. (and peppered pickles)

Examining bolocat

Just in case we were wondering, the V1 bolocat is completely inconsistent with a lognormal distribution, but is perfectly consistent (or... at least reasonably consistent...) with a power law distribution.

These plots show power-law fits (red) and lognormal fits (blue) to the data (black). It's pretty obvious that the lognormal is a bad fit, but in case you're unconvinced, the ks test for the "source flux" has a probability 1.6e-7, and it is the highest likelihood by 17 orders of magnitude out of the 4 flux types.

By contrast, the simulations are on average (though not uniformly) more consistent with lognormal than powerlaw distributions:

Even in those examples where the KS test is slightly more favorable for the powerlaw distribution, the lognormal is a pretty good fit, and the failure points for the two distributions are in about the same place. The smoothness of the lognormal distribution is required to reproduce the observed distribution.

Note that the first 4 plots are for the whole BGPS survey. What about an individual field? For obvious reasons, I choose l30 again.

This gets to be a little more interesting - apparently the "source flux" has a tendency to pick up the power-law distributed background structure, since it is consistent with a lognormal (but note that it is also consistent with a powerlaw! The ks test doesn't really say definitively which is better). Although the fits look bad at high flux, note that this is a log-log plot and therefore the difference in probability is rather small.

What does this all indicate? It's not entirely clear whether individual fields are genuinely more lognormally-distributed or whether the number statistics are just worse. However, even the source flux is consistent with a power-law, while many realizations of the simulations are not.

Therefore, we should perform the next logical test - add point sources drawn from a power-law distribution (and a log-normal distribution?) and see what bolocat retrieves. We can at least say now that the point source contribution cannot be ignored, since there is no power-law distribution that can reproduce the observed bolocat flux distribution.