Deconvolution vs. Not

In the W5 maps, everywhere except AFGL 4029 agrees to within about 20% (better than our supposed offset) independent of deconvolution scheme / modeling scheme, but AFGL 4029 disagrees by a factor of 2-3, indicating a severe dependence on map size.

Here's the AFGL 4029 comparison originally:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.1515
Sum: 194.028 Mean: 1.03758 Median: 0.78441 RMS: 0.760825 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 5.77585
Sum: 137.481 Mean: 0.735194 Median: 0.517222 RMS: 0.663245 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 14.4043
Sum: 342.863 Mean: 1.83349 Median: 1.62579 RMS: 0.820079 NPIX: 187

Here it is after flagging out another bad bolo:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.14533
Sum: 193.882 Mean: 1.0368 Median: 0.783409 RMS: 0.764047 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 10.1518
Sum: 241.642 Mean: 1.2922 Median: 1.06321 RMS: 0.777165 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 13.9344
Sum: 331.677 Mean: 1.77367 Median: 1.56277 RMS: 0.820139 NPIX: 187

order is deconvtest, nodeconvtest, 'reconv'

deconvtest has virtually no change. nodeconvtest goes up by a LOT... must be something about the weighting. reconv drops, which is good, but not enough...

curiously, it is not clear that nodeconv converges: [10, 15, 20 iters]
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.70939
Sum: 207.308 Mean: 1.34615 Median: 1.09348 RMS: 0.760452 NPIX: 154
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 9.14028
Sum: 217.564 Mean: 1.41275 Median: 1.17223 RMS: 0.759027 NPIX: 154
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 9.42483
Sum: 224.337 Mean: 1.45673 Median: 1.2172 RMS: 0.758978 NPIX: 154

similarly, reconv does not converge:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 11.5623
Sum: 275.214 Mean: 1.61891 Median: 1.36357 RMS: 0.809771 NPIX: 170
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 12.5848
Sum: 299.553 Mean: 1.76208 Median: 1.51426 RMS: 0.807963 NPIX: 170
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 13.3102
Sum: 316.82 Mean: 1.86365 Median: 1.62092 RMS: 0.804582 NPIX: 170


by contrast, deconv converges rapidly:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.27004
Sum: 196.85 Mean: 0.937381 Median: 0.659373 RMS: 0.759002 NPIX: 210
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.39255
Sum: 199.766 Mean: 0.951267 Median: 0.683644 RMS: 0.758766 NPIX: 210
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.42881
Sum: 200.629 Mean: 0.955377 Median: 0.690816 RMS: 0.75878 NPIX: 210

while I'm at it, gaussfits for nodeconv and reconv:
Guess: 427,1113 Fit peak: 2.90411 Background: 0.0102637 X,Y position: 426.514822,1113.354863 X,Y FWHM: 64.253564,72.615868 Angle: 0.000000
Guess: 427,1113 Fit peak: 3.03769 Background: 0.000596982 X,Y position: 426.080980,1113.634660 X,Y FWHM: 77.405551,104.292823 Angle: 329.589335
well, isn't that neat! Better than 5% agreement on the peaks. Curiously, the background is higher for nodeconv, which is false: it is directly evident that the background is higher in reconv. So I think this is just a failed fit, unfortunately. The true peak of the reconv AFGL 4029 is 3.9 Jy, so there's a huge residual


Found an error in the noisemap computations that was artificially driving up the noise around NAN pixels in undersampled maps; this probably led to major problems. I fixed it by only "flagging out" pixels with >2 NAN neighbors, i.e. so sparsely sampled that they should be ignored, I hope.... this may have affected modeling in all maps.

Realized the problems with noisemap: if the model exactly equals the weighted mean at a given pixel, the residual will be EXACTLY zero (to within numerical precision). This leads to underestimates of the noise! What we really want is an estimate of the standard deviation on the mean, which is easily computed! Just do a normal weighted sum of the difference between the data and the model (data and the mean); this is also equivalent (conveniently) to a chi^2 statistic, I think... Whoa. How did I not do this before? LETS FIND OUT I'M SURE THERE ARE AWFUL CONSEQUENCES!


Just had another idea to throw in - what if we downweight the scan edges? It won't matter for individual maps or maps of the same size if done uniformly, but it could help incorporate "pointing" maps into "real" maps more accurately.

Delining and the Cleaning process

One item I forgot to mention last night was the effects of lines/delining on
PCA subtraction. These should be the primary effects on the final map for all
epochs except 2010, in which case the primary effect SHOULD be to reduce
substantial noise.


In the examples below, there are PSDs of whole timestreams (left) and
example timestreams from single scans (right). The first thing to note is that
the delined timestreams still have correlated components in the line region,
but they are suppressed - their amplitudes, and therefore their sort order in
the PCA removal scheme, are changed. Since PCA cleaning is by its nature adaptive
(the number of components remains fixed, but the order changes), these effects
can be significant and dangerous. If the line noise is more correlated, a PCA
component will be dedicated to removing it instead of atmospheric signal.


Below are examples from l089 (epoch 0709) first. These have less correlated
line noise and are more typical of BGPS observations. The first PCA component,
the average, does not change much with PCA cleaning. However it is clear that
the second component changes substantially, from large-amplitude high-frequency
noise to small-amplitude variations that are very likely to describe
atmosphere.

---

Example 1 - Zeroing out the lines:
Example 2 - Fitting and removing the lines:
Example 3 - Suppressing the lines with a non-fitted Gaussian:
The next examples are from December 2010 observations of Uranus. In this case, the correlated noise component is clearly dominant.
Zeroing lines:
Fitted lines:
Non-fitted gaussian suppression:
Finally, these two are demonstrations of what you might expect to see for a purely noiseless images of a planet (it was constructed from a PSF). PCA is first:
A single bolometer's timestream and PSD: