Deconvolution vs. Not

In the W5 maps, everywhere except AFGL 4029 agrees to within about 20% (better than our supposed offset) independent of deconvolution scheme / modeling scheme, but AFGL 4029 disagrees by a factor of 2-3, indicating a severe dependence on map size.

Here's the AFGL 4029 comparison originally:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.1515
Sum: 194.028 Mean: 1.03758 Median: 0.78441 RMS: 0.760825 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 5.77585
Sum: 137.481 Mean: 0.735194 Median: 0.517222 RMS: 0.663245 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 14.4043
Sum: 342.863 Mean: 1.83349 Median: 1.62579 RMS: 0.820079 NPIX: 187

Here it is after flagging out another bad bolo:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.14533
Sum: 193.882 Mean: 1.0368 Median: 0.783409 RMS: 0.764047 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 10.1518
Sum: 241.642 Mean: 1.2922 Median: 1.06321 RMS: 0.777165 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 13.9344
Sum: 331.677 Mean: 1.77367 Median: 1.56277 RMS: 0.820139 NPIX: 187

order is deconvtest, nodeconvtest, 'reconv'

deconvtest has virtually no change. nodeconvtest goes up by a LOT... must be something about the weighting. reconv drops, which is good, but not enough...

curiously, it is not clear that nodeconv converges: [10, 15, 20 iters]
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.70939
Sum: 207.308 Mean: 1.34615 Median: 1.09348 RMS: 0.760452 NPIX: 154
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 9.14028
Sum: 217.564 Mean: 1.41275 Median: 1.17223 RMS: 0.759027 NPIX: 154
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 9.42483
Sum: 224.337 Mean: 1.45673 Median: 1.2172 RMS: 0.758978 NPIX: 154

similarly, reconv does not converge:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 11.5623
Sum: 275.214 Mean: 1.61891 Median: 1.36357 RMS: 0.809771 NPIX: 170
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 12.5848
Sum: 299.553 Mean: 1.76208 Median: 1.51426 RMS: 0.807963 NPIX: 170
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 13.3102
Sum: 316.82 Mean: 1.86365 Median: 1.62092 RMS: 0.804582 NPIX: 170


by contrast, deconv converges rapidly:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.27004
Sum: 196.85 Mean: 0.937381 Median: 0.659373 RMS: 0.759002 NPIX: 210
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.39255
Sum: 199.766 Mean: 0.951267 Median: 0.683644 RMS: 0.758766 NPIX: 210
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.42881
Sum: 200.629 Mean: 0.955377 Median: 0.690816 RMS: 0.75878 NPIX: 210

while I'm at it, gaussfits for nodeconv and reconv:
Guess: 427,1113 Fit peak: 2.90411 Background: 0.0102637 X,Y position: 426.514822,1113.354863 X,Y FWHM: 64.253564,72.615868 Angle: 0.000000
Guess: 427,1113 Fit peak: 3.03769 Background: 0.000596982 X,Y position: 426.080980,1113.634660 X,Y FWHM: 77.405551,104.292823 Angle: 329.589335
well, isn't that neat! Better than 5% agreement on the peaks. Curiously, the background is higher for nodeconv, which is false: it is directly evident that the background is higher in reconv. So I think this is just a failed fit, unfortunately. The true peak of the reconv AFGL 4029 is 3.9 Jy, so there's a huge residual


Found an error in the noisemap computations that was artificially driving up the noise around NAN pixels in undersampled maps; this probably led to major problems. I fixed it by only "flagging out" pixels with >2 NAN neighbors, i.e. so sparsely sampled that they should be ignored, I hope.... this may have affected modeling in all maps.

Realized the problems with noisemap: if the model exactly equals the weighted mean at a given pixel, the residual will be EXACTLY zero (to within numerical precision). This leads to underestimates of the noise! What we really want is an estimate of the standard deviation on the mean, which is easily computed! Just do a normal weighted sum of the difference between the data and the model (data and the mean); this is also equivalent (conveniently) to a chi^2 statistic, I think... Whoa. How did I not do this before? LETS FIND OUT I'M SURE THERE ARE AWFUL CONSEQUENCES!


Just had another idea to throw in - what if we downweight the scan edges? It won't matter for individual maps or maps of the same size if done uniformly, but it could help incorporate "pointing" maps into "real" maps more accurately.

A workaround for individual maps?

I closely examined the timestreams of 101208_ob7 as I said I would yesterday. Unfortunately, all I can do is describe the symptoms: the first deconvolution model looks good, though it isn't quite as wide as the true source (this should be OK; it is an iterative method, after all). In the second iteration, though, the deconvolution model is even smaller and lower amplitude... and it goes on like that.

Not deconvolving results in a healthy-looking clean map - pretty much what you expect and want to see.

This implies that somehow removing an incomplete deconvolved model leads to more of the source being included in the 'atmosphere' than would have been included with no model subtraction at all. I'm not sure how this is possible. In fact... I'm really quite sure that it is not.

The workaround is to only add positive changes to the model. This should 'definitely work' but may be non-convergent and assumes that the model never has anything wrong with it at any iteration. I have demonstrated that this works nicely for the two Uranus observations I tested on, but now I have to run the gamut of tests.... the first (very obvious) problem is that the background is now positive, which is dead wrong. This workaround is not viable.

Alright, so what next? I've described the symptoms and that I think they can't occur...
A closer look shows that new_astro is not being incorporated into astro_model at the second iteration. Why?


AHA! Pyflagger + find_all_points reveals the problem!

Map value: 16.939728   Weighted average: 17.476323   Unweighted Average: 524.573136
scan,bolo,time:       mapped       astro       flags      weight       scale
   3,  22,  12:     8.380408   13.561113    0.000000    0.025132    1.000000
   4, 124,  23:   822.005327   13.561113    0.000000    0.000038    1.118012
   4,  21,  38:   719.408983   13.561113    0.000000    0.000037    0.946721
   5,  20,   7:     4.470616   13.561113    0.000000    0.013303    1.400000
   5, 119,  23:   882.508303   13.561113    0.000000    0.000033    0.926887
   5, 100,  35:   327.007750   13.561113    0.000000    0.000074    1.184397
   5, 106,  38:   162.562098   13.561113    0.000000    0.000704    0.970000
   6, 116,  27:   779.075640   13.561113    0.000000    0.000033    0.891768
   8, 112,   3:   235.557390   13.561113    0.000000    0.000147    0.947130
   9,   3,  14:   966.721773   13.561113    0.000000    0.000032    1.166292
   9, 109,  41:   139.753656   13.561113    0.000000    0.000753    1.075269
  10, 104,   8:   641.121935   13.561113    0.000000    0.000050    0.927827
  10, 105,  24:     4.323228   13.561113    0.000000    0.032759    0.019022
  10,  32,  36:   847.646990   13.561113    0.000000    0.000034    1.099406
  11,  36,   9:   834.757586   13.561113    0.000000    0.000038    1.184751
  11,  76,  37:   566.851891   13.561113    0.000000    0.000040    1.111000
  12,  77,  13:   834.603090   13.561113    0.000000    0.000034    1.128464
  12,  44,  44:   335.465654   13.561113    0.000000    0.000195    2.165775
  13,  26,  17:    50.423143   13.561113    0.000000    0.004826    0.829932
  13,  75,  29:   724.884676   13.561113    0.000000    0.000042    0.923077
  14,  49,  21:   797.618990   13.561113    0.000000    0.000038    1.091918
  14,  29,  33:   743.856012   13.561113    0.000000    0.000035    1.050360
  15,  33,  13:   660.670099   13.561113    0.000000    0.000031    0.832180
  15,  53,  25:   604.174286   13.561113    0.000000    0.000047    0.889922
  15,  88,  40:     4.626476   13.561113    0.000000    0.008241    0.191489
  17,  64,  20:   778.950533   13.561113    0.000000    0.000037    1.233108
  18,  68,  30:   686.048136   13.561113    0.000000    0.000040    1.387283

Note that the lowest points have the highest weights. They DEFINITELY shouldn't. What's wrong with them?

Apparently they have NO sensitivity to the sky! What?! There were a bunch of bad bolos in Dec2010 that weren't flagged out... I wonder if that problem persists to other epochs. Still, why does it only affect pointing observations? Looking at the power spectra... the large-timescale stuff becomes less dominant when scans are longer, but the noisy spectra are still clearly noise-only. How odd.

Dropped to 112 good bolos from 134. That is much more believable. Have to go back and fix Dec09 data too...

Even after fixing the bad bolos, the model drops with iteration number. Why why why?

Well, looking at deconv_map, I've always returned the truly deconvolved version, not the reconvolved... maybe the reconvolved really is better? Again, this will have to be extensively tested, but it certainly gets rid of the obvious/dominant error that the model kept dropping off. However, FINALLY, based on how ridiculously good the reconv-deconvolved map looks, I think I'm ready to do the extensive pipeline tests. So, 10dec_caltest has been started up with all of the new bolo_params applied and the changes in place to deconv_map... let's see what happens.


After that runs, I'll have to re-run the fit_and_plot routines

Iterating to Convergence

Some sample plots of flux-vs-iteration number in the Galactic Center. The above is for a 13-pca-component map, and the plot is of the northeast peak of SGR B2. The other plots (available in postscript below) are of two other points in the GC, and the last one is of the whole of SGR B2 (it's a very large aperture).

ps version here:
http://sites.google.com/site/bolocam/pipeline/mapping/iteratetoconvergence.ps?attredirects=0

or /scratch/adam_work/plots/iteratetoconvergence.ps