Weird problem

I'm remapping everything, and there's a really strange situation in ic1396... only one field has a source that doesn't have rotation problems, every other observation is clearly improperly rotated. The weird thing is that it's NOT the one you'd expect from the information below:


readcol,'/Volumes/disk2/sliced/infiles/ic1396_infile.txt',filelist,format='A'
for i=0,6 do begin & ncdf_varget_scale,filelist[i],'array_params',ap & print,filelist[i],ap & endfor
/Volumes/disk2/sliced/ic1396/070911_o19_raw_ds5.nc 7.70000 31.2000 70.7000 0.00000 0.00000
/Volumes/disk2/sliced/ic1396/070912_o26_raw_ds5.nc 7.70000 31.2000 84.0000 0.00000 0.00000
/Volumes/disk2/sliced/ic1396_d/070913_o19_raw_ds5.nc 7.70000 31.2000 84.0000 0.00000 0.00000
/Volumes/disk2/sliced/ic1396_l/070913_o17_raw_ds5.nc 7.70000 31.2000 84.0000 0.00000 0.00000
/Volumes/disk2/sliced/ic1396_r/070913_o18_raw_ds5.nc 7.70000 31.2000 84.0000 0.00000 0.00000
/Volumes/disk2/sliced/ic1396_u/070913_o20_raw_ds5.nc 7.70000 31.2000 84.0000 0.00000 0.00000


One ofthese things is not like the others... but it's 070913_o18_raw_ds5.nc, not /Volumes/disk2/sliced/ic1396/070911_o19_raw_ds5.nc

Progress, but still ds1-ds5 issues

ds1 and ds5 agree pretty well with the recent upgrades to delining and deconvolution. However, there are still counterexamples, e.g. 101208_o13, in which ds5 < ds1:


The 'fitted' values agree better than the 'measured' values now that NANs are treated properly.


Spent a few hours today trying to figure out if weighting can explain the difference between ds1 and ds5; it appears to make up for most of it so I'm doing some more experiments. Why is there so much parameter space? Why can't weighting just work? It doesn't....

also wasted a few hours trying to write a python drizzling algorithm, which unfortunately is impossible so I had to resort to an inefficient for loop.

Finally got some minor results. It really looks like there is a trend pushing up the recovered flux (i.e. higher volts/Jy) for ds5 over ds1. There is a discrepancy between map types for ds1 but not for ds5, which is actually backwards from what I would have expected, since ds1 will get better-sampled maps.

Luckily, the difference between peak fitting and "measuring" results in very small (insignificant) changes to the calibration curve (recall fitting is direct gaussian fitting; 'measuring' is using the gaussian-fit width and total flux in an ellipse to infer a peak assuming a point source):

Since this work has all been done for the 'bootstrapping' observations that are supposed to tell us if different map sizes are compatible, I have included the map sizes in the diagrams now. However, to really understand the ds1/ds5 difference, there are much better data sets, which I'm now reprocessing using the new and improved methods.

(the Whole BGPS is also processing with the new methods in the background, though since the methods are being updated live there may be more changes and it will have to be re-run.... initial looks at W5 are BAD but L030 is GOOD (bordering on amazing))

Careful comparison of ds1 and ds5

I looked very closely at the timestream and maps of 101208_o11 and had a pretty hard time figuring out why the data were different, but it looked like the data really did differ on a point-by-point basis (according to pyflagger). The only conclusion I was able to draw is that the scaling must be off. I realized that the scaling was being done before delining. I moved scaling from readall_pc to premap, and it brought at least this one source into agreement. Time to run ds1-ds5 comparisons again!

(this means that ds1 data MUST have deline run on it, but ds5 data doesn't really need it)

Here are examples of ds1 and ds5 timestreams, with and without scaling, and ds1 with and without delining:

Trying to bootstrap

I've concluded, based on previous posts http://bolocam.blogspot.com/2011/02/downsampling-why-is-dec-2010-different.html, http://bolocam.blogspot.com/2011/03/revisiting-calibration-yet-again.html, and http://bolocam.blogspot.com/2011/03/workaround-for-individual-maps.html, that ds5 is a problem primarily for undersampled images, i.e. those taken in the normal mapping mode. This makes bootstrapping a bit tricky.

There are two options:
1. Map Uranus and AFGL 4029 both in Volts and figure out what flux density AFGL 4029 must have to lie on that curve
2. Map Uranus and compute a calibration curve, apply that calibration curve to AFGL 4029, and then compare derived flux densities.

Both have the major problem that the individual AFGL 4029 maps will forcibly be undersampled if I use ds5 data (which is normally OK, according to the first paragraph). In the second case, it is possible to co-add the images and get around the under-sampling issue, while in the first case it is not because of the dependence on total loading (MEANDC).

The real problem is that the whole goal of these observations was to compare the different observing methods and see if they agree (1x1, 3x1, pointing, etc.) since the pointing-style observations were used to calibrate the others. But if the 1x1s are just straight-up unreliable, how can we do the comparison? I think the co-added AFGL 4029 is the only option, but then how do I test if it's correct? It would be really nice to have AFGL 4029 observed with both scan types...


Alright, onto the data. After last week's fix of the bad bolos, I really hope ds1 and ds5 agree. However, first glance at the cal curves says they don't. ds1 and ds2 agree, but ds5 is different.

After checking them out with ds9 *ds[15]*13pca*_map10.fits -scale limits -1 1000 -log -cmap hsv -match colorbars -match scales -match frames wcs &, it appears that the _mask_ data is all... wrong, somehow. That's OK, I want to discard the mask data anyway, so I'm happy to NOT spend time debugging it.

Even after careful examination showing that the fits look good - and noting that the fluxes look pretty much the same - the calibration curves still look rather different. Unfortunately I had to spend 3 hours debugging IDL plotting commands; I want to show the fits each time and save them as postscripts. What does "xyouts" with "/device,/normal" do? I thought that should plot x,y text at the coordinates specified in the plot window... but no, that is JUST /normalize.

Anyway, realized that centroid_map treated NANs as zero. Added ERR keyword (with a reasonable estimate of the error) in centroid_map to ignore NANs. It looks like improper treatment of NANs is responsible for a lot of the scatter seen in the calibration plots.

There is a substantial difference between the "fitted" peak and the "measured" peak (the latter computed by taking the sum of the pixels divided by the area of the fitted gaussian). It looks like the "measured" version is more robust, at first glance. However, unfortunately, for 101208_o11, the difference between ds1 and ds5 exists in both quantities. I will have to examine timestreams now... ARGH.

Well, the timestreams show... that indeed the model is lower in ds1, but not why. The "remainder" (new_astro; the stuff that never gets incorporated into the model but DOES get incorporated into the map) appears to be the same in both. Similarly, there is little to no flux in the PCA atmosphere, so it's not simply being cleaned out. Where is the flux going or coming from?

A workaround for individual maps?

I closely examined the timestreams of 101208_ob7 as I said I would yesterday. Unfortunately, all I can do is describe the symptoms: the first deconvolution model looks good, though it isn't quite as wide as the true source (this should be OK; it is an iterative method, after all). In the second iteration, though, the deconvolution model is even smaller and lower amplitude... and it goes on like that.

Not deconvolving results in a healthy-looking clean map - pretty much what you expect and want to see.

This implies that somehow removing an incomplete deconvolved model leads to more of the source being included in the 'atmosphere' than would have been included with no model subtraction at all. I'm not sure how this is possible. In fact... I'm really quite sure that it is not.

The workaround is to only add positive changes to the model. This should 'definitely work' but may be non-convergent and assumes that the model never has anything wrong with it at any iteration. I have demonstrated that this works nicely for the two Uranus observations I tested on, but now I have to run the gamut of tests.... the first (very obvious) problem is that the background is now positive, which is dead wrong. This workaround is not viable.

Alright, so what next? I've described the symptoms and that I think they can't occur...
A closer look shows that new_astro is not being incorporated into astro_model at the second iteration. Why?


AHA! Pyflagger + find_all_points reveals the problem!

Map value: 16.939728   Weighted average: 17.476323   Unweighted Average: 524.573136
scan,bolo,time:       mapped       astro       flags      weight       scale
   3,  22,  12:     8.380408   13.561113    0.000000    0.025132    1.000000
   4, 124,  23:   822.005327   13.561113    0.000000    0.000038    1.118012
   4,  21,  38:   719.408983   13.561113    0.000000    0.000037    0.946721
   5,  20,   7:     4.470616   13.561113    0.000000    0.013303    1.400000
   5, 119,  23:   882.508303   13.561113    0.000000    0.000033    0.926887
   5, 100,  35:   327.007750   13.561113    0.000000    0.000074    1.184397
   5, 106,  38:   162.562098   13.561113    0.000000    0.000704    0.970000
   6, 116,  27:   779.075640   13.561113    0.000000    0.000033    0.891768
   8, 112,   3:   235.557390   13.561113    0.000000    0.000147    0.947130
   9,   3,  14:   966.721773   13.561113    0.000000    0.000032    1.166292
   9, 109,  41:   139.753656   13.561113    0.000000    0.000753    1.075269
  10, 104,   8:   641.121935   13.561113    0.000000    0.000050    0.927827
  10, 105,  24:     4.323228   13.561113    0.000000    0.032759    0.019022
  10,  32,  36:   847.646990   13.561113    0.000000    0.000034    1.099406
  11,  36,   9:   834.757586   13.561113    0.000000    0.000038    1.184751
  11,  76,  37:   566.851891   13.561113    0.000000    0.000040    1.111000
  12,  77,  13:   834.603090   13.561113    0.000000    0.000034    1.128464
  12,  44,  44:   335.465654   13.561113    0.000000    0.000195    2.165775
  13,  26,  17:    50.423143   13.561113    0.000000    0.004826    0.829932
  13,  75,  29:   724.884676   13.561113    0.000000    0.000042    0.923077
  14,  49,  21:   797.618990   13.561113    0.000000    0.000038    1.091918
  14,  29,  33:   743.856012   13.561113    0.000000    0.000035    1.050360
  15,  33,  13:   660.670099   13.561113    0.000000    0.000031    0.832180
  15,  53,  25:   604.174286   13.561113    0.000000    0.000047    0.889922
  15,  88,  40:     4.626476   13.561113    0.000000    0.008241    0.191489
  17,  64,  20:   778.950533   13.561113    0.000000    0.000037    1.233108
  18,  68,  30:   686.048136   13.561113    0.000000    0.000040    1.387283

Note that the lowest points have the highest weights. They DEFINITELY shouldn't. What's wrong with them?

Apparently they have NO sensitivity to the sky! What?! There were a bunch of bad bolos in Dec2010 that weren't flagged out... I wonder if that problem persists to other epochs. Still, why does it only affect pointing observations? Looking at the power spectra... the large-timescale stuff becomes less dominant when scans are longer, but the noisy spectra are still clearly noise-only. How odd.

Dropped to 112 good bolos from 134. That is much more believable. Have to go back and fix Dec09 data too...

Even after fixing the bad bolos, the model drops with iteration number. Why why why?

Well, looking at deconv_map, I've always returned the truly deconvolved version, not the reconvolved... maybe the reconvolved really is better? Again, this will have to be extensively tested, but it certainly gets rid of the obvious/dominant error that the model kept dropping off. However, FINALLY, based on how ridiculously good the reconv-deconvolved map looks, I think I'm ready to do the extensive pipeline tests. So, 10dec_caltest has been started up with all of the new bolo_params applied and the changes in place to deconv_map... let's see what happens.


After that runs, I'll have to re-run the fit_and_plot routines

Revisiting calibration yet again

The recent hiatus for paper revisions has, unfortunately, come to an end.

Re-examining my work, I did quite a lot but encountered many dead-ends.

First, we would very much like to use an *identical* reduction process on both the calibration data and the science data. That way, we could feel very confident that the reduction process isn't introducing any weird artifacts.

Unfortunately, I discovered early on that using ds5 data, 13 pca components, and n>1 iterations resulted in strange shape and flux conservation failures. These errors do NOT occur in co-added maps; they are unique to single-observation scans (though I don't recollect whether 2 scans is enough or if you need more).

I spent many hours banging my head against this problem and have never gotten a satisfactory solution. But perhaps it's time to approach it again. The map00 images look MUCH rounder and generally better than the map10 images.

So, the problem I need to examine is the iterative process. Why does it fail for single images? Is it something about the noise properties? model00 looks fine... what gets put into the timestream? Examining timestreams is a terrible and horrendous process... but what else can I do?

The next step will be to examine the timestreams of a particular observation. I think a good choice is 101208_ob7; the next observation, 101208_ob8 was a large-area map and it looks fine (i.e., it improves with iteration). So I can start looking at the effects of polysub, iteration, etc. on this particular source.

Of course, the stupid trick with the pipeline - every time - is that "fixing" a problem for one source has a nasty tendency to break it for all other sources. That's why there are so many flags that can be passed around. Still, this is the approach I have to take...

Downsampling - Why is Dec 2010 different?

As you'll recall from my previous post (and references therein...), the 2005 Orion data shows a discrepancy between ds1 and ds5 data in which the ds1 data is significantly (~10%) higher than the ds5 data. However, the 2010 Uranus observations show much larger discrepancies between ds1 and ds5 favoring the ds5 data! Because that was somewhat unbelievable to me, I ran ds1-ds5 comparisons on Uranus data from other epochs, and discovered that ds1>ds5 uniformly (also, it looks a LOT better).

So, the question remains, WHY is the Dec 2010 data brighter in ds5? More confusing to me, why do the ds5 PSFs from 2010 look so reasonable, while the ds5 PSFs from all earlier epochs look terrible?

For example, I use the observations 070727_o31 and _o32. These show clearly the blurring and flux-loss that happens when ds5 data are used for 'normal' point-sources. The disagreement is, surprisingly, worse for the co-added image, though the correlation is better... I'm not entirely clear what that means besides there being a positional offset between the individual maps that is not entirely believable.

However, the point is that these maps do not share the properties of the 2010 maps, in which the ds5 source is brighter and possibly even more peaked than the ds1 source, despite the ds1 data having lower noise (as you'd expect):

What could be boosting the flux in the Dec 2010 downsampled data? Is there something about the lines - before downsampling - that is changing the relative scaling? Perhaps I can answer this with the w5 maps... but I think that's unlikely...