STF measurement attempts, round 3?

Below are plots of the spatial transfer function for a variety of simulation parameters. Each line represents the median (i.e., mean of the center 2 of 4) spatial transfer function of 4 realizations of the listed parameters.

A few unfortunate details are apparent:

1. The STF is background dependent, though not strongly
   
2. The STF appears to never exceed 85%, and be more typically 75%. If this is not matched by point-source PSDs, an explanation for the flux discrepancy becomes possible
   
3. The 50% recovery point is much closer in than previously stated, at closer to 150" than 300".
   

Note for comparison that the recovery fraction for point sources is much more nearly 1, at least in the calibrator type maps. We need to wait for Jared's simulations with point sources in a full size map to confirm this, but it seems pretty likely that the STF is different for point sources and extended structures.

Examining bolocat

Just in case we were wondering, the V1 bolocat is completely inconsistent with a lognormal distribution, but is perfectly consistent (or... at least reasonably consistent...) with a power law distribution.

These plots show power-law fits (red) and lognormal fits (blue) to the data (black). It's pretty obvious that the lognormal is a bad fit, but in case you're unconvinced, the ks test for the "source flux" has a probability 1.6e-7, and it is the highest likelihood by 17 orders of magnitude out of the 4 flux types.

By contrast, the simulations are on average (though not uniformly) more consistent with lognormal than powerlaw distributions:

Even in those examples where the KS test is slightly more favorable for the powerlaw distribution, the lognormal is a pretty good fit, and the failure points for the two distributions are in about the same place. The smoothness of the lognormal distribution is required to reproduce the observed distribution.

Note that the first 4 plots are for the whole BGPS survey. What about an individual field? For obvious reasons, I choose l30 again.

This gets to be a little more interesting - apparently the "source flux" has a tendency to pick up the power-law distributed background structure, since it is consistent with a lognormal (but note that it is also consistent with a powerlaw! The ks test doesn't really say definitively which is better). Although the fits look bad at high flux, note that this is a log-log plot and therefore the difference in probability is rather small.

What does this all indicate? It's not entirely clear whether individual fields are genuinely more lognormally-distributed or whether the number statistics are just worse. However, even the source flux is consistent with a power-law, while many realizations of the simulations are not.

Therefore, we should perform the next logical test - add point sources drawn from a power-law distribution (and a log-normal distribution?) and see what bolocat retrieves. We can at least say now that the point source contribution cannot be ignored, since there is no power-law distribution that can reproduce the observed bolocat flux distribution.

Bolocat examination part 1

This figure shows the flux recovery of bolocat. The Y axis shows the Input flux (green) and the Filtered flux (blue/cyan/purple) with a few different large-scale filters. The red line is the 1-1 line. Obviously, bolocat doesn't recover everything that was there - we removed a lot of flux, so recovery fractions are in the few % range. However, bolocat does a much better job than the simple filter at recovering positive fluxes. Therefore, the filter function still isn't good enough.

when things go really really bad

that's all.

Open Questions I'm trying to answer

I still haven't dealt with http://bolocam.blogspot.com/2011/07/additional-problems.html, or essentiall any of http://bolocam.blogspot.com/2011/07/minor-ongoing-problems.html.

However, they're probably related to this problem:

% CLEAN_ITER_STRUCT: There were 1 bolometers with high weights: 13.5047 indices: 142 or flagged indices 142

which indicates that in a simulation in which there can be no outliers (in terms of weight/scale), there is one being rejected as an outlier. That indicates that weights are being computed incorrectly, despite the fact that scales look right (so far):

Relsens calibration: Scaled to bolometer # 0
% RELSENS_CAL_PCA: There were 0 NAN scales and 0 very low scales
% RELSENS_CAL_PCA: Scale Median/Mad: 1.0005510+/- 0.010990833 led to 0 scales set to zero for a total of 0 bad bolos
% RELSENS_CAL_PCA: Scales avg+/-std = 1.0007304355062783+/- 0.0103252880998329
Relsens calibration: Scaled to bolometer # 0
% RELSENS_CAL_PCA: There were 0 NAN scales and 0 very low scales
% RELSENS_CAL_PCA: Scale Median/Mad: 0.99955294+/- 0.0093122063 led to 0 scales set to zero for a total of 0 bad bolos
% RELSENS_CAL_PCA: Scales avg+/-std = 0.9990063123041220+/- 0.0096663438876613


Fourier transforms are not commutative or distributive

Problem with yesterday's work:

A*B = IFT(FT(A)*FT(B))
A*B - A*C != IFT(FT(A)*(FT(B)-FT(C)))

If instead of yesterday's "bandpass filter" you use the convolution - convolution 'filter', the agreement in real space and most of fourier space is much better:

though map20 clearly reproduces some structures better (higher) but is overall less powerful than the filtered map.

Testing out analytic filter functions

I've attempted to model the spatial filter function as a gaussian (or PSF) plus an inverse gaussian. i.e., the high-spatial-frequency components are smoothed with the PSF, and the low spatial frequency components are convolved with a (1-gaussian) high-pass filter.

First, the mildly good news: With a 300" FWHM large-scale cutoff, the filter PSD reasonably resembles the iterative map PSD:

Luckily, the double-filter goes a very long way in explaining the scale-free flux loss. In the following diagram, I show the effect of the filter compared to the input map.

The filter only recovers about 75% of the flux at ANY wavenumber. The map does slightly worse at high frequencies, which I can't explain yet.

These show the recovery fraction of the iterative maps, a gaussian smoothing function with FWHM=33", and the mid-pass-filter. Map20 (no smooth) has a lot of additional "noise power" at high spatial frequencies; if it wasn't for the telescope filter function, we would apparently have pretty good high-frequency recovery. Hmph.

Note that map20 is higher than the filter at some intermediate frequencies, but quite a bit lower at higher frequencies. Also note the moderately poor agreement between the 'smoothed' and 'smoothed (theory)' lines.

Finally, look at the comparison between map20 and fiiltered. The agreement is not bad for positive points; filtered is apparently slightly higher but that can be adjusted. The problem: the filter forces some structures that are negative or zero to be positive. For example, look at the feature at 210,300 that is negative in Map20 but positive in Filtered. In the real (input) map, this feature is lower than its surroundings - it is legitimately negative.

Masked edge effects?

I raised the possibility that the ratty edges of a scan could affect the power spectrum measurements, leading to a scale-free power loss. This doesn't appear to happen... instead, the masking adds some small-scale power and apparently a small amount of large-scale power.

Spatial Transfer Functions

The majority of the past week has been dedicated to debugging; it looks like cross-scanned simulations finally work.

The plot below is a derivation of the spatial transfer function for a number of different intrinsic sky power-law power spectra.

Justifying the above plot is essential.

First, the very steep power-laws [-3 in the example below] show a recovery fraction >1. This is simply because their S/N was inadequate - the output power spectrum is nearly flat, but at a level higher than the sky.

Second, the most plausible power-laws [-1.5 in the example below] show pretty good recovery (90-95% over the relevant range):

There are some "white" power losses, particularly in the flatter power-spectra. My best guess is that this has something to do with the relative scales being offset from a mean of 1, but so far all tests to show that that is the cause have in fact shown no problems at all. What else could cause a scale-independent power loss?

Also, the flat power spectrum (and inverted) aren't quite flat because I impose a "galactic scale height" on them. Should I stop doing that?

additional problems

There are a few cases in the L=30 field that look awful. 050706_o31 probably observed the inside of the dish. 070727_ob6 shows some streaking that I can't easily explain... though it appears that there are some bad bolos that need to get flagged out. I wonder if that's systematic over 070727 observations....

[minor] Ongoing problems...

Since I have many tasks running in parallel, I need to summarize them sometimes...

1. There are many "bad" observations that haven't been placed in "bad" directories. A list of the error messages generated is here: Making Infiles wiki
   
2. There may be streaking in the BGPS ds2 images of l030 and l032. Still trying to re-run cleanly to find out
   
3. There may be unflagged high-points in l030 and l032. Also under examination
   
4. Simulations have been generating bulk-offset outputs; my suspicion was that the relative scales were being set incorrectly because astro dominated atmo, so I bumped up the atmo scale. The tests have run but I haven't examined the outputs
   
5. V1 sims have shown streaking, possibly because of the previous bullet, but certainly (in part) because cross-scans haven't worked