Spatial Transfer Functions, revisit 4

Last report was a bit of a fiasco. There were problems all over the place I didn't understand. I still don't but I've fixed them. My best guess at this point is that a pass-by-reference led to an unacceptable modification of an image. That doesn't even make sense - there was no place it could have happened - but, there you have it.

So, going through the process step by step.

This is the effect of smoothing an image:

Note from the 3rd figure that 100% recovery isn't reached until ~700 arcseconds.


Next question: What is happening at large spatial scales in the flat-spectrum simulations?

No obvious problems there.

Hmm, no apparent problem here either, though one might ask why the two curves approach each other in sky05 (alpha=-0.5).

So it appears that the reason for the bump up at low frequencies (long wavelengths) must be because of edge effects. After much hassle, I've addressed that by cropping images.

Finally, the averaged results:

So we've got an Official Spatial Transfer Function.

However, of course, we must note that there is a dependence on the atmosphere amplitude to source amplitude ratio: it appears that large-scale structure is *easier* to recover when the atmosphere is at higher amplitude. This makes sense: it is easier to distinguish faint astrophysical signal from bright atmosphere in this case. The reason I didn't run simulations to test this more is that the S/N ratio on small scales becomes poor for the low astrophysical amplitudes.

Examining bolocat

Just in case we were wondering, the V1 bolocat is completely inconsistent with a lognormal distribution, but is perfectly consistent (or... at least reasonably consistent...) with a power law distribution.

These plots show power-law fits (red) and lognormal fits (blue) to the data (black). It's pretty obvious that the lognormal is a bad fit, but in case you're unconvinced, the ks test for the "source flux" has a probability 1.6e-7, and it is the highest likelihood by 17 orders of magnitude out of the 4 flux types.

By contrast, the simulations are on average (though not uniformly) more consistent with lognormal than powerlaw distributions:

Even in those examples where the KS test is slightly more favorable for the powerlaw distribution, the lognormal is a pretty good fit, and the failure points for the two distributions are in about the same place. The smoothness of the lognormal distribution is required to reproduce the observed distribution.

Note that the first 4 plots are for the whole BGPS survey. What about an individual field? For obvious reasons, I choose l30 again.

This gets to be a little more interesting - apparently the "source flux" has a tendency to pick up the power-law distributed background structure, since it is consistent with a lognormal (but note that it is also consistent with a powerlaw! The ks test doesn't really say definitively which is better). Although the fits look bad at high flux, note that this is a log-log plot and therefore the difference in probability is rather small.

What does this all indicate? It's not entirely clear whether individual fields are genuinely more lognormally-distributed or whether the number statistics are just worse. However, even the source flux is consistent with a power-law, while many realizations of the simulations are not.

Therefore, we should perform the next logical test - add point sources drawn from a power-law distribution (and a log-normal distribution?) and see what bolocat retrieves. We can at least say now that the point source contribution cannot be ignored, since there is no power-law distribution that can reproduce the observed bolocat flux distribution.

Bolocat examination part 1

This figure shows the flux recovery of bolocat. The Y axis shows the Input flux (green) and the Filtered flux (blue/cyan/purple) with a few different large-scale filters. The red line is the 1-1 line. Obviously, bolocat doesn't recover everything that was there - we removed a lot of flux, so recovery fractions are in the few % range. However, bolocat does a much better job than the simple filter at recovering positive fluxes. Therefore, the filter function still isn't good enough.

Fourier transforms are not commutative or distributive

Problem with yesterday's work:

A*B = IFT(FT(A)*FT(B))
A*B - A*C != IFT(FT(A)*(FT(B)-FT(C)))

If instead of yesterday's "bandpass filter" you use the convolution - convolution 'filter', the agreement in real space and most of fourier space is much better:

though map20 clearly reproduces some structures better (higher) but is overall less powerful than the filtered map.

Testing out analytic filter functions

I've attempted to model the spatial filter function as a gaussian (or PSF) plus an inverse gaussian. i.e., the high-spatial-frequency components are smoothed with the PSF, and the low spatial frequency components are convolved with a (1-gaussian) high-pass filter.

First, the mildly good news: With a 300" FWHM large-scale cutoff, the filter PSD reasonably resembles the iterative map PSD:

Luckily, the double-filter goes a very long way in explaining the scale-free flux loss. In the following diagram, I show the effect of the filter compared to the input map.

The filter only recovers about 75% of the flux at ANY wavenumber. The map does slightly worse at high frequencies, which I can't explain yet.

These show the recovery fraction of the iterative maps, a gaussian smoothing function with FWHM=33", and the mid-pass-filter. Map20 (no smooth) has a lot of additional "noise power" at high spatial frequencies; if it wasn't for the telescope filter function, we would apparently have pretty good high-frequency recovery. Hmph.

Note that map20 is higher than the filter at some intermediate frequencies, but quite a bit lower at higher frequencies. Also note the moderately poor agreement between the 'smoothed' and 'smoothed (theory)' lines.

Finally, look at the comparison between map20 and fiiltered. The agreement is not bad for positive points; filtered is apparently slightly higher but that can be adjusted. The problem: the filter forces some structures that are negative or zero to be positive. For example, look at the feature at 210,300 that is negative in Map20 but positive in Filtered. In the real (input) map, this feature is lower than its surroundings - it is legitimately negative.

Direct comparison of column-density power spectra

I've multiplied the 13CO integrated data cube, the Herschel 500 micron*, and the BGPS v1.0 and v2.0 by the appropriate conversion factor to get the maps into units of column density assuming T=20K and some opacity for the dust maps. BGPS v1 has been multiplied by the 1.5 "correction" factor.

* The Herschel maps are arbitrarily scaled; I didn't derive an actual column conversion but just guess-and-checked once or twice until I got something pretty close.

The power spectra look pretty outstanding:

The bumps and wiggles in the 50-200" range are quite well-matched in Herschel and Bolocam. Some map edge effects are visible in the Herschel maps, resulting in multi-frequency bumps at small spatial scales. The Herschel noise floor is also quite evidently lower. Also noteworthy is that BGPS v1 (scaled up by 1.5) matches the others "pretty well" but is a worse match, in general, to the Herschel data than is the BGPS v2.

Here are some zoomed-in plots on the inverse scale:

And finally, the 13CO data is totally unrepresentative of the dust data. There is very little agreement on any scale. This may imply that the 13CO and/or dust temperature is too high, as a decreased T_D or T_ex will decrease 13CO column and increase dust column. However, it also raises a question - on what scales should dust and CO agree?

This is getting into some real science, perhaps. The shapes of the CO and dust power spectra disagree: the CO is pretty well-fit by a power law, while the dust is not. What hypotheses can explain this?

1. There is a systematic temperature difference / preference in which CO or dust is hotter on the largest scales.
   
   -Dust is probably warmer on larger scales, however CO should be less abundant / more readily dissociated on these largest, most diffuse scales. CO shouldn't exist (or at least, should be underabundant) in regions with high temperature. Maybe? This probably needs to be quantified.
   
2. There is a systematic dust opacity difference on large scales resulting in lower dust emission.
   
   -This is almost certainly true: the dust population increases in opacity with age, following OH94. Dust on the largest spatial scales should not have coagulated / collected ice, leading to a lower opacity on the largest scales
   
   -This may also be true even though CO is present: dust coagulation is less efficient than CO formation at n~10^3-10^4 (I think - again, off the cuff, but consistent with OH94)
   
3. The CO overestimates all scales, either because of incorrect bulk abundance or temperature considerations.
   
   This is problematic..... if you drop the CO values at all scales, it becomes deficient in the 50-200 arcsecond range, where the dust measurements agree quite well
   
4. There is a preferred distance in both images
   
   -It is not clear that the observed effects would occur because of this
   
   -It is also quite evident from other analyses that there IS a preferred velocity, at least, and it completely dominates all others and has the same shape as the integrated power spectrum. So a distance effect is most likely ruled out.
   

Power Spectra of 13CO and Bolocam

Examples of power-spectra in the left panel:

The BGPS power law is fitted only over the "valid" range, which is shown in black. The BGPS power spectrum agrees pretty well with the Herschel power spectrum. Both are substantially steeper than (any) of the 13CO power spectra. Unfortunately, this somewhat calls into question any comparison between the data sets - it seems most likely that 13CO and dust actually trace different mass, particularly since both Herschel and Bolocam show a "bump" centered at 6' (10^-1 image sizes) that is not evident in 13CO.

PPS comparison completed

The PPS scale factor seems to hold for many different (and ridiculous) aperture sizes.  The full results are here.

PPS analysis: suggests a possible solution to the discrepancy

The comparisons I mentioned in the previous post are sort of done.  They are pretty suggestive of a solution to the "flux recovery problem" we think must be true.  However, even if it is a solution, it doesn't really solve the problem completely.

It looks like v1.0 should be scaled up by a factor of 1.3-1.4 (not 1.5).  v2.0 is consistent with the PPS sources to within 5%, and might even be slightly too high.

The comparison was done by taking the flux in a 60" radius aperture (equivalent to bolocat 120" diameter apertures) and subtracting off the background measured in a 120" radius annulus around the source.  Without the background subtraction, these numbers would look very different: in the science fields, most of the sources sit on an extended background.  Even though the "background flux" isn't recovered in the PPS fields, it should contribute to the source background because it is involved in the atmosphere subtraction (it's sort of "already subtracted" so you have to subtract from the science fields).

Next step: direct comparison between v1.0 and v2.0.  Pixel by pixel, aperture, and powerspectrum

Eimers project

Project for Marc Eimers:
Determine velocities to molecular clouds by a variety of methods. Start with l=30

1. Find archival data, particularly from the JCMT, for each core.
2. Compare morphologically to 13CO from GRS
3. Find Vizier data

filling factors

to do: histogram per square degree, plot cumulative > some bin number (100 mJy, 300 mJy, 1 Jy) vs l, then do it again for b=+/-.1, +/-.3.