Getting rid of haloes

Haloes are when images look like this:

instead of this, as they should:

Things to try:

1. Pass /return_deconv to deconv_map
   
2. Pass /linear to deconv_map
   
3. Disable deconvolve - deconvolve=0
   

In l123 & l169, at least, /return_deconv worked

Deconvolution vs. Not

In the W5 maps, everywhere except AFGL 4029 agrees to within about 20% (better than our supposed offset) independent of deconvolution scheme / modeling scheme, but AFGL 4029 disagrees by a factor of 2-3, indicating a severe dependence on map size.

Here's the AFGL 4029 comparison originally:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.1515
Sum: 194.028 Mean: 1.03758 Median: 0.78441 RMS: 0.760825 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 5.77585
Sum: 137.481 Mean: 0.735194 Median: 0.517222 RMS: 0.663245 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 14.4043
Sum: 342.863 Mean: 1.83349 Median: 1.62579 RMS: 0.820079 NPIX: 187

Here it is after flagging out another bad bolo:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.14533
Sum: 193.882 Mean: 1.0368 Median: 0.783409 RMS: 0.764047 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 10.1518
Sum: 241.642 Mean: 1.2922 Median: 1.06321 RMS: 0.777165 NPIX: 187
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 13.9344
Sum: 331.677 Mean: 1.77367 Median: 1.56277 RMS: 0.820139 NPIX: 187

order is deconvtest, nodeconvtest, 'reconv'

deconvtest has virtually no change. nodeconvtest goes up by a LOT... must be something about the weighting. reconv drops, which is good, but not enough...

curiously, it is not clear that nodeconv converges: [10, 15, 20 iters]
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.70939
Sum: 207.308 Mean: 1.34615 Median: 1.09348 RMS: 0.760452 NPIX: 154
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 9.14028
Sum: 217.564 Mean: 1.41275 Median: 1.17223 RMS: 0.759027 NPIX: 154
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 9.42483
Sum: 224.337 Mean: 1.45673 Median: 1.2172 RMS: 0.758978 NPIX: 154

similarly, reconv does not converge:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 11.5623
Sum: 275.214 Mean: 1.61891 Median: 1.36357 RMS: 0.809771 NPIX: 170
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 12.5848
Sum: 299.553 Mean: 1.76208 Median: 1.51426 RMS: 0.807963 NPIX: 170
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 13.3102
Sum: 316.82 Mean: 1.86365 Median: 1.62092 RMS: 0.804582 NPIX: 170


by contrast, deconv converges rapidly:
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.27004
Sum: 196.85 Mean: 0.937381 Median: 0.659373 RMS: 0.759002 NPIX: 210
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.39255
Sum: 199.766 Mean: 0.951267 Median: 0.683644 RMS: 0.758766 NPIX: 210
BMAJ: 0.00916667 BMIN: 0.00916667 PPBEAM: 23.8028 SUM/PPBEAM: 8.42881
Sum: 200.629 Mean: 0.955377 Median: 0.690816 RMS: 0.75878 NPIX: 210

while I'm at it, gaussfits for nodeconv and reconv:
Guess: 427,1113 Fit peak: 2.90411 Background: 0.0102637 X,Y position: 426.514822,1113.354863 X,Y FWHM: 64.253564,72.615868 Angle: 0.000000
Guess: 427,1113 Fit peak: 3.03769 Background: 0.000596982 X,Y position: 426.080980,1113.634660 X,Y FWHM: 77.405551,104.292823 Angle: 329.589335
well, isn't that neat! Better than 5% agreement on the peaks. Curiously, the background is higher for nodeconv, which is false: it is directly evident that the background is higher in reconv. So I think this is just a failed fit, unfortunately. The true peak of the reconv AFGL 4029 is 3.9 Jy, so there's a huge residual


Found an error in the noisemap computations that was artificially driving up the noise around NAN pixels in undersampled maps; this probably led to major problems. I fixed it by only "flagging out" pixels with >2 NAN neighbors, i.e. so sparsely sampled that they should be ignored, I hope.... this may have affected modeling in all maps.

Realized the problems with noisemap: if the model exactly equals the weighted mean at a given pixel, the residual will be EXACTLY zero (to within numerical precision). This leads to underestimates of the noise! What we really want is an estimate of the standard deviation on the mean, which is easily computed! Just do a normal weighted sum of the difference between the data and the model (data and the mean); this is also equivalent (conveniently) to a chi^2 statistic, I think... Whoa. How did I not do this before? LETS FIND OUT I'M SURE THERE ARE AWFUL CONSEQUENCES!


Just had another idea to throw in - what if we downweight the scan edges? It won't matter for individual maps or maps of the same size if done uniformly, but it could help incorporate "pointing" maps into "real" maps more accurately.

A workaround for individual maps?

I closely examined the timestreams of 101208_ob7 as I said I would yesterday. Unfortunately, all I can do is describe the symptoms: the first deconvolution model looks good, though it isn't quite as wide as the true source (this should be OK; it is an iterative method, after all). In the second iteration, though, the deconvolution model is even smaller and lower amplitude... and it goes on like that.

Not deconvolving results in a healthy-looking clean map - pretty much what you expect and want to see.

This implies that somehow removing an incomplete deconvolved model leads to more of the source being included in the 'atmosphere' than would have been included with no model subtraction at all. I'm not sure how this is possible. In fact... I'm really quite sure that it is not.

The workaround is to only add positive changes to the model. This should 'definitely work' but may be non-convergent and assumes that the model never has anything wrong with it at any iteration. I have demonstrated that this works nicely for the two Uranus observations I tested on, but now I have to run the gamut of tests.... the first (very obvious) problem is that the background is now positive, which is dead wrong. This workaround is not viable.

Alright, so what next? I've described the symptoms and that I think they can't occur...
A closer look shows that new_astro is not being incorporated into astro_model at the second iteration. Why?


AHA! Pyflagger + find_all_points reveals the problem!

Map value: 16.939728   Weighted average: 17.476323   Unweighted Average: 524.573136
scan,bolo,time:       mapped       astro       flags      weight       scale
   3,  22,  12:     8.380408   13.561113    0.000000    0.025132    1.000000
   4, 124,  23:   822.005327   13.561113    0.000000    0.000038    1.118012
   4,  21,  38:   719.408983   13.561113    0.000000    0.000037    0.946721
   5,  20,   7:     4.470616   13.561113    0.000000    0.013303    1.400000
   5, 119,  23:   882.508303   13.561113    0.000000    0.000033    0.926887
   5, 100,  35:   327.007750   13.561113    0.000000    0.000074    1.184397
   5, 106,  38:   162.562098   13.561113    0.000000    0.000704    0.970000
   6, 116,  27:   779.075640   13.561113    0.000000    0.000033    0.891768
   8, 112,   3:   235.557390   13.561113    0.000000    0.000147    0.947130
   9,   3,  14:   966.721773   13.561113    0.000000    0.000032    1.166292
   9, 109,  41:   139.753656   13.561113    0.000000    0.000753    1.075269
  10, 104,   8:   641.121935   13.561113    0.000000    0.000050    0.927827
  10, 105,  24:     4.323228   13.561113    0.000000    0.032759    0.019022
  10,  32,  36:   847.646990   13.561113    0.000000    0.000034    1.099406
  11,  36,   9:   834.757586   13.561113    0.000000    0.000038    1.184751
  11,  76,  37:   566.851891   13.561113    0.000000    0.000040    1.111000
  12,  77,  13:   834.603090   13.561113    0.000000    0.000034    1.128464
  12,  44,  44:   335.465654   13.561113    0.000000    0.000195    2.165775
  13,  26,  17:    50.423143   13.561113    0.000000    0.004826    0.829932
  13,  75,  29:   724.884676   13.561113    0.000000    0.000042    0.923077
  14,  49,  21:   797.618990   13.561113    0.000000    0.000038    1.091918
  14,  29,  33:   743.856012   13.561113    0.000000    0.000035    1.050360
  15,  33,  13:   660.670099   13.561113    0.000000    0.000031    0.832180
  15,  53,  25:   604.174286   13.561113    0.000000    0.000047    0.889922
  15,  88,  40:     4.626476   13.561113    0.000000    0.008241    0.191489
  17,  64,  20:   778.950533   13.561113    0.000000    0.000037    1.233108
  18,  68,  30:   686.048136   13.561113    0.000000    0.000040    1.387283

Note that the lowest points have the highest weights. They DEFINITELY shouldn't. What's wrong with them?

Apparently they have NO sensitivity to the sky! What?! There were a bunch of bad bolos in Dec2010 that weren't flagged out... I wonder if that problem persists to other epochs. Still, why does it only affect pointing observations? Looking at the power spectra... the large-timescale stuff becomes less dominant when scans are longer, but the noisy spectra are still clearly noise-only. How odd.

Dropped to 112 good bolos from 134. That is much more believable. Have to go back and fix Dec09 data too...

Even after fixing the bad bolos, the model drops with iteration number. Why why why?

Well, looking at deconv_map, I've always returned the truly deconvolved version, not the reconvolved... maybe the reconvolved really is better? Again, this will have to be extensively tested, but it certainly gets rid of the obvious/dominant error that the model kept dropping off. However, FINALLY, based on how ridiculously good the reconv-deconvolved map looks, I think I'm ready to do the extensive pipeline tests. So, 10dec_caltest has been started up with all of the new bolo_params applied and the changes in place to deconv_map... let's see what happens.


After that runs, I'll have to re-run the fit_and_plot routines

Making maps faster

The fundamental problem at this point is making the pipeline run faster. At current speeds, with undownsampled data, it may take ~days to process a single map. Ideas for faster processing:

1. Find out how long it takes to converge to 1%, 5%.... If it only requires 10 iterations, that's a factor of 2 savings over current strategies.
   
2. Use downsampled data of some sort if possible. Does DS2 match DS1? How do we measure it? Flux-flux comparison and PSF point-source size measurements are the most important. Need to automate PSF comparison....
   
3. Can we use median-combined individual images as a 0th order model? I bet the answer is 'yes' and will probably increase the speed of convergence by a large amount. Tests to run? This is probably needed if we are to split up the 'super-fields' into smaller sub-fields, otherwise overlapping data will be used less effectively.
   
4. Find some way to keep bgps.raw, bgps.ra, bgps.dec, and other items that are only used once on the HD during the iterative process. Is there any way to separate out data in a struct in this manner?
   

Delining - Maps

First comment - delining has no effect on downsampled data. At least for the 0709 epoch, there were NO lines AT ALL in the data. From 0-5 Hz, it was just empty. So we don't have to worry about that... the problem only affects fully-sampled data.

Then, onto map comparisons. Curiously, the noise levels don't drop after delining. They actually go up a bit. This may be because of the effects on PCA cleaning.

However, flux levels in the sources go up by 0-10%. As usual, the change in flux changes from field to field without any obvious reason.

Example 1: A pointing field. The source is ~2% brighter in the delined version, but otherwise the match between the two is nearly perfect.

Example 2: A bigger map, where the flux recovery is much greater when delining, but the background levels are also higher.

More evidence that downsampling causes problems

The captions are pretty much the same as for the previous post, but this is a larger field and the effects are more serious.

Downsampling has serious negative effects on the data

Background: Downsampling is performed using Old Pipeline code called
process_ncdf. All BGPS data was downsampled by a factor of 5
before mapping because of data size concerns. I did this 'blindly' (i.e., just
accepted that I should) because James said I could.

However, I had previously noted that the pointing files could not be done with
downsampled data because the beams 'looked funny' or something along those
lines; it may also have been a simple map sampling issue in which not all
pixels were filled with a downsampled image.

Anyway, I decided to go back and quantify the effects. The plots below are from the
single "pointing-style" observation of OMC1 from 2009. The units are volts. 'ds1' indicates
sampling at 0.02 seconds, 'ds5' indicates sampling every 0.1 seconds. The scan rate was
120"/s.

---

The beam sizes were measured from the autocorrelation maps. However, because there is structure on many scales
in this map, I had to use a rather ad-hoc method to remove the correlated structure. I fitted a gaussian
to the elliptical northwest-southeast structure, removed it, then fitted a gaussian to the remaining circular
thing in the center, which is approximately the beam.

If I fit the "beam" gaussian with an ellipse, I get:
Beamsize 1_1: 36.15,26.23
Beamsize 1_5: 48.39,30.21

With a circle:
Beamsize 1_1: 29.51
Beamsize 1_5: 35.31

The ds1 and ds5 images compared.

The PSDs of the two images (on identical grids). Note that ds5 loses power at small spatial scales, 50% at 40"!

The pixel-pixel plot with a fit that shows a 10% overall flux loss (best-fit).

Gem OB1 comparisons

I'm running 0,1,2,3,5,7,10,16, and 19 PCA component 51 iteration maps of Gem OB1 with deconvolution. No clue when they'll be done because they're at the end of a long queue.

Next (important!) step is to re-run the simulations with linear source sizes but with different numbers of PCA components, different kernel sizes, etc..... there is a LOT of parameter space to cover.

A detailed look at l086

Despite a slew of alignment errors, it appears that the alignment for MOST fields turns out OK using Method 3 of the pixel-shift code; the signal to noise is VERY low in a lot of fields.

070724_o38 does not come up with a good fit, for very good reason - there appears to be no signal at all.

070907_o20 is a problem. The offset was 27 pixels, which is too large, but nonetheless correct. I had to institute the plane fitter at an earlier stage to get it to work.

However, the biggest problem: the SCUBA source aligns with 070907_o20 but not the rest of the maps. So I needed to re-fit everything. That was a BIG mistake, we need to check carefully for it in other fields.

Return to Deconvolution

I ran the v0.7 reductions with deconvolution on for 50 iterations. I had cut out deconvolution originally because of the funky noise maps, but that was partly an error on my part. There is also an issue with bright sources being largely left over in the noise maps.

The deconvolver does MUCH better at filtering out the fuzzy atmospheric emission, so I want to use it. It leaves some flux from bright point sources behind, though, so I decided to try to make the deconvolution kernel smaller to see if that recovers more of the pointlike flux.

0709 rotators fixed, latest run

0709 11-13 needed to be 83.88 degrees, they're all fixed now.

I flagged the l060 070911 maps; I'm not convinced that was weather, there were some fluctuations up to 2400 Jy! Maybe a cloud would do that though.

Problems in the latest run:
l359 - missing files?
Reading files from /scratch/sliced/INFILES/l359_infile.txt
FIELD v0.7_l359 BEGUN at Fri Jan 16 20:11:34 2009
MRDFITS: File access error
% HEULER: ERROR - First parameter must be a FITS header or astrometry
structure
l012 - missing files?
% READFITS: ERROR - Unable to locate file
/scratch/adam_work/l012/060614_o10_raw_ds5.nc_indiv13pca_map01.fi
ts

New mapping procedure

I cut down the number of maps made by the pipeline from 80 to 30 by merging regions with severe overlap. I don't think this will result in any net change in efficiency of the mapping process because a few of the maps will go over to swap, but it will reduce the amount of overlap / noisy edges and make debugging simpler. It should also reduce hard drive usage once I decide that I'm satisfied with the results and remove the old (v0.6) redundant data.

As part of this process, the reference fields for coalignment have been re-set to the v0.6.2 coadds + some (crappy) mosaics from v0.6.2 so that we no longer need to rely on 3x1's for the alignment.

See /data/bgps/releases/v0.7/readme.txt for details (reproduced below)

Created 01/14/09
Version 0.7 does not exist yet. This is the 'release notes' indicating what will change

-Overlapping redundant fields will be eliminated
Mapping of field name to area covered, followed by number of observations
included:
gemob1: l189-l192 11 - 2 = 9
w5: l135-l138 46 = 46
l133: l133-l134 34 - 1 = 33
l111: l110-l111 88 = 88
l086: l085-l086 17 = 17
l082: l080-l084 33 = 33
l077: l075-l080 43 - 1 = 42
l072: l070-l074 10 - 2 = 8
l065: l059-l069 19 = 19
l055: l053-l058 10 - 1 = 9
l050: l048-l052 19 = 19
l045: l043-l047 20 - 4 = 16
l040: l038-l042 33 - 4 = 27
l035: l034-l037 53 = 53
l032: l031-l033 61 = 61
l030: l030 61 - 3 = 58
l029: l028-l029 53 - 4 = 49
l024: l021-l027 29 = 29
l018: l015-l021 29 - 3 = 26
l012: l011-l014 7 = 7
l009: l008-l010 11 = 11
l006: l005-l007 15 - 1 = 14
l003: l002-l004 25 - 1 = 24
l001: l001 20 = 20
l000: l000 23 - 1 = 23
l359: l359 14 = 14
l357: l356-l358 9 - 1 = 8
l354: l353-l355 3 = 3
l351: l350-l352 9 = 9

Gem OB1

With Miranda's help, I discovered some serious errors in Gem OB1 and fixed them.

1. L189 was using L134 data for no apparent reason
   
2. many September 2007 observations have the WRONG rotation angle in place in array_params! Unfortunately it's not clear yet which ones suffer from this problem: I need a list of when rotangle was/was not used, and it looks like it'll be a pain to fix the problem.
   
3. GemOB1link wasn't mapped, it will be now.
   

Deconvolution vs. Not

I've done some by-eye comparisons of deconvolutions vs no deconvolution. The no-deconvolution clearly does better on the bright sources: probably the deconvolved beam size is a little bit different from (larger? smaller?) the actual source size. Deconvolution does better in putting noise from noisy regions into the noisemap and keeping it out of the astromap.

Both might be useful - the deconvolved maps may end up being prettier, but the no-deconvolve maps will probably have more reliable fluxes.

This all probably relies on testing / simulation.

Version 0.6 begun

I started on (what I hope will be) v0.6 tonight.

New things:
-Sigma-rejection flagging in the spatial domain
-Inverse-variance weighting

Median drizzling

http://groups.google.com/group/comp.lang.idl-pvwave/browse_thread/thread/762770933591238d

Someone going by the name 'wox' introduced me to a spectacularly simple drizzling algorithm for medianing. It will be a little less efficient because it can't take advantage of some 'partial sum' tricks that the average can, but it will median over ALL data points, which is an advantage.

New to-do list, updated things....

do_maptests.pro is running a bunch of different mapping parameters (pca components, deconvolution, etc.) on l000, l002, l003, l033, l083. We'll then run bolocat on it and look for the following:
-number of sources found
-flux in sources found as a function of n_pca
-size of sources as function of n_pca
-BADNESS, e.g. blurring / unremoved atmosphere / oversubtracted sources

bolocat2reg makes a region file out of a bolocat catalog. I'd like to make a separate ds9 region file that includes the actual pointing error + the centroiding error rather than just the elliptical fit to a given source; that will be more useful for finder charts.

My current goal is to get a nice set of images I can combine to release as a poster to Jason Glenn's student who is making pretty posters for publicity purposes; probably to promote CCAT. Umm.... I blame the late hour for the alliteration.

That goal also means I've been using an IRAF task to do some mosaicing (easier than writing the IDL code AGAIN):

mscstack l001_5pca_map09.fits,l002_5pca_map09.fits,l000_5pca_map09.fits,l359_5pca_map09.fits,l003_5pca_map09.fits GCCOMBINE_5pca.fits lthresh=-1
l001_13pca_map09_scuba_aligned.fits,l002_13pca_map09_scuba_aligned.fits,l000_13pca_map09_scuba_aligned.fits,l359_13pca_map09_scuba_aligned.fits,l003_13pca_map09_scuba_aligned.fits GCCOMBINE.fits lthresh=-1
mscstack l029_13pca_map09_scuba_aligned.fits,l030_13pca_map09_scuba_aligned.fits,l031_13pca_map09_scuba_aligned.fits,l032_13pca_map09_scuba_aligned.fits,l033_13pca_map09_scuba_aligned.fits,l034_13pca_map09_scuba_aligned.fits L33COMBINE.fits lthresh=-1 hsig=5 lsig=5


outer galaxy fields done

they're small files

logs/log_101008_coalign.log:/scratch/adam_work/l059/l059_13pca took 286.08181 sec.
logs/log_101008_coalign.log:/scratch/adam_work/l062/l062_13pca took 390.88015 sec.
logs/log_101008_coalign.log:/scratch/adam_work/l065/l065_13pca took 296.88441 sec.
logs/log_101008_coalign.log:/scratch/adam_work/l068/l068_13pca took 1136.2623 sec.
logs/log_101008_coalign.log:/scratch/adam_work/l071/l071_13pca took 899.41245 sec.
logs/log_101208_coalign.log:/scratch/adam_work/l189p5/l189p5_13pca took 375.06840 sec.
logs/log_101208_coalign.log:/scratch/adam_work/l192/l192_13pca took 403.81112 sec.
logs/log_101208_coalign.log:/scratch/adam_work/l060/l060_13pca took 90.815500 sec.
logs/log_101208_coalign.log:/scratch/adam_work/l069/l069_13pca took 160.17454 sec.


Maps with problems

l002: fuzzy stuff - bad sky sub?
l033: misalignment
l004: possible misalignment? Pattern recognition fails me here
l354: misalignment

Mapping update

I made a lot of mistakes in my script, so progress is about 12 computer hours behind what it should be.

These maps are done:

/scratch/adam_work/l003/l003_13pca_map09.fits
/scratch/adam_work/l006/l006_13pca_map09.fits
/scratch/adam_work/l009/l009_13pca_map09.fits
/scratch/adam_work/l018/l018_13pca_map09.fits
/scratch/adam_work/l021/l021_13pca_map09.fits
/scratch/adam_work/l024/l024_13pca_map09.fits
/scratch/adam_work/l027/l027_13pca_map09.fits
/scratch/adam_work/l030/l030_13pca_map09.fits
/scratch/adam_work/l033/l033_13pca_map09.fits
/scratch/adam_work/l036/l036_13pca_map09.fits
/scratch/adam_work/l039/l039_13pca_map09.fits
/scratch/adam_work/l042/l042_13pca_map09.fits
/scratch/adam_work/l044/l044_13pca_map09.fits
/scratch/adam_work/l048/l048_13pca_map09.fits
/scratch/adam_work/l050/l050_13pca_map09.fits
/scratch/adam_work/l054/l054_13pca_map09.fits
/scratch/adam_work/l057/l057_13pca_map09.fits

Things I've learned so far:
There is an observation (a master?) in L003 that should be the galactic center master.
L018 has some really bad noise that needs fixing.
The wackiness I saw in L024 seems to be gone now.
L044 has the same noise stripes as L357.

/scratch/adam_work/l018/l018_13pca took 1118.3684 sec.
/scratch/adam_work/l018/l018_13pca_nooffs took 1106.6423 sec.
/scratch/adam_work/l021/l021_13pca took 1703.7266 sec.
/scratch/adam_work/l021/l021_13pca_nooffs took 1716.7393 sec.
/scratch/adam_work/l024/l024_13pca took 1622.3164 sec.
/scratch/adam_work/l024/l024_13pca_nooffs took 1672.3263 sec.
/scratch/adam_work/l027/l027_13pca took 1376.1939 sec.
/scratch/adam_work/l027/l027_13pca_nooffs took 1383.4335 sec.
/scratch/adam_work/l030/l030_13pca took 2487.1311 sec.
/scratch/adam_work/l030/l030_13pca_nooffs took 2543.1740 sec.
/scratch/adam_work/l036/l036_13pca took 549.76983 sec.
/scratch/adam_work/l036/l036_13pca_nooffs took 548.84132 sec.
/scratch/adam_work/l039/l039_13pca took 299.96889 sec.
/scratch/adam_work/l039/l039_13pca_nooffs took 299.16787 sec.
/scratch/adam_work/l042/l042_13pca took 366.34139 sec.
/scratch/adam_work/l042/l042_13pca_nooffs took 364.31475 sec.
/scratch/adam_work/l044/l044_13pca took 427.26923 sec.
/scratch/adam_work/l044/l044_13pca_nooffs took 424.42403 sec.
/scratch/adam_work/l048/l048_13pca took 313.86412 sec.
/scratch/adam_work/l048/l048_13pca_nooffs took 303.52545 sec.
/scratch/adam_work/l050/l050_13pca took 421.22899 sec.
/scratch/adam_work/l050/l050_13pca_nooffs took 477.16070 sec.
/scratch/adam_work/l054/l054_13pca took 315.14965 sec.
/scratch/adam_work/l054/l054_13pca_nooffs took 342.69137 sec.