prepare_map issues

The problem: There is a bulk offset that can range from small to ~22' (largest observed so far) in the combined maps that is not present in the individual maps.

Observations: I THINK the offset between ra/dec and l/b is most pronounced when the map boundaries are most different.

We have been putting CRPIX in the middle of the map (i.e. number of x/y elements divided by 2), which is not the same position as the median ra/dec or l/b. I figured this might be a problem, but there's not really a way around it - if you just set CRPIX to be 0,0, it doesn't change anything. xy2ad and ad2xy produce self-consistent results, but that doesn't mean anything either.

I think the problem is that AD2XY is largely ignorant of CRPIX: it calculates x/y offsets from the CRVAL assuming that the CRVAL is at CRPIX.... but I don't know why that should be wrong. Ideas? More to come if I figure anything out.

Update: from doing the individual maps in l/b vs ra/dec, I don't think the offset is between l/b and ra/dec.... something less fundamental.

Update 2: still no clues after reading ad2xy and trying to plow through parts of wcssph2xy. gotta be rect_pix_tstream, but I think it's right....

Update 3: I think I have figured out the problem, currently testing. My idea has to do with the fact that all x,y positions in rect_pix_tstream are positive, but ad2xy is not constrained to return positive values. In principle, the two-iteration ad2xy should take care of this, but in the case where the middle of the map in pixels is not the middle of the map in the WCS coordinate system, it is still possible to get negative pixel mappings out of the 2nd iteration. I'm still confused about whether this is really possible - makes my head fuzzy - but I tried implementing a solution where I simply shift the crpix rather than recalculating with AD2XY.

Update 4: Based on L111, my test in update 3 fixed the problem. Scuba contours are correct, and reasonably consistent across the field - still need to do the more complete test, e.g. Cygnus

Update 5: Cygnus tests still await completion, but a very close inspection of single L111 maps reveals a ~2 pixel difference that could go a long way to explaining my high-RMS pointing calculations. Still doesn't explain the sine curve, but anything helps...

Update 6: My correction definitely fixed the problem, EXCEPT there's still an ambiguity at the .5 pixel level. Specifically, is AD2XY treating the pixel center as .5,.5 or 0,0? Ditto IDL. I believe (from my simulated data) that this ambiguity is still causing a problem.

Update 7: New correction fixed problem to better than 1" (checked by making a simulated map with 1" pixels) - probably 'perfect' now. Read prepare_map.pro for details. The issue was a difference in pixel centers.

Confirmed error in L/B mapping

I've used my simulated data and confirmed that there is in fact an error in the l/b mapping, but I haven't tracked it down yet. Stay tuned.


Update: Something is not self-consistent in prepare_map. I've tried a variety of different things without success yet.

Updates today

I updated the FITS header writing pieces of the software today, we should now have:
N_PCA
ITERNUM
PPBEAM

in all new maps.

I'm mapping all of Cygnus degree-by-degree in RA/Dec and Galactic to try to measure offsets... possibly offsets as a function of l/b or ra/dec. I should really be doing this with PPSes, though.

My priority list:

1. find out what's causing bulk offsets
2. beat down noise in my pointing models by rejecting bad sources/points [has to be done from campus]
   
3. test my personal special pointing models
4. make maps!
5. figure out the rest of pointing
   

Iterating to Convergence

Some sample plots of flux-vs-iteration number in the Galactic Center. The above is for a 13-pca-component map, and the plot is of the northeast peak of SGR B2. The other plots (available in postscript below) are of two other points in the GC, and the last one is of the whole of SGR B2 (it's a very large aperture).

ps version here:
http://sites.google.com/site/bolocam/pipeline/mapping/iteratetoconvergence.ps?attredirects=0

or /scratch/adam_work/plots/iteratetoconvergence.ps

G34.3 PCA subtraction comparison

From top-left to the blue highlighted box, the 20th iteration of N pca components subtracted is shown. N goes from 1 to 19. The other plots are 'ascending' and 0-pca components with median and baseline subtraction - ask if you want more details on those.

19 iterations recovers a LOT of flux. I haven't generated the plots for flux as a function of iteration, but I have software to do that available.

Things to note: at 10 iterations and higher, you start to generate negative bowls.
One observation is probably independently responsible for the chunks of glitchy data present even in the 19 PCA component subtraction. While the bowls do worsen with higher number of PCA components, we also find fainter structure (e.g. those tongues to the southeast and northwest).

Some good news, some.... not news.

I've been mapping all weekend. It turns out that if you do 25 mapping runs that take ~1hr apiece, that takes ~25 hours. Who knew?

Whole observation took 3965.3506 sec.
Whole observation took 3071.1403 sec.
Whole observation took 4205.3337 sec.
Whole observation took 7362.5490 sec.
Whole observation took 9313.6064 sec.
Whole observation took 3755.2383 sec.
Whole observation took 3755.8278 sec.
Whole observation took 3760.7032 sec.
Whole observation took 3758.5719 sec.
Whole observation took 3763.1436 sec.
Whole observation took 3766.0066 sec.
Whole observation took 3789.5806 sec.
Whole observation took 3760.9774 sec.
Whole observation took 3762.7124 sec.
Whole observation took 3763.1120 sec.
Whole observation took 3767.1255 sec.
Whole observation took 3766.9085 sec.
Whole observation took 3758.5867 sec.
Whole observation took 3760.2131 sec.
Whole observation took 3755.8963 sec.
Whole observation took 3759.9199 sec.
Whole observation took 3764.4494 sec.
Whole observation took 3766.9686 sec.
Whole observation took 3756.5876 sec.


The good news is that I haven't encountered any crashes. Smooth sailing so far.

Re: Convergence Tests

Made a little code to check out convergence, but frankly it's pretty easy to just build a mapcube and plot lines in the iteration axis. The code is in bgps_pipeline/postproc/compareiters.pro, and it is not a single program.

Comparing deconvolution to no deconvolution, deconvolution is a lot better. Without it, there are much more substantial negative regions. In the GC, my test region, the noise-dominated areas were about the same, though the no-deconvolve map had a little bit more large scale structure. The signal-dominated regions were very nearly uniformly brighter. The deconvolved map was ~3 Jy brighter in SGR B2, or 4%.

On the list for today....

Since my gigantic mapping run is still going, I'm going to work primarily on developing code for testing the maps, and determining important information about the mapper.

The data paper will require an iteration-to-convergence figure, or at least an estimate of degree of convergence.

1. Need to develop code to show iteration-to-convergence
   
   1. need it for both NxNpca and ascending/descending
   2. have to use region files? make some sort of region reader?
   3. pick demonstration sources
2. Compare N_pca results for bright, faint sources
3. Compare iterating with/without deconvolution
4. Compare median/average/baseline sky subtraction
5. Analyze efficiency...
6. Try to figure out what's causing huge offsets. Is it Galactic Coordinate mapping?
   

-bash-3.00$ less /scratch/adam_work/logs/log_082208_domemiter.log | grep Whole
Whole observation took 5122.6795 sec.
Whole observation took 4774.3424 sec.
Whole observation took 2687.7761 sec.
Whole observation took 2102.5577 sec.
Whole observation took 3247.5344 sec.
Whole observation took 5515.2907 sec.
Whole observation took 2877.6389 sec.
Whole observation took 2757.0235 sec.
Whole observation took 1453.0918 sec.
Whole observation took 1459.7892 sec.
Whole observation took 1381.3762 sec.
Whole observation took 1443.1440 sec.
Whole observation took 614.39645 sec.
Whole observation took 2001.6709 sec.
Whole observation took 1817.1512 sec.
Whole observation took 1879.1385 sec.

Note that the longest set there - 5515s = 1.53 hours - was 40 iterations!

TEN!?!

Look at that. Seriously? a 10' offset? That's ridiculous! There is no WAY our pointing models could be off by that much! Even if I got the signs entirely wrong, that's just not possible.

What's going on? First theory: Galactic coordinates fail. Any other suggestions?

Task List

For the moment, I'm going to leave the pointing alone. This leaves us (me) with a very long list of things to do before the data release:

1. Determine ideal iterative mapping strategy for low-flux fields
2. Determine ideal iterative mapping strategy for fields with bright sources
3. Optimize at least the following:
   
   1. scan flatting (polynomial subtraction)
   2. PCA subtraction
   3. Pre-PCA sky subtraction (necessary?)
   4. Deconvolution/No deconvolution
      
4. GENERATE MAPS [top priority, but it can't really happen til after the above]
   
5. Assure consistency with catalogs - e.g. Motte, SCUBA, etc.
   
6. Figure out what needs to go into fits header
7. Survive semester before comps [lowest priority]
8. sleep [oh, right, real lowest priority]

Sarcasm helps me survive. Almost as much as beer.

New pointing models

I decided to try to fit the pointing model as a full alt/az or ra/dec dependent thing using mpfitfun and a 12 (or 16) parameter model. It didn't work very well.

I could fit out the alt dependence, then independently fit out the az dependence, and see how that goes. I think it will work brilliantly.

Got a sign wrong again

Last night's run failed because I had a wrong sign in the pointing model. Do over time!

Notice anything different?

About milkyway? This post is a red herring, but:
I decided to go ahead and run the mapper on Cygnus, L33, L111, and the Galactic Center again. The pointing has gotten to the stage where I'm certain I can't do anything more without a stroke of pure brilliance or a conversation with someone who hasn't touched a thing - e.g. Jason - that suddenly enlightens me.

In order to fully reproduce Meredith's results, I'd probably have to go back through and follow her 'pipeline' process step by step as well, and I suspect that, if I had used her method on her RA/Dec maps, I would have come up with exactly the same problem I currently see. Therefore, I won't do anything about it.

I think the most appropriate response at this point MAY be to just fit a damned polynomial/sine curve in az and include that as part of the pointing model, but I can't justify where that comes from. All I know is that it's present in Meredith's data as well as my own.

What's that next step with the PPSes? We need to do that.

More important, though, is getting some image optimization ready FAST. I need to be running this stuff before September! Tonight's run will be a test of numbers of PCA components. At the very least, the v0.5 should have a consistent set of images even if we cut out the high-flux ones. We'll deal with that later.

For the high flux objects, e.g. g34.3, since we have so much overlapping data, a simple average/magical baseline subtraction might be just as if not more effective than PCA subtraction, so that's one way around it.

Modifications

Things that have been changed in the past 24 hours that are very significant:

1. In apply_pointing_model, the signs of FAZO and FZAO have been swapped. I BELIEVE this is correct but somehow things aren't working out still.
2. In do_the_pointing I have changed from "eq2hor" and "hor2eq" to "my_eq2hor" and "my_hor2eq". These implement two major changes:
   
   1. The LST is passed as a parameter rather than calculated within the ASTROLIB code
      
   2. The conversion is calculated with BOTH hadec2altaz and getaltaz (and similar for the opposite transformation) and compared for error checking purposes. If they differ by more than one arcsecond, the code will spit out an error message and use getaltaz.
   
   

Inconsistent

James: "I think it's fair to say, though, that there is some problem with the simultaneous assumption that CSO coords are geo and that we are applying the ab/nut correction correctly."

Yep. The relevant files are in /scratch/adam_work/plots/:
models_noabnut_radec_0707.ps
models_noabnut_rawcsoptg_radec_0707.ps
models_oppositeabnut_radec_0707.ps
models_oppositeabnut_rawcso_radec_0707.ps

I'm afraid they're pretty confusing.

'noabnut' means no aberration/nutation correction was applied during the mapping process.
'oppositeabnut' means that an aberration/nutation correction that, according to the eq2hor and hor2eq texts, should actually convert heliocentric to geocentric, is being used on data that we believe is starting in a geocentric frame. There are two possibilities: 1. We are wrong 2. eq2hor/hor2eq are wrong.

The 'rawcso' files have FAZO/FZAO "subtracted out" (removed) in pages 1 and 3, and FAZO/FZAO added back in on pages 2 and 4. 'rawcso' means that we're looking at the ra/dec the CSO gave without the users' FAZO/FZAO corrections.

The non-rawcso have the NCDF ra/dec vectors, with precession correction applied (and some form of ab/nut correction), but the FAZO/FZAO are still present. THESE should be equivalent to Meredith's plots, e.g. where FAZO_SET/FZAO_SET are on the y axis. However, that's only true for pages 1 and 3. Pages 2 and 4 have FAZO/FZAO essentially double-subtracted, so they should be ignored. Pages 1 and 3 of the non-rawcso files should be equivalent to pages 2 and 4 of the rawcso files with the exception that the sigma-rejection used to choose the yellow points is different.

So what's in each page? On pages 1 and 2, the red lines are my best fit to the yellow data, which is the black data with an iterative sigma rejection applied (i.e. reject at 1 sigma, recalculate sigma from good data, reject at 2 sigma). The blue lines are Meredith's models. The right side includes only the yellow data points with the red line subtracted. On pages 3 and 4, the same is pretty much true except that ONLY the altitude-dependent line has been subtracted: there is no fit to azimuth.

Also, the 'sourcecompare' files are similar. I'd check those out too.

I still don't have the extremely low RMS that Meredith saw. I'm going to go through and try to reject bad data points by hand to see if I can get to that level. We'll see.

Direct comparison with Meredith's pointing calculations

I've gotten to the point that I'm directly comparing my pointing calculations to Merediths. So far, it looks like:
1. I can very nearly reproduce Meredith's results on her maps, though there are a few differences in centroids and I missed the source a few times where she didn't.
2. There is still a spread between my map centroids and hers.

The code I've written to enable this comparison: publish/debug_testing/compare_me_meredith.pro

I'm still a bit stumped, but I think I have a path to the answer mapped out.

Centroiding and Meredith's files

I tried the pointing offset calculations on Meredith's maps and I still get an RMS of ~9" in both RA and DEC. Check out the files:
/scratch/adam_work/plots/models_meredith*

So... what?

Searching for more offsets

The defining feature of my pointing calculations - and their inconsistency - is that the spread for individual sources is bad, which means that there's still something wrong in the way the bulk ra/dec are being calculated.

I checked apply_distortion_map_radec to see if it might have inserted some bulk offset, but it only changes the average position by <2". That might be an issue, but the boresight isn't necessarily aligned with the mean location of the bolometers. It should be shifted by at least .3" according to a simple average of sin/cos of the bolo_params angles.


Comparing my pointing code to map_ncdf_reading directly:
readstruct=map_ncdf_reading(filename,/nopixoff)
offsets are small (e.g. -0.050665803 -0.0029540043 arcseconds ra/dec) if I don't subtract fazo/fzao
offsets in ra/dec are remarkably close to fazo/fzao if I subtract them:
92.830656 -126.12773
95.000000 -120.00000
which is because ra/dec are pretty closely aligned to az/el.

So, any difference comes from at least one of the keyword parameters. Therefore I AM missing something, and that something is one of the keywords to map_ncdf_reading

FFTs and unlucky primes

I've encountered a lot (!) of unlucky prime numbers that I'm trying to FFT in too many places, and it basically stops the code.

211151 was impossibly bad, 5827 is pretty bad too.

I solved the first by splitting into by-scan delining, which is better anyway. But the second is a map, in which I KNOW zero padding is OK. So... are there any functions that find the nearest reasonably efficient map size?

More cleaning modifications

I had removed sigmadeglitch from deline, now it's permanently gone. I think it might be worth exploring re-inserting sigmadeglitch somewhere.

My higher-order polynomial fit is creating deeper bowls around sources, so that's a big problem, but if I use a lower order I get the bad streaks. What's the best way to deal with this? I'm thinking perhaps only doing the polysub on the second iteration (i.e. after a source model has been subtracted).

Delining runs into some issues now, though, because not all frequencies are sampled (?).

Wow - FFT failure

On my huge Cygnus run, the fft keeps failing in the deline code. It's actually pretty impressive, but there are 844604 points in the timestream. The prime factorization of 844604 is 2^2 * 211151. This is just damned bad luck, because I think an FFT is extremely inefficient when it can't factorize. What's the best workaround? ....

9 AM update: I've rewritten the deliner to work on a scan-by-scan basis. It's possible that the delining failed in the past because it was essentially removing a constant amplitude at the line frequencies across the observation (or combined observations!) which is not likely to be true.

Cleaning ideas

It's proving very difficult to get rid of glitches, so here are some more ideas:
Median filter the whole timestream (downsampled) at a resolution that will pull down the glitch peak. Subtract out the median-filtered timestream from the original, and look for outliers in that distribution: in principle, those should be glitches.

Another option: subtract out the noise map AND the best-model map from the original timestream before trying to pull out additional astro model stuff. The map-to-timestream astro model can't include glitches because they're averaged over, but the sky-subtracted timestream DOES include glitches (no matter how many PCA components are removed) because glitches are NOT correlated across detectors. The weird thing is that this is effectively subtracting out exactly what was calculated from the sky subtraction - the question is whether subtracting it out BEFORE sky subtracting again gets you any benefit. If it was just noise, subtracting noise from noise is fine, but the "noise" does include some residual signal.

Glitches and errors

It's definitely important to get rid of the spikes and, more importantly, the exponential decay. The current function is pretty good, but possibly not good enough because there are some situations in which the turnaround decay is obviously not well-enough dealt with and results in high/low streaks.

The glitches can be ~20 Jy. The problem is that they sometimes show up in the middle of sources (e.g. in 060609_o26). That will badly distort fluxes.

My ideal is still to use a median instead of mean combination of data points into image points, but that seems to be impractical to implement. Other ideas...

Big run this weekend...

In case anyone is wondering why Milkyway is going really slowly, I'm mapping a 69-observation set of Cygnus. It ought to prove to be an interesting test of Milkyway's swap capacity, but other than that I doubt it will be useful. While the pointing is reasonably good at this point (30" still, but whatever), I haven't done ANY work on filtering out bad observations / flagging stuff in Cyg. Data massaging is going to be a long process, it would be great if I could do that instead of pointing stuff. Argh.

One thing to note is that this file:
/scratch/sliced/l078/070702_o33_raw_ds5.nc
is a "_ds5.nc" but is NOT downsampled!


Update:
Mapped the individual files successfully, picked out noisy ones. The overall map failed - just not enough memory to do a field that large. I split it up in to two sets of 25 observations, plus I'll be mapping each L 70-L 90 field separately (I didn't get ride of noisy observations for this). I'm also remapping the individual observations with PSD flagging enabled to see how that works.

For notes on the P Cyg observations, see the file
/scratch/adam_work/texts/cygnus_for_pat.in

RMS fails to shrink

Spent a while today working on these plots after rejecting every obvious bad point:
/scratch/adam_work/plots/sourcecompare_0_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_10_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_1_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_2_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_3_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_4_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_5_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_6_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_7_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_8_rawcsoptg_0707.ps
/scratch/adam_work/plots/sourcecompare_9_rawcsoptg_0707.ps
/scratch/adam_work/plots/models_rawcsoptg_0707.ps

The net result is, I still don't have a nice small RMS offset

Update 8/12/08: using RA/DEC mapping doesn't help.

Images with artifacts

Here's the list from 0507:

Streaky offsets:
/scratch/adam_work/g34.3/rawcsoptg050706_o47_raw_ds1.nc_indiv3pca_map00.fits

Totally messed up (two sets of images many degrees apart):
/scratch/adam_work/g34.3/rawcsoptg050713_o40_raw_ds1.nc_indiv3pca_map00.fits


070709_ob7 :


070703_o48 :

Looking back at the old pipeline

I've obviously missed something. So to try to figure out what it is, I'm going back to the old pipeline... again...

in map_ncdf_reading, lines 441-448, there is something curious that goes back to a definition-of-variables problem: ddec and dra are ADDED to ra and dec to get the new "ra_all" and "dec_all" variables. ddec and dra are calculated from eaz and eel: ERROR OFFSETS in Az and El. Why? What?!

I added a new piece of code, correct_eaz_eel.pro. It is extremely short, but extremely necessary.

pro correct_eaz_eel,ra,dec,el,az,eel,eaz,pa
    dra=-eaz*cos(!dtor*pa)*cos(!dtor*el)+eel*sin(!dtor*pa)
    ddec=eaz*sin(!dtor*pa)*cos(!dtor*el)+eel*cos(!dtor*pa)
    dec += ddec/3600.
    ra  += dra/3600. / cos(!dtor*dec) / 15.
end    

This comes back to the fact that I don't know what ANY of the variables in the NCDF header are supposed to be. Why are "error" variables actually OFFSET variables, and why didn't anyone know about them?

something is wrong in 0507....

Look at that zoomed in. Note that the first 20ish are from 0507. I don't know what's causing the double source. Maybe a 90 degree rotation would do it, I don't know. I do know that I'm applying an array angle of 76.2 degrees for those July observations and 113.6 degrees for the June 05 observations. Maybe that got swapped or something....

It looks like Meredith ran into the same problems when mapping 0506, but not 0507.



comparing mine to hers (I even used the same downsampled/cleand file), it is obvious that something went wrong with the mapping. I tested a number of different fiducial array angles and none of them generated real maps, so the mapping is correct except for offsets in the scan direction, as usual. What the heck.

Distortion (not distortion maps)

I still see distortion in a lot of maps, but not all. I think there is a correlation with altitude ~70+/-2 degrees. So far it's most obvious in the 0606 pointing sources.

Occasional bad centroid

I've gotten some bad centroids for 3c273 that don't make sense.

0507 mapping

Most of the 0507 pointing source maps seem to have failed. Some of them look like rotator and position angle problems, others have multiple copies of sources mapped to different locations. I don't know what's up, but my first bet would be to change the fiducial array angle. After that, I'd check on rotang and then, if desperate, see what the PA is doing.

Mapped, but no dice

Finished mapping what I think are all of the pointing sources. However, the RMS of the centroids has not gone down. Now I need to go through each individual source and image and try to get rid of the baddies. There are presumably some sources with bulk offsets that can be removed to improve the fits. I just have to find them.

Nothing

No progress today. Spent the whole day re-mapping pointing sources and figuring out which observations ARE pointing sources. Mapping STILL not done - it takes a long time when there are hundreds of separate observations. Also, finding some corrupt files and noting them in /scratch/adam_work/texts/toolarge.txt . A lot more of that work to be done.

Observation distribution

-bash-3.00$ wc texts/all_*_observations.txt                                                                                                 
    0      0      0 texts/all_0505_observations.txt
    0      0      0 texts/all_0506_observations.txt
  151    151   6039 texts/all_0507_observations.txt
   33     33   1377 texts/all_0509_observations.txt
   16     16    633 texts/all_0605_observations.txt
  135    135   5309 texts/all_0606_observations.txt
    0      0      0 texts/all_0607_observations.txt
   94     94   3780 texts/all_0609_observations.txt
   34     34   1386 texts/all_0705_observations.txt
   26     26   1028 texts/all_0706_observations.txt
  201    201   7989 texts/all_0707_observations.txt
   73     73   2907 texts/all_0709_observations.txt
    0      0      0 texts/all_polychrome_0505_observations.txt
  222    222  11898 texts/all_polychrome_0506_observations.txt
  471    471  24861 texts/all_polychrome_0507_observations.txt
  196    196  10346 texts/all_polychrome_0509_observations.txt
   43     43   2200 texts/all_polychrome_0605_observations.txt
  730    730  38040 texts/all_polychrome_0606_observations.txt
   22     22   1195 texts/all_polychrome_0607_observations.txt
  471    471  24751 texts/all_polychrome_0609_observations.txt
   66     66   3399 texts/all_polychrome_0705_observations.txt
  145    145   7533 texts/all_polychrome_0706_observations.txt
 1051   1051  54801 texts/all_polychrome_0707_observations.txt
  511    511  27049 texts/all_polychrome_0709_observations.txt
 4691   4691 236521 total

L111 observation notes

In L111, the failures noted in the previous post are specific to September 2005. By comparing the pointing model images to the "raw cso pointing" and "no pointing model" (the latter still have FAZO/FZAO applied, all that has been done to them is aberration/nutation and precession correction) it appears that the Sep 05 model is simply wrong.

I haven't had time to check on W3/4/5 and L033, but presumably they have a similar issue. W3/4/5 includes late July 07 and September 06. L033 includes July 05, June 06, and July 07. I will need to generate pointing plots for September 05 and 06, and probably regenerate June 06. I have the rest and I think Meredith's models are fine for those dates - they certainly don't create recognizable errors in the maps.

It is possible that my pointing_model.pro has some pointing models (the problematic ones) entered incorrectly. It would be nice to get those double-checked.

Pointing model failure

I attempted to map L111, L033, and W3/4/5 with the pointing model corrections applied. That was a failure. Two images are attached to illustrate the problem - relative pointing is clearly not correct (middle of L33, bottom left of L111).

So.... possibilities:

1. Pointing models are incorrect
2. Pointing model signs have been misinterpreted (don't think this one is possible based on other graphs)
3. Uhh... ideas?

Next step, I'll use the individual maps with no ptg mdl and find out what the offsets should be....

News, modifications

RA / Dec mapping did change centroid locations, but not really for the better. A deeper analysis is probably necessary, but nevertheless I'm not convinced mapping type is the problem. The list of possible answers to the question of why my pointing models don't match Meredith's:

1. The new mapping is shifting the centroid
   
2. The centroiding method I'm using is different / incorrect
   
3. There is still something in the pointing we haven't caught
   

The lack of systematics suggests that the third option is not correct. The first seems the most likely, but also the most difficult to track down. I don't know where to go with #2.


I've updated the code so that the pointing model correction offsets in RA/Dec are written to the FITS header. This will be the standard in all future runs for individual observations. The calculation of this offset post-mapping is straightforward but it's nice to have a redundant error check.


The report for Monday will be kind of empty, sadly. However, I think I can say that I'll now move on to testing the previous problems we faced, in particular that we could not match the pointing across whole fields (L111). If THAT is fixed, then at least we know that whatever my pipeline is doing differently (e.g. PA at all times...) is useful if not 100% correct.

I'll need to be in the office to test the field mappings, though, because ds9 display doesn't work well over wireless. I'll do a remote run of the individual L111 and W5 maps to compare to previous ones.

Pointing update

I believe I've completely dealt with the projection issues. There are a lot of new plots available now, pretty much everything for Jun/Jul 05 and July 07. Low on my to-do list is the rest of the pointing model tests.

My conclusion now is that I can't match the 3-4" RMS in dAlt and dAz. Even with aggressive sigma-rejection of outliers, I only get down to ~10" RMS (which adds to ~15" RMS). My calculated pointing models differ significantly from Meredith's, but not extremely so for 0707.

The code I've used to generate the plots is centroid_plots.pro under plotting/. Using that code, I am able to essentially reproduce each change made in do_the_pointing besides the distortion mapping. Because of the weak dependence of the pointing model on altitude, even errors on the scale of precession won't really hurt the pointing model calculations. The only unanswered question I think may be responsible for some errors - but not enough, I think - is whether eq2hor/hor2eq could be double-correcting for aberration and nutation, but this wouldn't affect the actual data reduction. It WOULD affect my plotting!

I think my next test is to try to coadd different epochal data for particular fields, e.g. l001, l033. Not l000 because of the galactic coordinate issue.