APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday April 29, 2006 (MJD 53855) ---=== OBSERVING TEAM ===--- John Barentine Viktor Malanushenko Olena Malanushenko (swing shift help) French Leger (super 2AM phone support) ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Finished 1 plate tonight (2491) and got signal on two others (2140 and 1729). Also imaged SEGUE stripe 1260 N from +105 <= Mu <= +132 in marginal to poor seeing conditions. We had serious partly-telescope, mostly-operator-related problems tonight at the instChange from imaging back to spectro. See the Problems section for the post-game highlights. There was also a problem with Viktor's SOP session that caused us to lose about 40 minutes of time. ---=== OBSERVING LOG ===--- Afternoon --------- More issues with the video cards feeding the monitors for ad1 and ad2, with suggestions of the same problem (horizontal bars at the far left edge of the monitors) on ad3 and even id1. Is it contagious? Spectrograph focus has again been set for +3 deg C as we expect it to be a little warmer tonight than it was last night. Numbers are in the Problems section. Night ----- Clear at sunset with light winds out of the NW, low humidity and low dust. Open at 01:45Z. To ascertain seeing conditions for possible imaging later, we began with spectro. 02:29Z Cart 1 Plate 2491 (tile 1729, chunk143) GSOGTF. Did 2 x 900 s exposures and 1 x 1100 s exposures as we noted a drop in efficiency during the final exposure on this plate. Done. We switched to imaging at 04:25Z. 04:38Z Run 6117: On-telescope bias. 04:48Z Run 6118 (SEGUE Stripe 1260 N, startMu=+105). The beginning of this stripe, within the given limits of start times in the science plan, was only 33 degrees off the northern horizon when we started observing it. That's an airmass of about 1.8 and it's almost guaranteed we will not achieve survey-quality seeing conditions as such a low elevation. And predictably, that's exactly what we saw: subarcsecond seeing near zenith, 2+ arcseconds at the start of this stripe. seeingRun called the data bad during the ramp, from +105 <= Mu <= +110. Seeing improved during the ramp, however. Skippy was happy from Frame 10, and we were in focus by Frame 18. Seeing slowly improved as the drift continued and our zenith distance diminished. That didn't last, however. Seeing took a big rollercoaster ride throughout this stripe that had nearly adjacent sections of the strip called good and bad by seeingRun. We let it run to the end of science at Frame 163 (Mu=+129), and ended the drift at Frame 189 (Mu=+132.6). Between roughly 06:00Z and 07:30Z there is essentially no available imaging according to the science plan. We decided to back up and re-do part of Stripe 1260 N that seeingRun called largely bad in hopes of getting data with better seeing. 06:50Z Run 6119 (SEGUE Stripe 1260 N, startMu=+120). Seeing blew up alomst immediately on the start of this stripe to well above 2 arcseconds on DIMM. Skippy wouldn't run, complaining about not being able to find gangs. I tried repeatedly forcing a restart of the teamster, but kept getting the message: "Error: Failed to kill old teamster, 12531". There was indeed a PID 12531, listed as "iop.bin [defunct]", but it wouldn't go away with a kill -9. Didn't matter anyway, seeing was stuck above 2 arcsec, so we ended this drift and declared the whole thing bad. Switched back to spectroscopy at about 07:00Z. Problems, mostly due to operator error, happened during this timne that cost us a total of about 90 minutes of lost time. A full accounting is in the Problems section. 08:48Z Cart 6 Plate 2140 (tile 1544, chunk109). No GSOGTF. We had some seriously too-large-to-correct errors on altitude during this time (~13000) that were likely due to slips that happened as the result of wrong counterweight setting during test slews before I released the telescope back to Viktor. We forced fiducials and Viktor slewed back to the field; no errors too large to correct this time. This time we went right to an fk5InFiber number 10, which found the star offset by about D=+5 from center. Made the offset, did another goToField, and we had stars. Seeing was again highly variable throughout. At about 09:55Z, Viktor's SOP session became unresponsive; this was the session from which he had started the science exposure that was underway at the time. His SOPGUI had frozen reading an amount of time remaining as 460 s. That means whatever happened occurred at around 09:49:20Z. I had him close out all the affected windows and start up a new SOP session; when he tried taking an initial bias, he got the error message "There is an ongoing exposure on sp1" and SOP refused to take the bias. But it was already past the expected readout time yet no readout happened. His SOP session was still frozen, so I assumed that the science exposure was lost. We killed a couple of rogue SOP processes listed in a "ps -a U viktorm | grep sop" on sdsshost2. Started a new SOP session and requested an initial bias, which worked. Altogether, including time to set up again on the field, we lost nearly a half hour, but by the time we got going again (or something close to it...) the cart had expired. But before giving up we made one last try. In Viktor's SOP session, the software seemed to be waiting for tcc replies that were not coming back; the message "I think that we have not started slewing" scrolled by repeatedly in his SOP terminal window, but the telescope did not move. I had him exit and restart the session because before we switched catridges I wanted to be sure that we could observe rather than doing the switch for what amounted to nothing. After this trick -- exiting and restarting SOP -- we were able to acquire the field of Cart 6. That left just enough time for... 10:30Z Cart 9 Plate 1729 (tile 1236, chunk88). GSOGTF. We had about enough time to get one exposure on this plate before the onset of twilight. We finally put this night out of its misery at 11:00Z with a faint gloaming in the east. endNight seems to be the only participant that rejoiced tonight... ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 6116 23:18Z 23:35Z 100 O -83.39 -79.26 42 ignore 6117 04:23Z 04:44Z 100 O -6.92 -1.86 48 bias 6118 04:54Z 06:43Z 1260 N -79.37 -51.96 197 science Seeing mostly out of spec 6119 06:48Z 07:09Z 1260 N -64.53 -59.44 48 science Seeing *totally* out of spec ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 6118 10 4 129 1.500 0.188 -1.700 0.175 -0.00455 353 33 6118 40 4 116 1.200 0.148 -1.200 0.137 -0.00587 353 37 6118 62 4 96 1.500 0.144 -1.200 0.119 -0.00518 352 40 6118 89 4 98 1.500 0.138 -1.000 0.142 -0.00551 351 44 6118 96 4 86 1.600 0.166 -1.100 0.165 -0.00589 351 45 6118 113 4 90 1.400 0.128 -1.400 0.126 -0.00518 349 47 6118 165 4 27 1.400 0.107 -2.000 0.148 -0.00413 342 53 6119 8 4 0 0.000 0.000 0.000 0.000 -0.00000 0 0 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 6118 19 1 35 187 27 0.025 0.353 2.245 0.287 -0.00755 6118 19 1 35 187 30 0.101 0.415 2.362 0.259 -0.00794 6118 19 1 35 187 29 0.039 0.443 2.471 0.234 -0.00831 6118 19 1 35 187 22 0.263 0.434 2.396 0.301 -0.00805 6118 19 1 35 187 20 -0.239 0.648 2.699 0.392 -0.00907 6118 37 1 38 187 20 -1.028 0.259 2.060 0.318 -0.00692 6118 37 1 38 187 15 -1.167 0.251 2.598 0.108 -0.00873 6118 37 1 38 187 29 -1.108 0.388 2.622 0.234 -0.00881 6118 37 1 38 187 28 -1.312 0.398 2.792 0.164 -0.00939 6118 37 1 38 187 22 -1.309 0.184 2.834 0.203 -0.00953 6118 71 1 42 188 21 -0.332 0.294 1.511 0.182 -0.00508 6118 71 1 42 188 23 -0.310 0.236 1.545 0.193 -0.00519 6118 71 1 42 188 27 -0.028 0.210 1.691 0.193 -0.00568 6118 71 1 42 188 29 0.155 0.182 1.908 0.201 -0.00641 6118 71 1 42 188 21 0.125 0.280 1.960 0.186 -0.00659 6118 89 1 45 189 17 0.397 0.348 0.809 0.258 -0.00272 6118 89 1 45 189 27 0.284 0.342 0.935 0.138 -0.00314 6118 89 1 45 189 24 0.213 0.201 1.098 0.132 -0.00369 6118 89 1 45 189 19 0.329 0.131 1.117 0.148 -0.00375 6118 89 1 45 189 21 0.261 0.161 1.374 0.210 -0.00462 6118 124 1 49 193 7 0.667 0.591 2.532 4.663 -0.00851 6118 124 1 49 193 20 0.568 0.384 0.887 0.148 -0.00298 6118 124 1 49 193 24 0.493 0.526 1.025 0.182 -0.00345 6118 124 1 49 193 16 0.664 0.323 1.275 0.151 -0.00429 6118 124 1 49 193 16 0.600 0.609 1.413 0.172 -0.00475 6118 142 1 51 195 12 -0.049 0.320 0.431 0.250 -0.00145 6118 142 1 51 195 19 -0.125 0.420 0.536 0.177 -0.00180 6118 142 1 51 195 13 -0.028 0.409 0.791 0.309 -0.00266 6118 142 1 51 195 15 -0.319 0.396 1.069 0.213 -0.00360 6118 142 1 51 195 17 -0.224 0.432 1.003 0.265 -0.00337 6118 176 1 54 201 12 1.548 0.286 -0.893 0.160 0.00300 6118 176 1 54 201 18 1.322 0.198 -0.663 0.445 0.00223 6118 176 1 54 201 12 1.238 0.238 -0.473 0.408 0.00159 6118 176 1 54 201 12 1.045 0.147 -0.316 0.388 0.00106 6118 176 1 54 201 11 1.021 0.228 -0.222 0.260 0.00075 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): y QA Procedures Done (y/n): y UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 38939, plate -9999 ------- 23:23 38939 0.0 bias 23:27 38940 0.0 bias ----- sequence 38941, plate 1729 ------- 23:35 38941 10.0 flat 23:37 38942 2.0 arc 23:40 38943 2.1 arc 23:43 38944 2.1 arc 23:59 38945 0.0 bias ----- sequence 38946, plate 2491 ------- 18.6 15.3 23.3 19.9 DONE 02:32 38946 10.0 flat 02:34 38947 2.0 arc 03:15 38948 900.1 target 03:32 38949 900.1 target 03:53 38950 1100.1 target 03:56 38951 10.0 flat 03:58 38952 2.0 arc ----- sequence 38953, plate 2140 ------- 11.0 11.1 12.5 10.3 09:03 38953 10.0 flat 09:05 38954 2.0 arc 09:24 38955 900.1 target 09:41 38956 900.1 target ----- sequence 38957, plate -9999 ------- Done while re-initializing 10:03 38957 0.0 bias Viktor's SOP session 10:15 38958 0.0 bias ----- sequence 38959, plate 1729 ------- 4.5 3.6 3.3 5.2 10:31 38959 10.0 flat 10:34 38960 2.0 arc 10:52 38961 900.1 target ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 02:30Z 1 2491 49.76 0.0042 67.14 0.0012 221.99 0.0078 1.000140 09:08Z 6 2140 283.47 -0.0038 62.61 0.0009 117.23 0.0070 1.000000 10:39Z 9 1729 -55.22 -0.0013 51.33 0.0010 135.78 -0.0077 1.000140 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 02:30Z 1 2491 1.00014 -1383 17 25 50 67.1 5.4 6 294 -- 1.3 09:08Z 6 2140 1.00000 0 1171 25 284 62.7 5.8 5 188 -- 1.4 10:05Z 6 0 1.00004 -395 851 0 273 50.8 5.4 7 219 10:17Z 6 2140 1.00004 -395 860 0 271 48.2 5.2 7 188 10:39Z 9 1729 1.00014 -1383 82 50 -55 51.4 4.9 8 218 -- 1.4 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:08Z 43 19 14 331 (NNW) 1368 - Clear 22:40Z 42 17 13 345 (NNW) 1261 - Clear 23:12Z 43 14 21 303 (WNW) 1221 - Clear 23:42Z 43 14 10 335 (NNW) 1296 - Clear 00:12Z 44 15 11 359 (NNW) 1420 - Clear 00:43Z 44 14 10 321 (NW) 1509 - Clear 01:16Z 43 14 4 328 (NNW) 1579 - Clear 01:47Z 42 15 7 354 (NNW) 1596 - Clear 02:19Z 41 17 10 298 (WNW) 1440 - Clear 02:51Z 42 18 6 299 (WNW) 1385 - Clear 03:22Z 42 18 4 273 (W) 1428 - Clear 03:53Z 42 22 16 297 (WNW) 1583 - Clear 04:24Z 42 23 12 274 (W) 2321 - Clear 04:56Z 43 22 11 268 (W) 2518 - Clear 05:27Z 43 21 11 266 (W) 2508 - Clear 05:58Z 42 21 12 248 (WSW) 2415 - Clear 06:28Z 42 21 9 216 (SW) 2242 - Clear 06:59Z 42 20 11 231 (SW) 2443 - Clear 07:29Z 42 20 6 255 (WSW) 2447 - Clear 07:59Z 42 20 6 242 (WSW) 2444 - Clear 08:29Z 42 20 7 226 (SW) 2363 - Clear 08:59Z 41 22 7 233 (SW) 2349 - Clear 09:30Z 41 22 5 195 (SSW) 2368 - Clear 10:01Z 41 22 6 194 (SSW) 2456 - Clear 10:32Z 41 22 7 185 (S) 2577 - Clear 11:03Z 42 19 6 210 (SSW) 2971 - Clear ---=== TELESCOPE STATUS ===--- 00:40Z East door open, west door cracked, fans and louvers open 01:45Z Enclosure off 11:00Z Enclosure on, fans off, doors and louvers closed Telescope parked at 121,30 with Cart 9 (Plate 1729) mounted Counterweights at 255 Spectro autofill reconnected No interlocks bypassed Dewar weights at 11:15Z Imager: 300lbs 25psi Spectro: 198lbs 21psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_21_0 Watcher: v2_34_0 MCP: v6_3_1 TPM: tpm_v3_3_0 AstroDa: v15_11_1 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_14 SoS: v5_1_0 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.8540 -5.8440 Axial B -5.6740 -5.6770 Axial C 0.9410 0.9420 Trans D -9.9950 -9.9940 Lateral E 10.3759 10.3759 Lateral F 11.9507 11.9380 GALILS Commanded: 4700. -2450. 3000. -7800. 3300. 3000. Actual: 4711. -2457. 2997. -7825. 3296. 3024. SETMIR VALUES PriDesOrient: 0.00 -1.68 7.05 385.25 92.02 PriOrient: 0.00 -1.52 7.02 384.99 93.81 SECONDARY: ---------- Focus: 0.00 Air Temp.: 6.9 Alt.: 29.658793 MIGS TONIGHT NOMINAL Axial A 1.5400 1.5420 Axial B 1.0530 1.0470 Axial C 1.1720 1.1650 Trans D -0.1180 -0.1240 GALILS Commanded: 1593438. 1604191. 1554395. -6800. -5900. Actual: 1593447. 1603642. 1554316. -6820. -5882. SETMIR VALUES SecDesOrient: 1257.00 15.87 0.00 -50.00 136.76 SecOrient: 1257.16 15.97 0.15 -49.46 136.67 ---=== PROBLEMS IN DETAIL ===--- Spectro Focus ------------- Temperatures at time of focus check: tpm_TM_PMSSAvg 3.80357 tpm_TM_TelairAvg.VALA 5.4275 Motor positions before check: sp1 Coll_motor_A 7164 Coll_motor_B 7182 Coll_motor_C 7149 sp2 Coll_motor_A -5340 Coll_motor_B -8723 Coll_motor_C -7952 postCalib looks fine. First iteration of goSpecFocus gave: sp1 red l-r = 0.090 pixels, tolerance = 0.15 pixels, focused sp2 red l-r = 0.133 pixels, tolerance = 0.15 pixels, focused sp1 blue l-r = 0.121 pixels, tolerance = 0.15 pixels, focused sp2 blue l-r = -0.269 pixels , tolerance = 0.15 pixels, not focused yet Predict a movement of 10 steps for sp2 blue Looks like we're in focus. What does collimate say? COLLIMATE: Mean offset = 0.14 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of -5.1 degrees for spectro-1 COLLIMATE: Mean offset = -0.15 pix COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 5.4 degrees for spectro-2 COLLIMATE: Mean offset = 0.07 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -625 steps for spectro-1 COLLIMATE: Mean offset = 0.06 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -548 steps for spectro-2 We'll call it good and watch SoS returns on b2 just in case it decides to go out of spec on us during the night, but odds are it won't get that cold. (Turns out we were fine...) Bias Percentiles ---------------- Filename 02% 05% 10% 50% 90% 95% 98% ------------------- ------ ------ ------ ------ ------ ------ ------ sdR-b1-00038945.fit -5.7 -4.6 -2.6 2.0 6.8 8.6 10.0 sdR-b1-00038945.fit -5.7 -4.6 -2.6 2.0 6.8 8.6 10.0 sdR-b2-00038945.fit -8.2 -6.4 -4.4 1.0 7.1 8.3 10.7 sdR-b2-00038945.fit -8.2 -6.4 -4.4 1.0 7.1 8.3 10.7 sdR-r1-00038945.fit -7.0 -5.3 -3.8 1.7 6.7 8.2 10.4 sdR-r1-00038945.fit -7.0 -5.3 -3.8 1.7 6.7 8.2 10.4 sdR-r2-00038945.fit -8.0 -6.6 -4.9 0.8 6.7 8.1 10.2 sdR-r2-00038945.fit -8.0 -6.6 -4.9 0.8 6.7 8.1 10.2 The instChange That Almost Didn't Happen ---------------------------------------- Around 07:00Z, after decising that further imaging was a vain effort at best, we headed down to the telescope to switch back to spectroscopy. Viktor decided not to issue the inst change from the control room but rather to track the telescope to (121,90,0) from the laptop at the telescope. I prepared the lindy cart with Cart 6 and got the turtle ready to put the corrector back on the telescope. The telescope reached zenith and the proper azimuth and I went downstairs to switch the tbar latch from telescope to doghouse, thinking that the scope had come to a full and complete stop after putting its seatback up and returning its tray table to the full and upright position. I came back up from downstairs and went round the south side of the telescope, engaging the altitude pin as I went by. At about the same instant, Viktor opened the door to the doghouse. What neither of us noticed was that the rotator was still about twenty degrees out from a rotation of zero degrees. The altitude brake engaged, as it well should have. Thus we found ourselves in a rather curious situation. The doghouse door was up, but the rotator was positioned such that the red camera cold head and assorted attachments from sp2 prevented the door from closing. The clearance was a matter of milimeters, or a couple of degrees of rotation. And now, with the doghouse door open, the altitude brake refused to release. But with the spectrograph camera head in the way, we couldn't lower the door to get the brake to release. We checked everything else we could think of that would restrict altitude motion: stop buttons, the locking pin in the imager cart, disengaged the telescope altitude pin. But it was clear what the problem was, and that was the inability to close the doghouse door. My instinct was to find a way around the restriction of the door being open, preventing the altitude brake from releasing. We tried looking for a kind of limit switch I expected would sense the open/closed state of the doghouse, but without results (there is a strange pushbutton-type switch on the south side of the doghouse, when the scope is parked at stow, that is connected to nothing but an aluminum block... I never noticed it before. What's that all about?) That didn't work, so I thought ot try disabling the interlocks I expected to relate to this situation. I went downstairs and did indeed disable them, but neglected to put in the key that actually causes the bypasses to activate. Needless to say, that accomplished nothing. So we did the next best thing, and that was to get French out of bed in the middle of the night. (Let me use this brief aside to send a shout-out to French, who by the way is *THE MAN*. We would have been stuck stuck STUCK without him tonight.) French walked us through all of the possible/likely reasons for what was happening and ultimately decided to have us move the rotator by hand, such that the doghouse door would clear the spectrograph camera head in the way. We turned off the rotator amplifier and found the rotator easier to move by hand than I expected. With the camera head out of the way, there was plenty of clearance for the doghouse door, and we closed it. Turned the rotator amplifier back on. The altitude pin was already out, so we tried getting back up on our collective horse by issuing an 'axis init' in TCC. That was all fine and well except that the altitude brake would not release. We commanded the rotator back to 0 degrees and verified with the tape mark on the front of the scope that we didn't cause too terrible a slip when we forced the rotator by hand. Yet the al brake would still not release. So then we entered a troubleshooting chain for that problem. Again, we considered all of the factors that would render the altitude axis incapable of moving. No stop buttons in, doghouse door closed, alt pin disengaged, wind baffle powered up, etc. We were nearly stumped when French asked me to look at the "root causes" window in the MCP Interlocks display. That window said something about "sad_mount1" and "sad_mount2", which to French meant the tbar latch was in a bad or wrong state. Since the imager was still on the telescope, we went downstairs and switched the handle back to "telescope", then came upstairs and tried releasing the alt brake again. This time, it worked. We then proceeded with the rest of the instChange as normal, returning the tbar handle to "doghouse" for the protection of the imager. After putting on the corrector and Cart 6, I sent Viktor back to the control room thinking I was just going to do a test slew and then hand the scope over to him. But I had run the counterweights up too far when we put the cartridge on the telescope, and after moving about 20 degrees down in altitude from the zenith, we had an overcurrent on the alt motors because of the out-of-balance condition and the brake came back on. There was also a slip of the altitude axis at this time. I cleared the slip but could not get the brake to release again, so I did the only thing I could think to do, which was to run the counterweights down to their proper values while the scope was not at zenith. While far from ideal cirumstances, it worked. It produced errors too large to correct on the alt axis when we went on to the field of Cart 6, but we forced fiducials and were back in business. Bottom line: Not enough communication between the observers doing the instrument change to make clear who was doing what at what time. We'll try to avoid this in the future. Bad news: We lost almost an hour and forty-five minutes chasing down these somewhat illusory ghosts. Good news: The seeing was no good during this time anyway, at least according to DIMM. Ah, well. It could have been worse.... somehow.