APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday May 14, 2004 (MJD 53140) ---=== OBSERVING TEAM ===--- John Barentine Jurek Krzesinski Mike Harvanek (swing) Craig Loomis (tech support by phone) ---=== OBSERVING PLAN ===--- Science and then collimation after moonrise. ---=== OBSERVING SUMMARY ===--- Non-photometric for imaging; attempted three plates (1609, 1625, 1643). Finished out 1625, got one useful exposure on 1609 and none on 1643. Did not attempt requested collimation check with imager because of clouds. ---=== OBSERVING LOG ===--- Afternoon: ---------- Spectro collimation was checked; no adjustments were necessary. Details are in the Problems section. Night: ------ 01:25Z Fans on, doors cracked. 02:05Z Retracted the enclosure. Fairly thick cirrus to our west, winds increasing after sunset. 02:20Z Loaded Cart 3, Plate 1609. No stars on gotoField; found an FK5 star after clearing last night's offsets. We note that SOS has this plate erroneously marked as "Plate 0" (see Problems section). During the second science exposure the guider efficiency dropped like a rock, so we ended the stare early at 03:58Z. Did a third exposure at 05:38Z after the clouds thinned, but we did not finish the plate. Only our first exposure is useful. 06:06Z Loaded Cart 5, Plate 1625. Stars on gotoField after clearing previous offsets. We have had trouble getting the guider to keep running once started; it hangs repeatedly, requiring restarts. Finished this plate with one exposure. 06:55Z Loaded Cart 6, Plate 1643. Guider continues to give us trouble (frames are now not transferring properly; see Problems section for details). During this time the sky cleared up considerably. By the time the guider problem was solved, the sky became soupy once again. 09:02Z We just noticed that a critical system error has occured in lavaLamp. See Problems section for details. 11:00Z Closed up at twilight. Sky did not clear up remainder of night. Checked dewar weights as requested: spectro, 110lbs; imager, 211lbs. No dewar swap necessary. SOS claims the arcs on Plate 1417 are not reducible but the data are okay -- they were taken earlier in the afternoon for collimation checks. endNight rejoiced ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4635 00:02Z 00:18Z 100 O -57.18 -53.11 41 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 26674, plate -9999 ------- 23:57 26674 0.0 bias ----- sequence 26675, plate 1417 ------- 00:03 26675 10.0 flat collimation 00:05 26676 2.0 arc " 00:09 26677 2.1 arc " 00:12 26678 2.1 arc " ----- sequence 26680, plate 0 ------- 03:07 26680 10.0 flat 03:10 26681 2.0 arc 03:35 26682 1200.1 target 03:59 26683 1239.2 target endStare early; clouds 04:42 26684 1800.1 target 05:28 26685 10.0 flat 05:30 26686 2.0 arc 06:05 26687 1500.1 target ----- sequence 26688, plate 1625 ------- 06:24 26688 10.0 flat 06:26 26689 2.0 arc 06:50 26690 1200.1 target ----- sequence 26691, plate -9999 ------- 07:16 26691 0.0 bias ----- sequence 26692, plate -9999 ------- 07:55 26692 0.0 bias ----- sequence 26693, plate 1643 ------- 08:17 26693 10.0 flat 08:19 26694 2.0 arc 09:56 26695 0.0 bias 10:08 26696 10.0 flat 10:10 26697 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 03:19Z 3 1609 0.48 -0.0001 69.60 0.0031 180.35 0.0279 1.000250 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 03:20Z 3 1609 1.00025 -2468 -712 200 0 69.6 7.5 18 143 BG38 1.6 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:33Z 58 24 15 253 (WSW) 955 - 00:06Z 55 23 12 192 (SSW) 929 - 01:34Z 49 24 8 180 (S) 1188 - 02:07Z 47 25 16 151 (SSE) 1273 - 02:39Z 46 26 20 138 (SE) 1320 - 03:10Z 45 26 15 135 (SE) 1203 - 03:43Z 46 21 15 165 (SSE) 1375 - 04:16Z 47 18 15 152 (SSE) 1264 - 04:47Z 46 12 17 142 (SE) 1435 - 05:19Z 47 6 15 138 (SE) 1418 - 05:52Z 49 18 8 178 (S) 1399 - 06:25Z 49 18 9 178 (S) 1401 - 06:55Z 49 18 8 180 (S) 1393 - 07:25Z 49 20 7 181 (S) 1543 - 07:56Z 49 21 8 180 (S) 1559 - 08:28Z 48 20 10 181 (S) 1459 - 08:58Z 47 18 6 181 (S) 1472 - 09:29Z 48 19 10 179 (S) 1416 - 10:01Z 48 20 11 180 (S) 1335 - 10:35Z 48 25 10 185 (S) 1308 - 11:07Z 49 25 9 170 (S) 1353 - 11:39Z 48 22 9 178 (S) 1290 - ---=== TELESCOPE STATUS ===--- ---=== SOFTWARE USED ===--- IOP/SOP: v3_137_0 Watcher: v2_29_0 MCP: v5_25_0 TPM: tpm_v2_42_0 AstroDa: v14_47 TCC: TCC 2.7.2 November 11 2003 sdssProcedures: v1_90 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5380 0.5430 Axial B 0.6200 0.6240 Axial C 1.2230 1.2260 Trans D -8.4530 -8.4440 Lateral E 10.8458 10.8712 Lateral F 10.8331 10.8458 GALILS Commanded: 4600. -2850. 850. -9250. 18400. 17850. Actual: 4600. -2866. 866. -9275. 18411. 17825. SETMIR VALUES PriDesOrient: 0.00 0.00 15.20 829.90 1.60 PriOrient: 0.00 0.00 14.91 830.49 3.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 13.2 Alt.: 29.999588 MIGS TONIGHT NOMINAL Axial A 1.5100 1.5080 Axial B 1.0990 1.0960 Axial C 1.1100 1.1070 Trans D -0.0010 0.0040 GALILS Commanded: 1628506. 1548071. 1574213. -3900. -8750. Actual: 1628641. 1548187. 1574504. -3899. -8726. SETMIR VALUES SecDesOrient: 1257.00 0.00 -25.00 0.00 146.27 SecOrient: 1256.86 0.06 -25.01 0.26 146.52 ---=== PROBLEMS IN DETAIL ===--- Spectro collimation check: -------------------------- inital sp1 Coll_motor_A 4312 Coll_motor_B 13698 Coll_motor_C 10412 inital sp2 Coll_motor_A 2508 Coll_motor_B 2504 Coll_motor_C 2510 im> caget tpm_TM_PMSSAvg tpm_TM_TelairAvg.VALA tpm_TM_PMSSAvg 9.86143 tpm_TM_TelairAvg.VALA 11.245 rotator at 45 degrees mount Everything was in focus so no adjustments were necessary: COLLIMATE: Mean offset = 0.02 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Mean offset = -0.04 pix COLLIMATE: Camera b2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Mean offset = 0.08 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Mean offset = 0.00 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) Notorious "Plate 0": -------------------- On inspecting SOS results for Plate 1609, we found that SOS had labeled this plate "0". We first manually reset the plateId using apofix but SOS evidently didn't pick up the change after the next exposure. SOS was then stopped and the reduction directories were removed from sos.apo.nmsu.edu. We restarted SOS and all files have been reduced properly. SOS now identifies the plate correctly as "1609". Spectro guider problem: ----------------------- During the acquisition of the field of Plate 1643, guider frames stopped being transferred ot the guider monitor. They were being taken and written to disk, but without a display we had no way of keeping an eye on them. In addition, startGuider and stopGuider commands were backing up and would suddenly be issued in rapid succession: 07:02:18 34221 SOPGUI: startGuider 10 07:02:27 34221 SOPGUI: startGuider 10 07:05:15 34221 SOPGUI: stopGuider 07:05:25 34221 SOPGUI: startGuider 10 07:05:52 34221 SOPGUI: stopGuider 07:05:53 34221 SOPGUI: startGuider 10 We found that when a stopMon was issued, transferring of frames would resume. Jurek repeatedly tried exiting and restarting SOPGUI, then tried restarting SOP altogether. Meanwhile, the guider frames eventually stopped being written to disk altogether (tccmon indicated exposures *were* being taken, however). From the guider log: 2.64 2004-05-15 06:03:27.608Z 06:03:50.246Z 0 0 I GCamReply="2 2 0 0 384 256 20. 00 6 10.00 "image: binXY begXY sizeXY expTime camID temp"" 4.17 2004-05-15 06:03:59.119Z 06:04:23.286Z 0 0 I GCamReply="2 2 0 0 384 256 20. 00 6 10.00 "image: binXY begXY sizeXY expTime camID temp"" 11.12 2004-05-15 06:06:07.289Z 06:06:38.406Z 0 0 I GCamReply="2 2 0 0 384 256 20 .00 6 10.00 "image: binXY begXY sizeXY expTime camID temp"" 28.21 2004-05-15 06:12:47.358Z 06:13:20.569Z 0 0 I GCamReply="1 1 0 0 768 512 5. 00 6 10.00 "image: binXY begXY sizeXY expTime camID temp"" could not reply: 0 0 F GCamReply="Time limit exceeded; exposure aborted " 17.01 2004-05-15 06:14:00.398Z 06:14:22.409Z 0 0 F GCamReply="Time limit exceede d; exposure aborted " could not reply: 0 0 F GCamReply="Time limit exceeded; exposure aborted Craig said this problem happens when a guider image comes in as SOS is getting its information, causing the system to momentarily become slow. The guider Mac then gets confused as it lags behind what's actually happening and first slows, then halts. An occasional failure then becomes catastrophic as the Mac gets out of sync. A diagnostic Craig recommended is: ssh arc@pod (password in control room) cat logs/from.tcc25m/current.log | gcamtime.py (Returns a log of guider events) - or - tail -f logs/from.tcc25m/current.log | gcamtime.py (Updates events to screen as they are logged) (This information will be folklored for future reference.) Command initiation-to-completion times are given in the first column; normal times for routine guiding are ~2-3 seconds (the unbinned images taken in field acquisition require longer times). The output copied above shows that normal guiding ended at 06:03:59Z; times got progressively longer until the command timed out at 06:12:47Z. From that time on, only rebooting of the guider Mac (to resync the operations) would have saved us. Craig mentioned a problem with the NFS stacks on the guider Mac, which fixing might solve the problem. The problem remained that the guider would not run in a loop once started. One exposure would be taken and displayed in the guider monitor, then the sequence would quit. Craig, still on the phone, noted complaints in astrolog about the guider software not find biases, darks, flats, etc. for the current frame. This is evidently a bug in SOP which does not take into account the fact that the guider Mac was restarted. That restart also resets the image sequence number. The solution was to do a new gotoField -gflat, which then allows incoming guider images to be properly calibrated. After doing this, we were on our way again. lavaLamp Critical System Error: ------------------------------- Shortly after 09:00Z, we noticed the lavaLamp was apparently switched off and began considering this the source of our poor luck tonight. After some testing, we determined a burnt out lightbulb is to blame. Watcher initially indicated an alert, but it could be cleared by clicking on the icon -- failure of TCC to talk to the lavaLamp? A PR has been filed...