APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday January 29, 2006 (MJD 53765) ---=== OBSERVING TEAM ===--- John Barentine Steph Snedden Craig Loomis - phone support French Leger - phone support ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Four plates completed - 2374, 1987, 2400 and 2350. Problems: 1) At low altitude, we often need more than one axis init to clear the alt brake. Are the windbaffle touch sensors in need of alignment? 2) All data transfer to the APO weather page except satellite images stopped at 21:36 MST. 3) The DIMM acted up in azimuth (again). 4) Two of the four flat field lamps failed to turn on during a preCal. ---=== OBSERVING LOG ===--- Afternoon: ---------- Focus checked but not adjusted, since it should be in range of tonight's expected lows. We note that it usually takes more than one axis init to clear the altitude brake when we are at low altitude (6 degrees). Could the engineering staff have a look at the windbaffle bump sensors, please? The first attempt to take a spectro bias -init returned with the message: SOPGUI: goStare 0 -bias -init -plateId -9999 MECH: FAILURE on ftelnet 1000000 1500000 2 t-g-sdss-2.apo.nmsu.edu 2300 sp1> The second attempt was fine. The rest of afternoon checkout was trouble-free. Night: ------ While setting up the DIMM, we noticed that no matter what slew speed the hand paddle was set to, the azimuth motion stayed at a fast slew speed, with the previously noted jerky motion. This seems a likely culprit for the loss of Polaris the last few nights. If the software tried to recenter Polaris at a fast slew speed, it would almost certainly lose it. This did happen a few times during the night and a manual offset to the E from the control program would bring Polaris back for a while. Later in the night, the VNC server got in an unhappy state. We could have gone and looked for the process to kill it, but we chose to not do this, since it was not photometric and seeing was less of an issue. The VNC problem is probably related to the fact that all weather information on Galileo stopped updating at 21:36 PM MST. 01:25Z - Cartridge 8, plate 2417 - segue faint Needed an FK5 to find the field. We also ran into the same problem that other observers had seen on previous nights with the UDP port conflict. We wound up following the folklore to resolve the problem, but we also called Craig. Craig tracked an intricate dance of instrument grabs by John and I that caused our problem tonight and is detailed in the problems section. It doesn't explain UDP port problems that people have had on previous nights. We plan to try to troubleshoot this better tomorrow afternoon / evening during the transition from swing to night shift. Almost finished the plate, just needs one more short exposure. Total exposure time: 9300 s. Not quite photometric, so we stay with spectroscopy. Note that we cleaned the all-sky camera mirror this afternoon, and the new photometric level appears to be at a sigma of about 3, not 6. Clouds were visible above 30 deg alt with the sigma < 6. 05:03Z - Cartridge 6, plate 2400 - segue faint GSOGTF. This plate is DONE with a total exposure time of 9000 s. 08:15Z - Cartridge 5, plate 1987 - Legacy Needed an FK5 to find the field. This plate declared DONE with a total exposure time of 1220 s (added to signal obtained on a previous night). 09:20Z - Cartridge 4, plate 2374 - Legacy GSOGTF. This plate is DONE with a total exposure time of 2160s (added to previous signal). 10:24Z - Cartridge 7, plate 2350 - Legacy GSOGTF. Finished this plate with a total exposure time of 2700 s. 11:43Z _ Cartridge 9, plate 2144 - Legacy GSOGTF. Two flat field lamps failed to come on during precals. Both failed at the same time, so if it is two bad lamps, it is quite a coincidence. We called French, and he will investigate today. He wondered if it was a bad relay. This problem prevented any observations on this plate. Since it was so close to twilight, we closed. endNight finished happily. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 6001 22:50Z 23:22Z 100 O -179.10 -171.22 43 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n):y QA Procedures Done (y/n):y UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 23:11 36867 2.1 arc focus check 23:13 36868 2.1 arc focus check ----- sequence 36869, plate 2417 ------- 01:58 36869 0.0 bias 02:02 36870 10.0 flat 02:04 36871 2.0 arc 02:29 36872 1200.1 target 02:57 36873 1500.2 target 03:24 36874 1500.2 target 03:52 36875 1500.2 target 04:24 36876 1800.2 target 04:57 36877 1800.1 target 05:00 36878 10.0 flat 05:02 36879 2.0 arc ----- sequence 36880, plate 2400 ------- 05:18 36880 10.0 flat 05:20 36881 2.0 arc 05:54 36882 1800.1 target 06:27 36883 1800.1 target 07:00 36884 1800.1 target 07:32 36885 1800.1 target 08:05 36886 1800.1 target 08:08 36887 10.0 flat 08:10 36888 2.0 arc ----- sequence 36889, plate 1987 ------- 08:26 36889 10.0 flat 08:29 36890 2.0 arc 08:53 36891 1200.1 target 09:13 36892 1020.1 target ----- sequence 36893, plate 2374 ------- 09:29 36893 10.0 flat 09:32 36894 2.0 arc 09:58 36895 1200.1 target 10:17 36896 960.1 target ----- sequence 36897, plate 2350 ------- 10:32 36897 10.0 flat 10:35 36898 2.0 arc 10:56 36899 900.1 target 11:14 36900 900.1 target 11:32 36901 900.1 target ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 02:15Z 8 2417 216.11 0.0049 79.17 0.0010 41.14 0.0189 1.000100 03:37Z 8 2417 241.43 0.0049 65.58 0.0010 79.01 0.0189 1.000100 05:15Z 6 2400 108.45 0.0037 63.42 0.0020 -90.37 -0.0072 1.000100 07:04Z 6 2400 138.05 0.0037 84.00 0.0020 -44.86 -0.0072 1.000100 08:24Z 5 1987 103.84 0.0006 54.50 0.0015 -100.57 -0.0162 1.000100 09:27Z 4 2374 -38.74 -0.0028 74.00 0.0008 145.91 0.0084 1.000100 10:30Z 7 2350 277.92 -0.0010 66.41 0.0011 111.09 -0.0197 1.000100 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 02:14Z 8 2417 1.00010 -988 421 120 215 79.3 0.8 18 299 - 1.6 03:37Z 8 2417 1.00010 -988 472 150 241 65.6 0.8 13 330 - 1.8 05:15Z 6 2400 1.00010 -988 427 100 108 63.4 0.4 7 359 - 1.8 07:03Z 6 2400 1.00010 -988 473 175 138 84.0 -0.9 14 349 - 1.5 08:24Z 5 1987 1.00010 -988 510 160 104 54.5 -1.3 21 351 - 1.6 09:27Z 4 2374 1.00010 -988 478 170 -39 74.0 -0.6 18 10 - 1.9 10:30Z 7 2350 1.00010 -988 420 100 278 66.4 -0.9 23 18 - 1.6 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:05Z 34 12 17 315 (NW) 326 - pcl 23:35Z 34 11 10 281 (W) 306 - " 00:05Z 33 12 14 284 (WNW) 336 - " 00:35Z 33 11 17 289 (WNW) 307 - " 01:07Z 33 12 19 293 (WNW) 356 - " 01:37Z 33 12 14 298 (WNW) 326 - " 02:09Z 33 12 17 306 (NW) 381 - " 02:40Z 33 12 16 327 (NNW) 386 - " 03:10Z 33 12 9 342 (NNW) 479 - " 03:40Z 33 12 7 351 (NNW) 361 - " 04:11Z 32 11 13 9 (N) 317 - " 04:41Z 32 11 10 357 (NNW) 264 - mostly clear 05:11Z 32 11 15 11 (N) 318 - " 05:42Z 32 11 13 353 (NNW) 293 - " 06:13Z 31 11 23 22 (NNE) 241 - " 06:44Z 30 11 13 13 (NNE) 246 - " 07:14Z 30 11 20 36 (NE) 249 - " 07:45Z 29 12 11 4 (N) 242 - " 08:15Z 29 13 9 25 (NNE) 240 - " 08:45Z 29 13 16 18 (NNE) 217 - " 09:15Z 30 12 16 350 (NNW) 214 - " 09:47Z 30 13 6 74 (ENE) 211 - " 10:18Z 30 13 18 6 (N) 198 - " 10:48Z 30 13 15 350 (NNW) 157 - " 11:19Z 30 13 19 7 (N) 161 - clear 11:49Z 30 14 16 357 (NNW) 243 - " 12:20Z 30 14 14 7 (N) 266 - " 12:52Z 30 14 16 21 (NNE) 279 - " ---=== TELESCOPE STATUS ===--- 23:40Z - Doors opened, ventilation on. 00:30Z - Building opened. 12:27Z - Building closed, ventilation off. Telescope stowed at 30 deg alt with cartridge 9 mounted. Counterweights at 250. Spectro autofill connected. 12:49Z - Spectro dewar weight: 128 lb 12:49Z - Imager dewar weight: 303 lb ---=== SOFTWARE USED ===--- IOP/SOP: v4_12_0 Watcher: v2_31_0 MCP: v5_26_2 TPM: tpm_v3_3_0 AstroDa: v15_10 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_09 SoS: v5_1_0 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.8420 -5.8440 Axial B -5.6800 -5.6770 Axial C 0.9330 0.9420 Trans D -9.9930 -9.9940 Lateral E 10.3632 10.3759 Lateral F 11.9507 11.9380 GALILS Commanded: 4700. -2450. 3000. -7800. 3300. 3000. Actual: 4700. -2450. 3000. -7800. 3300. 3000. SETMIR VALUES PriDesOrient: 0.00 -1.68 7.05 385.25 92.02 PriOrient: 0.00 -1.52 7.02 384.99 93.81 SECONDARY: ---------- Focus: 0.00 Air Temp.: 1.6 Alt.: 29.999526 MIGS TONIGHT NOMINAL Axial A 1.5410 1.5420 Axial B 1.0610 1.0470 Axial C 1.1730 1.1650 Trans D -0.1210 -0.1240 GALILS Commanded: 1593362. 1603567. 1554106. -6800. -5900. Actual: 1593450. 1603650. 1554300. -6800. -5900. SETMIR VALUES SecDesOrient: 1257.00 15.87 0.00 -50.00 136.76 SecOrient: 1256.90 15.91 -0.01 -49.56 136.71 ---=== PROBLEMS IN DETAIL ===--- grabInst dance: -------------- Craig's notes: Just to give you a sense of how I'm reading this, and on what evidence.... egrep '((Stopping|Starting) grabInst)|( 30[34]10)|(IOP Running as)|(original:.*g rab)' /astrolog/iop.murmur.log 2006-01-30 00:58:43Z sdsshost2 IOP 17055 TEXTONLY trace 1, 0 snedden original: {grabInst -drop} 2006-01-30 00:58:43Z sdsshost2 IOP 17055 TEXTONLY Stopping grabInstruments ^^ End of afternoon checkout by snedden 2006-01-30 01:39:36Z sdsshost2 IDA 16411 TEXTONLY trace 1, 0 jcb original: {tk ButtonDown .setupgui.grabinst.button} 2006-01-30 01:39:45Z sdsshost2 IDA 16411 TEXTONLY Starting grabInstruments 2006-01-30 01:39:45Z sdsshost2 IOP 16411 TEXTONLY IOP running as Backup Relay Server ^^ jcb successfully grabs, but gets a "FAILURE on ftelnet" message which makes him wonder. 2006-01-30 01:40:16Z sdsshost2 IDA 17055 TEXTONLY trace 1, 0 snedden original: {tkButtonDown .setupgui.grabinst.button} 2006-01-30 01:40:29Z sdsshost2 IDA 17055 TEXTONLY Starting grabInstruments ^^ Looking into that, snedden tries to grab... 2006-01-30 01:40:29Z sdsshost2 IOP 16411 TEXTONLY trace 1, 0 jcb original: {gr abInst -drop} 2006-01-30 01:40:30Z sdsshost2 IOP 16411 TEXTONLY Stopping grabInstruments ^^ in the same second that the grab actually starts, jcb starts a drop, which completes 2006-01-30 01:40:37Z sdsshost2 IDA 16411 TEXTONLY trace 1, 0 jcb original: {gr abInst -handlers 900} ^^ jcb starts a -handlers cmd 2006-01-30 01:40:39Z sdsshost2 IOP 17055 TEXTONLY IOP running as Backup Relay Server ^^ the grab started by snedden succeeds 2006-01-30 01:40:49Z sdsshost2 IDA 16411 TEXTONLY Starting grabInstruments ^^ the -handlers command starts 2006-01-30 01:40:52Z sdsshost2 IOP 17055 TEXTONLY trace 1, 0 snedden original: {grabInst -dropgrabInst -drop} ^^ snedden typos a "grabInst -drop" -- 17055 still has the instruments here. 2006-01-30 01:41:01Z sdsshost2 IDA 16411 TEXTONLY Stopping grabInstruments ^^ jcb's -handlers continues 2006-01-30 01:41:05Z sdsshost2 IDA 16411 TEXTONLY Couldn't open udp socket 304 10 : Address already in use ^^ the first error about port 30410 At this point, snedden's PID 17055 has the grab, nobody realizes it, and jcb i s getting udp port 30410 errors. Kill-all-suspects recovery techniques begin. Va rious questions about what info the observers should but aren't getting come to mind. Craig then sleuthed the previous nights' UDP port conflicts. His notes for those cases follow: More-or-less the same plot line last night. Before that are the two on 53757/8, which are different (the keepalive triggers a drop, somehow. It is slightly intriguing that the failing -handlers were run at 600s instead of 900s). I'll keep looking. - craig 2006-01-28 23:54:27Z sdsshost2 IDA 8408 TEXTONLY trace 1, 0 jcb original: {grabInst -handlers 900} 2006-01-28 23:54:33Z sdsshost2 IDA 8408 TEXTONLY Starting grabInstruments 2006-01-28 23:54:33Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server 2006-01-29 00:04:36Z sdsshost2 IOP 8408 TEXTONLY trace 1, 0 jcb original: {schedule grabInstruments {-handlers 900 -reconnect} -600} 2006-01-29 00:04:36Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server 2006-01-29 00:14:44Z sdsshost2 IOP 8408 TEXTONLY trace 1, 0 jcb original: {schedule grabInstruments {-handlers 900 -reconnect} -600} 2006-01-29 00:14:44Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server 2006-01-29 00:24:49Z sdsshost2 IOP 8408 TEXTONLY trace 1, 0 jcb original: {schedule grabInstruments {-handlers 900 -reconnect} -600} 2006-01-29 00:24:49Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server 2006-01-29 00:34:54Z sdsshost2 IOP 8408 TEXTONLY trace 1, 0 jcb original: {schedule grabInstruments {-handlers 900 -reconnect} -600} 2006-01-29 00:34:54Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server 2006-01-29 00:45:04Z sdsshost2 IOP 8408 TEXTONLY trace 1, 0 jcb original: {schedule grabInstruments {-handlers 900 -reconnect} -600} 2006-01-29 00:45:04Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server ^^ long running jcb grab on pid 8408 -- the "schedule" lines are "keepalive"s 2006-01-29 00:55:04Z sdsshost2 IDA 14661 TEXTONLY trace 1, 0 jcb original: {grabInst -drop} 2006-01-29 00:55:04Z sdsshost2 IDA 14661 TEXTONLY Stopping grabInstruments ^^ jcb pid 14661 drops, but it's not pid 8408.... 2006-01-29 00:55:06Z sdsshost2 IDA 28156 TEXTONLY trace 1, 0 harvanek original: {tkButtonDown .setupgui.grabinst.button} ^^ harvanek tries a grab 2006-01-29 00:55:17Z sdsshost2 IOP 8408 TEXTONLY trace 1, 0 jcb original: {schedule grabInstruments {-handlers 900 -reconnect} -600} 2006-01-29 00:55:17Z sdsshost2 IOP 8408 TEXTONLY IOP running as Backup Relay Server ^^ the pid 8408 grab is still staying alive... 2006-01-29 00:55:18Z sdsshost2 IDA 28156 TEXTONLY Starting grabInstruments 2006-01-29 00:55:30Z sdsshost2 IDA 28156 TEXTONLY Stopping grabInstruments 2006-01-29 00:55:34Z sdsshost2 IDA 28156 TEXTONLY Couldn't open udp socket 30410 : Address already in use ^^ harvanek's grab hits the wall. APO Weather Page: ---------------- At about 21:36 MST, the APO weather page stopped updating data. A process died, but I don't know what process to restart. It also caused the VNC connection with the DIMM to fail uncleanly. Flat field lamps: ---------------- At approx. 11:52Z Two of the flat field lamps failed to come on during a pre-cal. We called French. He will investigate. Maybe a bad relay? The lamps that failed to come on were 1 and 3 as seen in the Watcher system status cartoon. apo_plotbias results: -------------------- Filename 02% 05% 10% 50% 90% 95% 98% ------------------- ------ ------ ------ ------ ------ ------ ------ sdR-b1-00036869.fit -5.5 -3.8 -2.2 2.2 6.7 8.8 9.9 sdR-b1-00036869.fit -5.5 -3.8 -2.2 2.2 6.7 8.8 9.9 sdR-b2-00036869.fit -8.2 -5.7 -4.5 1.7 7.0 9.0 10.7 sdR-b2-00036869.fit -8.2 -5.7 -4.5 1.7 7.0 9.0 10.7 sdR-r1-00036869.fit -6.9 -4.7 -3.7 1.8 7.2 8.4 10.5 sdR-r1-00036869.fit -6.9 -4.7 -3.7 1.8 7.2 8.4 10.5 sdR-r2-00036869.fit -8.4 -6.3 -4.4 1.1 6.3 8.4 10.5 sdR-r2-00036869.fit -8.4 -6.3 -4.4 1.1 6.3 8.4 10.5