APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday March 26, 2003 (MJD 52725) ---=== OBSERVING TEAM ===--- Swing: Atsuko Kleinman Night: Dan Long, Pete Newman Support: Craig Loomis, Jon Brinkmann, French Leger (FNAL) ---=== OBSERVING PLAN ===--- Standard survey science. ---=== OBSERVING SUMMARY ===--- Six plates done (994, 1042, 1045, 1215, 1231, 1286) under variable cloud and mediocre seeing. Possible progress towards solving the PTVME problems. ---=== OBSERVING LOG ===--- Afternoon: ---------- In the light of last night's problems, we rebooted the DAs was well as the HOST today. See also problem section #1 for further news on a possible identification of one underlying problem. Since I have not heard anything regarding Omaha's failure last night, I will consider it as non-existent until I hear that it is up and happy. Spectrographs in focus. Night: ------- Mixed cloud at sunset, so we start with spectroscopy. During set-up for the first plate, Dan's goStare's failed. We found that 'camera nag' commands in the murmur log were returning "bad file number" errors. We did grabInst -drop, then grabInst again, and all was well. During plate 1286, around 08:06, Dan's SOP session (PID 13094) froze then spewed quite spectacularly. See problem section #2 for details. We found guide stars on the gotoField for all plates unless stated otherwise. endNight is running as this log is being mailed. Observing sequence: 01:40Z Plate 1215, cartridge 1. Done, 3 exposures, 2700 s. Found the field long before 12deg twilight. Wind speed made us consider closing, but seemed to be dropping toward the end of the exposures. 01:51Z Plate 994, cartridge 2. Done, 3 exposures, 3900 s, plus 2 exposures lost to clouds. Encroaching clouds made finding guide stars or an FK5 difficult, but we got there. Variable clouds continued. SoS flagged the smear as having insufficient signal, but we did not mark it bad. Seeing quite poor. 06:28Z Plate 1231, cartridge 3. Done, 3 exposures, 2900 s. Variable clouds and mediocre seeing continued, though clouds cleared in final exposure. 08:10Z Plate 1286, cartridge 9. Done, 3 exposures, 2320 s. FK5 star found ~8 arcsec off pointing. More clouds arrived during the first exposure, but still a tremendous data rate. 09:34Z Plate 1042, cartridge 7. Done, 3 exposures, 2160 s. FK5 star found ~8 arcsec off pointing, but in almost opposite direction to the last plate on the same area of sky. 10:49Z Plate 1045, cartridge 5. Done, 3 exposures, 2420 s. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3820 00:03Z 00:59Z 100 O -106.01 -91.94 102 ignore doghouse bias ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): Yes QA Procedures Done (y/n): Science frames, no issues. UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 20037, plate -9999 ------- 00:00 20037 0.0 bias ----- sequence 20038, plate 1231 ------- 00:07 20038 10.0 flat Focus checks - ignore 00:09 20039 2.0 arc 00:12 20040 2.1 arc 00:14 20041 2.1 arc ----- sequence 20043, plate 1215 ------- 25.8 17.9 30.8 21.3 DONE 02:31 20043 10.0 flat 02:34 20044 2.0 arc 02:56 20045 900.1 target 03:15 20046 900.1 target 03:33 20047 900.1 target 03:40 20048 240.0 smear 03:43 20049 10.0 flat 03:45 20050 2.0 arc ----- sequence 20051, plate 994 ------- 19.1 15.5 22.3 15.5 DONE 04:11 20051 10.0 flat 04:13 20052 2.0 arc 04:38 20053 1200.1 target Bad, ignore 05:00 20054 1200.1 target 05:23 20055 1200.1 target 05:46 20056 1200.1 target Bad, ignore 06:14 20057 1500.1 target 06:23 20058 240.0 smear May be bad ----- sequence 20059, plate 1231 ------- 21.2 18.6 19.9 23.2 DONE 06:42 20059 10.0 flat 06:45 20060 2.0 arc 07:18 20061 1200.1 target 07:36 20062 900.1 target 07:52 20063 800.1 target 08:00 20064 240.0 smear 08:03 20065 10.0 flat 08:05 20066 2.0 arc ----- sequence 20067, plate 1286 ------- 22.0 22.3 21.0 25.0 DONE 08:31 20067 10.0 flat 08:34 20068 2.0 arc 08:50 20069 800.1 target 09:06 20070 800.1 target 09:21 20071 720.1 target 09:29 20072 240.0 smear ----- sequence 20073, plate 1042 ------- 16.8 16.5 16.6 17.6 DONE 09:49 20073 10.0 flat 09:51 20074 2.0 arc 10:06 20075 720.1 target 10:21 20076 720.1 target 10:36 20077 720.1 target 10:43 20078 240.0 smear ----- sequence 20079, plate 1045 ------- 16.2 15.7 16.9 14.8 DONE 10:54 20079 10.0 flat 10:57 20080 2.0 arc 11:12 20081 720.1 target 11:29 20082 800.1 target 11:47 20083 900.1 target 11:54 20084 240.0 smear ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ Cartridge 1 not recorded. 04:09Z 2 994 15.31 0.0044 62.41 0.0028 192.84 0.0000 1.000050 06:40Z 3 1231 18.92 0.0078 66.09 0.0024 196.00 0.0100 1.000260 08:29Z 9 1286 162.23 -0.0028 75.26 0.0026 -21.98 -0.0021 1.000230 09:59Z 7 1042 212.29 0.0031 63.24 0.0033 48.89 0.0000 0.999920 11:02Z 5 1045 220.64 0.0020 61.48 0.0020 61.02 -0.0148 0.999940 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL6190 Stays: JL6191 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ Cartridge 1 not recorded. 04:09Z 2 994 1.00005 -493 538 -200 15 62.4 8.0 23 268 BG38 1.8 06:40Z 3 1231 1.00026 -2566 -1207 -220 19 66.1 7.8 16 240 BG38 2.0 08:29Z 9 1286 1.00023 -2270 -837 -80 162 75.3 6.2 20 252 BG38 1.7 10:26Z 7 1042 0.99992 789 1671 -135 217 60.0 5.9 18 262 BG38 1.9 11:01Z 5 1045 0.99994 591 1504 -135 220 61.6 5.4 26 270 BG38 1.8 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:42Z 49 21 13 218 (SW) 563 - 00:14Z 49 21 17 256 (WSW) 574 - 01:04Z 46 18 21 266 (W) 484 - 02:12Z 45 16 24 241 (WSW) 454 - 02:43Z 45 13 26 255 (WSW) 505 - 03:13Z 46 10 28 264 (W) 533 - 03:46Z 46 9 29 266 (W) 442 - 04:17Z 46 9 19 276 (W) 508 - 04:49Z 46 9 18 266 (W) 676 - 05:21Z 45 9 14 231 (SW) 481 - 05:52Z 47 8 22 269 (W) 387 - 06:24Z 45 8 16 237 (WSW) 468 - 06:54Z 45 9 17 236 (SW) 452 - 07:26Z 45 11 15 225 (SW) 504 - 07:59Z 44 10 19 254 (WSW) 438 - 08:29Z 43 11 20 252 (WSW) 456 - 08:59Z 43 10 25 261 (W) 402 - 09:30Z 43 7 24 260 (W) 351 - 10:00Z 42 8 13 250 (WSW) 301 - 10:30Z 42 10 16 250 (WSW) 320 - 11:01Z 41 12 25 270 (W) 276 - 11:32Z 41 12 22 243 (WSW) 284 - 12:05Z 41 12 21 261 (W) 285 - ---=== TELESCOPE STATUS ===--- 23:45Z Doors open, fans on 06:15Z Enclosure off 12:10Z Enclosure on, shutdown. Status at 12:12Z: Telescope stowed at: 30 deg Instrument mounted: Cartridge 6 Counterweights at: 280 Autofill systems: On 180L LN2 dewar scales: Spectro 97 lb Imager 110 lb ---=== SOFTWARE USED ===--- IOP/SOP: v3_113_0 Watcher: v2_22_0 MCP: v5_18_0 TPM: tpm_v2_28_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_67 SoS: v4_9_13 hoggPT: v1_6_7 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0910 0.0790 Axial B 0.7970 0.7970 Axial C 0.8120 0.8240 Trans D -9.0660 -9.1130 Lateral E 1.8690 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5377. -3699. 923. -208. 31535. 30647. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 9.8 Alt.: 30.000304 MIGS TONIGHT NOMINAL Axial A 1.5160 1.5170 Axial B 1.1580 1.1560 Axial C 1.1000 1.0990 Trans D -0.6950 -0.7070 GALILS Commanded: 1619699. 1555114. 1576373. -3400. -7900. Actual: 1619636. 1555273. 1576326. -3383. -7925. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1256.97 -0.03 -20.04 -0.51 130.90 ---=== PROBLEMS IN DETAIL ===--- Problem #1 - PTVME errors ------------------------- We may have gotten closer to a positive identification of the cause of the PTVME errors that have been plagueing us recently. During afternoon checkout, we were running an imager bias, and everything was working as it should. During that run, Craig and Pete were discussing possible causes of the errors seen last night, and in the process of teaching Pete how the DA system is connected, we "jiggled" the connectors on the PTVME interconnect cable that runs between sdsshost and all the crates. A few minutes later we learned that Dan and Atsuko in the control room were recovering from a ptvme error in IOP. With Jon B and French, we then found that one part of the cable near one of the crate 1 connector looked like it had been crushed and abraded at some time - this was one of the connectors that Craig and Pete had jiggled immediately before IOP reported the ptvme error. We now believe a similar event about two months ago was a manifestation of the same thing: moving the cables produced a ptvme error, but at that time, we were less certain. I discussed the status of PR 2913 with Craig, and found two of the three changes suggested by Ron Rechenmacher have been done, namely to change the PTVME card in sdsshost, and to change to appropriate termination card at the end of the interconnection cable (in the spectro crate). That leaves the cable itself from Ron's suggested problem sources. At that time (2001-Dec-4) the cable was inspected but no problem was found, whereas now we think we may have found a near-failure in the cable. We have seen increasing failure rate on all DA nodes recently, and if we get to the point where we lose the DA or communications between the DA and sdsshost, then observing stops. We are therefore raising the priority of PR2913 from serios to critical, and request at least (a) a spare crate-to-sdsshost cable be available at APO and (b) that further tests be conducted this coming bright time to reproduce the problem and, we expect, see if relacing the cable solves it. French has agreed to chase action on this problem with FNAL, so we are assigning the PR to him. To reproduce the problem, we suggest starting a bias drift on the imager (or better, starting a science drift with real data flowing), then methodically move the cable to produce a ptvme fault on sdsshost. p.s. We have notes on recovery from the PTVME errors is wanted. Basically had to reboot crate 1. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Problem #2 - tccPutAndWait went berserk --------------------------------------- At ~08:06Z there were an insane number of tccPutAndWait messages (~290 per second!) scrolling by in the murmur log after instChange command after plate 1231 issued via SOPGUI. After a few minutes the insanity stopped and a traceback window popped up with the following traceback: timed out waiting for the TCC to complete command track 121,90 mount /rotangle=0 /rottype=mount while executing "error $errorMessage" invoked from within "if {[getclock] > $startTime + $timeout} { set errorMessage "timed out waiting for the TCC to complete command $tccCommand" murmur "tccPutAn ..." ("while" body line 5) invoked from within "while {!$TCCCommandComplete($commandIndex)} { murmur "tccPutAndWait trying listenerToTCC" listenerToTCC murmur "tccPutAndWait finished tryin ..." (procedure "tccPutAndWait" line 17) invoked from within "tccPutAndWait "track 121,90 mount /rotangle=$rot_dest /rottype=mount"" (procedure "instChange" line 105) invoked from within "instChange -noInit" ("eval" body line 1) invoked from within "eval "instChange $option"" (procedure "GUIinstChange" line 42) invoked from within "GUIinstChange $GUIinstChangeOption" invoked from within ".sgui.instchange.go invoke" ("uplevel" body line 1) invoked from within "uplevel #0 [list $w invoke]" invoked from within "if {($w == $tkPriv(window)) && ([$w cget -state] != "disabled")} { uplevel #0 [list $w invoke] }" invoked from within "if {$w == $tkPriv(buttonWindow)} { set tkPriv(buttonWindow) "" $w config -relief $tkPriv(relief) if {($w == $tkPriv(window)) && ([$w cget -state] ..." (procedure "tkButtonUp" line 3) invoked from within "tkButtonUp .sgui.instchange.go" (command bound to event) The messages in murmur log were continuos repeats of: 2003-03-27 08:06:55Z sdsshost IOP 13094 TEXTONLY tccPutAndWait trying listene rToTCC 2003-03-27 08:06:55Z sdsshost IOP 13094 TEXTONLY tccPutAndWait finished tryin g listenerToTCC We note that Dan runs SOP and SOPGUI under eXodus, if this is related to TCL button positions, as the traceback would suggest. Some 40 minutes later, Pete's logViewer is STILL trying to catch up with the murmur log!