APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday February 7, 2005 (MJD 53409) ---=== OBSERVING TEAM ===--- Swing: Atsuko Nitta Night: Pete Newman, John Barentine 24/7: Craig Loomis ---=== OBSERVING PLAN ===--- Science. ---=== OBSERVING SUMMARY ===--- Silence: icy weather continues. However, we seem to have developed a critcal DA problem that would stop us taking imaging data. ---=== OBSERVING LOG ===--- Afternoon: When I came in, I saw many messages on "waiting for response from ad4" messages. So I rebooted the astro crate. The bove message is a sign, usually, of ptvme unhappiness. But then, I saw many {ad1 {ICC ad1 (crate 3, board 0) is not available PT Link open failure}} for all ad crates (ad1-ad4). It looks like this message was seen yesterday as well (example murmur message): 2005-02-07 03:10:07Z sdsshost IOP 580640 TEXTONLY {ad2 {ICC ad2 (crate 3, board 1) is not available Bad ICC}} but maybe this is when host or the crates were rebooted. Anyway, reboting the astro crate did not help and Craig watched the messages as the crate rebooted but all were well. We decided to reboot host and then the message disappeared. Maybe I rebooted the crates before the server was quite up and confused the world. Since the enclosure temp is 5.2C with outside temp -3C, I did not check the collimation. THe postCalib shows everything is ok. It is possible that once open and spectros are in equillibrium, we need to tweak the collimation. Everything else checked out fine. Night: Conditions on and above the ground prevented operations. Still lots of snow and ice on the enclosure and on the working floor level outside the west enclosure door. Humidity was above 90% most of the night. However, the DA system kept us a little interested: we were seeing Watcher errors from the archiver all night, saying (e.g.): id2's archiver and tape drives no gang transfer (63: 2005-02-08 09:06:26Z) Most of the time, ptvmeTests returned no errors. We did find that the teamster had died, but on restarting it, the errors continued. We also restarted the servers thinking perhaps something was out of sync, and later tried a poolInit - no luck. Eventually, towards the end of the night (around 10:30Z, sorry exact time not noted), we tried a goDrift, and got more specific errors: im> goDrift -bias Waiting for command response from ad2 (crate 3, ICC 1), data server (client 1) Waiting for command response from ad2 (crate 3, ICC 1), data server (client 1) Waiting for command response from ad2 (crate 3, ICC 1), data server (client 1) Waiting for command response from ad2 (crate 3, ICC 1), data server (client 1) Waiting for command response from ad2 (crate 3, ICC 1), data server (client 1) ... Waiting for command response from ad2 (crate 3, ICC 1), data server (client 1) You should probably consider the system messed up quitting if breakCheck called..., or tell me three times (1) Error: Interrupted function call (We hit ctrl-C to break out of that loop.) A ptvmeTest on each of the photo and astro creates then showed the same error. Here's the astro crate: prn@sdsshost:~/ > ptvmeTest -n 3 -a 0x1800000 PID: 132002 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 3 --> /dev/ptvme0n3 remAddr: 0x01800000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n3", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 0 2066 ms 04:12 errno = 5: read (fd=3, 0x7F7F2D00, 0x00000004) = -1 (I/O error) CSB status: 0x80430010 AUX = (Timeout - LAN idle > 150 msec - target inited? or VMEbus error) CMP = (Command Completed With Error Status) INNA = 0 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> Not that the error from goDrift reported the error on ad2 this time, though we think we were seeing ad4 last night. We did NOT issue a ptvmeReset, thinking that leaving the system in this broken state might help debugging. We therefore also did not run endNight, but (apart from log housekeeping) there were no data to write to tape. PR 6350 opened at critical severity, but see also PR 5543 (2003-8-6) for a previous event. As noted above, we will NOT run endNight. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 5145 23:10Z 00:14Z 100 O -164.98 -148.93 115 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 30813, plate -9999 ------- 23:12 30813 0.0 bias ----- sequence 30814, plate 2061 ------- 23:43 30814 10.0 flat 23:46 30815 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- No data written to tape. ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:57Z 25 25 13 209 (SSW) -9999 - Variable cloud 23:28Z 25 25 13 209 (SSW) -9999 - " 23:58Z 25 25 13 209 (SSW) -9999 - " 00:30Z 26 26 14 265 (W) -9999 - " 01:02Z 26 24 11 276 (W) -9999 - " 04:24Z 27 26 11 270 (W) -9999 - " 04:56Z 27 26 9 263 (W) -9999 - " 05:28Z 28 26 14 274 (W) -9999 - " 06:01Z 28 26 11 265 (W) -9999 - " 06:33Z 28 24 12 244 (WSW) -9999 - " 07:05Z 27 24 10 277 (W) -9999 - " 07:35Z 27 23 13 254 (WSW) -9999 - " 08:07Z 27 25 14 233 (SW) -9999 - " 08:39Z 27 25 14 236 (SW) -9999 - " 09:12Z 27 25 17 234 (SW) -9999 - " 09:44Z 26 25 17 238 (WSW) -9999 - " 10:16Z 26 25 15 240 (WSW) -9999 - " 10:48Z 27 24 13 236 (SW) -9999 - " 11:21Z 26 25 12 267 (W) -9999 - " 11:53Z 26 24 18 233 (SW) -9999 - ---=== TELESCOPE STATUS ===--- Enclosure not opened or ventillated. No interlocks bypassed. Status at 11:35Z: Telescope stowed at: 30 deg altitude Instrument mounted: Cartridge 4 Counterweights at: 160 Autofill systems: On 180L LN2 dewar scales: Spectro 195 lb, 22 psi Imager 167 lb, 20 psi The spectro dewar pressure has been steadily climing all niht, from about 18 psi at the start to 22 now. This is abnormal. ---=== SOFTWARE USED ===--- IOP/SOP: v3_143_0a Watcher: v2_30_0 MCP: v5_26_0 TPM: tpm_v2_47_0 AstroDa: v14_47 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v1_97 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5790 0.5790 Axial B 0.5860 0.5880 Axial C 0.9460 0.9460 Trans D -8.1970 -8.1950 Lateral E 10.0457 10.0076 Lateral F 11.6078 11.5824 GALILS Commanded: 4300. -1900. 2850. -13300. -7600. -7650. Actual: 4299. -1917. 2868. -13303. -7614. -7660. SETMIR VALUES PriDesOrient: 0.00 4.57 1.18 64.25 -249.76 PriOrient: 0.00 4.56 0.88 65.00 -249.58 SECONDARY: ---------- Focus: 0.00 Air Temp.: -3.6 Alt.: 30.394673 MIGS TONIGHT NOMINAL Axial A 1.5710 1.5710 Axial B 1.0480 1.0490 Axial C 1.1610 1.1610 Trans D -0.2040 -0.2070 GALILS Commanded: 1619183. 1605797. 1528682. -13800. -9950. Actual: 1619188. 1604973. 1528571. -13820. -9964. SETMIR VALUES SecDesOrient: 1257.00 27.97 -10.00 -90.00 257.89 SecOrient: 1257.25 28.12 -9.77 -89.79 257.40 ---=== PROBLEMS IN DETAIL ===---