APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday March 9, 2003 (MJD 52708) ---=== OBSERVING TEAM ===--- Night: Steph Snedden, Pete Newman Visitors: Larry Carey (U.W), Tom Murphy (U.W, a.k.a. Dr. Laser) Also not featuring: Snowplow (failed): Jurek "My Kia Goes Anywhere" Krzesinski (U.AWOL) Towing: Mike "My Jeep REALLY Goes Anywhere" Harvanek, John "The Shovel" Barrentine (APO) ---=== OBSERVING PLAN ===--- Science. ---=== OBSERVING SUMMARY ===--- Four plates done (938, 1006, 1187, 1263) before moonset, then imaging, Strip 33 S, -8 <= lambda <= +25 (but see caveats below) While imaging that strip, we had a more serious repeat of the imager teamster death and very slow rebirth again, as on MJD 52705. This time we got the debugging info requested folklore, but we need someone (who?) to interpret the output and diagnose the problem. See problems section for the gory details. Unfortunately, while working on recovery of the teamster problem, we had no seeing information. After we recovered and ran seeingRun, we learned that it was flagging a 40-frame section as bad seeing, with a median as large as 1.9 arcsec. Also, it did not process frames beyond frame 187 (the run went to run 270+). Steve: we need your judgement as to whether this is unacceptable. On the up side, this strip crossed galaxy M101 starting at frame 187 in chip r3. That was a definite "Wow!" We also have another leak in the grade2 N2 imager dewar pressurization system. The N2 bottle is empty and disconnected. endNight completed sucessfully (phew!). ---=== OBSERVING LOG ===--- No problems during afternoon check-out, but spectro focus not checked. We ran the enclosure ThermoKing refrigeration unit for a few hours before opening to pre-chill everything. At sunset, cirrus below 10deg alt to the S, otherwise clear. We start with spectroscopy until moonset at least. SoS reported requirements for postCalibs on several plates tonight, more than we usually see. All plates were observed at high altitude (~60deg or higher). Most of the flexure warnings came from spectrograph sp2 (both cameras) with one for r1. At 09:09Z, the imager teamster died and again took hours to get restarted. See problem section #1 for details. Observing sequence: Guide stars seen after gotoField unless noted. 01:37Z Plate 1263, cartridge 1. Done in 3000 sec plus smear and postCalibs. FK5 star used to center up while sky too bright for guide stars. Seeing bounced a lot with the N wind. 03:51Z Plate 938, cartridge 2. Done in 2186 sec plus smear and postCalibs. Seeing more settled. In fact, in the smeared guider images, the seeing is so good you can see the wiggle as the exposure is smeared! 05:06Z Plate 1187, cartridge 3. Done in 2400 sec plus smear. As the mounting latches closed, a clatter was heard. The east-most black cartridge cover is somewhat loose. We will ask the plugging crew to check tomorrow. Seeing remains excellent. 06:20Z Plate 1006, cartridge 4. Done in 2700 sec plus smear and postcalibs. FK5 required to center. Wind now from SE and measured seeing more variable as we are looking downwind. 07:45Z Moon has set, seeing still good, so we switch to the imager. See imaging sections below. 11:17Z After ending imaging run 3712, we could not get a goDrift to start before twilight. Time logged to DA problem. After much messing about with tapes (see problem section), endNight completed successfully. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3709 23:26Z 23:43Z 100 O -131.92 -127.80 40 ignore Doghouse bias 3710 08:02Z 08:17Z 100 O -2.56 1.00 38 bias T-bars unlatched 3711 08:23Z 08:33Z 33 S -13.70 -11.04 31 ignore 3712 08:35Z 11:17Z 33 S -12.72 27.97 284 science ---=== IMAGING RUN DETAILS ===--- Run 3910: Bias, on telescope, T-bars unlatched ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Run 3711: IGNORE We aimed for 33 S, but we think we cleared offsets after the scanStripe command, and hence cleared the boresight offset. tccMon cleared showed no boresight offset and the strip as 33O instead of 33S, and skippy was lost. However, the focus loop did converge during this time. endDrift and try again. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Run 3712: Strip 33 S, -8 <= lambda <= 25 NB: Gangs transfers stopped at frame 62! We therefore have no definite seeing measurements beyond that until long after the drift was ended. Some of the frames in the interim had bad seeing. Frame Lambda Notes ----- ------ ---------------------------------------------------- 9 -12.4 Focus converged, seeing in spec. 10 -12.3 Skippy happy. nuErr 7.5", but is it increasing. 18 -11.1 Very bright star in dewar 5 r chip. 24 -10.2 NB: nuErr = 8.0", but did not get above 9.0". 25 -10.0 Start ramp up. 56 -5.4 Seeing as acceptable. We are looking downwind. 70 Seeing bad? 109 Seeing good again. 187(r) +14.4 M101 in dewar 3!! 255(r) +24.4 Meteor in dewar 5 r chip 259 +25.0 End of science, start ramp down. 272 +27.0 End of ramp. ---=== SKIPPY RESULTS ===--- ** |nuErr| >= 8.0 Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 3712 10 4 59 0.300 0.304 7.500 0.452 -0.00783 327 60 3712 24 4 65 -0.300 0.758 8.000 0.459 -0.00706 329 60 ** 3712 51 4 63 -1.200 0.183 8.400 0.146 -0.00708 333 61 ** 3712 92 4 68 -1.600 0.247 8.900 0.187 -0.00755 338 62 ** 3712 159 4 65 -2.200 0.689 9.000 0.311 -0.00712 346 66 ** 3712 257 4 45 -3.800 0.198 8.700 0.101 -0.00640 355 73 ** ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 3712 35 1 61 208 6 -0.189 0.134 2.804 0.355 -0.00943 3712 35 1 61 208 15 0.304 0.265 3.144 0.398 -0.01057 3712 35 1 61 208 15 1.024 0.167 2.997 0.263 -0.01008 3712 35 1 61 208 18 1.554 0.182 3.139 0.243 -0.01055 3712 35 1 61 208 9 1.932 0.257 3.238 0.289 -0.01089 3712 51 1 61 206 13 -0.241 0.246 3.066 0.341 -0.01031 3712 51 1 61 206 13 0.338 0.173 2.861 0.206 -0.00962 3712 51 1 61 206 13 1.017 0.139 2.910 0.180 -0.00978 3712 51 1 61 206 12 1.716 0.144 2.855 0.149 -0.00960 3712 51 1 61 206 18 2.041 0.187 2.748 0.297 -0.00924 3712 266 1 74 185 16 -0.621 0.203 1.806 0.169 -0.00607 3712 266 1 74 185 22 0.047 0.246 1.665 0.218 -0.00560 3712 266 1 74 185 13 0.468 0.304 1.742 0.273 -0.00586 3712 266 1 74 185 17 0.894 0.181 1.851 0.280 -0.00622 3712 266 1 74 185 16 1.317 0.169 1.901 0.144 -0.00639 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): Yes QA Procedures Done (y/n): QA top 10 on some QSOs only. UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 19797, plate -9999 ------- 23:10Z 19797 0.0 bias ----- sequence 19798, plate 1263 ------- 14.8 14.7 16.9 24.2 DONE 02:33 19798 10.0 flat 02:36 19799 2.0 arc 02:54 19800 900.1 target 03:17 19801 1200.1 target 03:36 19802 900.1 target 03:43 19803 240.0 smear 03:46 19804 10.0 flat 03:48 19805 2.0 arc ----- sequence 19806, plate 938 ------- 16.6 17.9 17.6 15.7 DONE 04:00 19806 10.0 flat 04:02 19807 2.0 arc 04:21 19808 900.1 target 04:37 19809 685.8 target 04:50 19810 600.1 target 04:57 19811 240.0 smear 05:00 19812 10.0 flat 05:02 19813 2.0 arc ----- sequence 19814, plate 1187 ------- 15.4 19.9 16.5 20.2 DONE 05:15 19814 10.0 flat 05:18 19815 2.0 arc 05:33 19816 600.1 target 05:51 19817 900.1 target 06:09 19818 900.1 target 06:16 19819 240.0 smear ----- sequence 19820, plate 1006 ------- 16.9 18.7 19.0 22.9 DONE 06:32 19820 10.0 flat 06:35 19821 2.0 arc 06:55 19822 900.1 target 07:13 19823 900.1 target 07:31 19824 900.1 target 07:39 19825 240.0 smear 07:42 19826 10.0 flat 07:45 19827 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 02:41Z 1 1263 51.15 0.0053 71.78 0.0004 224.29 0.0084 1.000000 04:17Z 2 938 126.72 0.0057 76.61 0.0009 -61.65 0.0103 1.000100 05:22Z 3 1187 -25.57 0.0061 60.65 0.0026 158.56 0.0087 1.000210 06:48Z 4 1006 206.81 -0.0025 70.41 0.0033 35.77 0.0000 1.000120 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL6049-54+6065 Stays: JL6056-61+6066 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 02:40Z 1 1263 1.00000 0 929 -200 52 71.6 6.6 6 338 BG38 1.4 04:16Z 2 938 1.00010 -986 124 -180 127 76.6 7.0 2 286 BG38 1.4 05:23Z 3 1187 1.00021 -2073 -746 -180 -26 60.6 6.2 1 179 BG38 1.3 06:48Z 4 1006 1.00012 -1184 -50 -200 207 70.4 5.7 10 174 BG38 1.6 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:12Z 49 6 4 261 (W) 130 - Clear 23:45Z 50 14 7 297 (WNW) 116 - " 00:21Z 47 -1 4 304 (NW) 116 - " 00:53Z 45 -2 0 247 (WSW) 124 - " 01:25Z 42 -2 0 47 (NE) 133 - " 01:56Z 42 1 3 24 (NNE) 139 - " 02:27Z 40 3 6 7 (N) 163 - " 02:57Z 43 1 6 357 (NNW) 150 - " 03:30Z 43 1 4 8 (N) 153 - " 04:03Z 44 0 1 332 (NNW) 146 - " 04:35Z 44 1 1 214 (SW) 160 - " 05:07Z 43 0 4 182 (S) 130 - " 05:39Z 43 -1 2 150 (SSE) 114 - " 06:10Z 43 0 4 179 (S) 130 - " 06:44Z 42 -8 6 174 (S) 95 - " 07:14Z 40 -1 9 167 (SSE) 121 - " 07:45Z 41 4 9 178 (S) 173 - " 08:17Z 41 2 9 181 (S) 157 - " 08:50Z 41 1 8 181 (S) 155 - " 09:22Z 41 2 8 179 (S) 164 - " 09:52Z 41 0 6 188 (S) 141 - " 10:22Z 41 1 7 214 (SW) 151 - " 10:55Z 41 1 3 221 (SW) 126 - " 11:27Z 41 1 6 227 (SW) 130 - " 11:59Z 41 0 6 252 (WSW) 140 - " 12:32Z 41 0 6 253 (WSW) 112 - " 13:04Z 41 0 7 252 (WSW) 142 - " 13:37Z 39 2 10 238 (WSW) 133 - 14:09Z 41 2 8 236 (SW) 125 - ---=== TELESCOPE STATUS ===--- 22:00Z ThermoKing on, louvres cracked, cone fans on, hatch open. Enclosure air temp reached ~41F 00:30Z Doors open, extractor on 01:00Z Enclosure off (10 min before sunset) 12:20Z Enclosure on, fans off, ThermoKing off. Status at 13:25Z: Telescope stowed at: 30 deg Instrument mounted: Imager Counterweights at: 720 Autofill systems: On 180L LN2 dewar weights: Spectro 184 lb Imager 71 lb NB: Imager dewar grade-2 N2 pressurization bottle is empty and disconnected. It dropped ~600 psi tonight, so we clearly have a new leak somewhere. ---=== SOFTWARE USED ===--- IOP/SOP: v3_109_4 Watcher: v2_21_0 MCP: v5_18_0 TPM: tpm_v2_27_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_64 SoS: v4_9_13 hoggPT: v1_6_7 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0910 0.0790 Axial B 0.7930 0.7970 Axial C 0.8090 0.8240 Trans D -9.0660 -9.1130 Lateral E 1.8880 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5377. -3699. 923. -208. 31535. 30647. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 7.2 Alt.: 30.013588 MIGS TONIGHT NOMINAL Axial A 1.5170 1.5170 Axial B 1.1570 1.1560 Axial C 1.1010 1.0990 Trans D -0.6970 -0.7070 GALILS Commanded: 1619668. 1555340. 1576354. -3400. -7900. Actual: 1619636. 1555273. 1576326. -3383. -7925. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1257.00 0.00 -20.01 -0.48 130.88 ---=== PROBLEMS IN DETAIL ===--- Problem #1 - Imager teamster died --------------------------------- NB: This is important because while the teamster is dead, we have no measurement of seeing and we have no astrometric gangs to write to tape (and so endNight will probably fail. We saw from seeingRun that gangs were not being transfered before we noticed the error in watcher. murmur log showed: 2003-03-10 09:09:09Z sdsshost TMS 16129 TEXTONLY Gangs transfer: id1:1/1 id2:1/1 id3:1/1 id4:1/1 id5:1/1 id6:1/1 ad1:1/1 ad2:1/1 ad3:1/1 ad4:1/1 -> unix 2003-03-10 09:09:11Z sdsshost TMS 16129 dsc_E_shareLost Access to shared data area at 0x18000000 lost 2003-03-10 09:09:11Z sdsshost TMS 16129 dsc_E_writeFail Write failed to PT icc20 CSB=0x80430010 2003-03-10 09:09:11Z sdsshost TMS 16129 dsc_E_writeFail Write failed to rmt=0x1808CE7C cnt=0x00002288 This is much the same error as experienced a few nights ago (MJD 52705). It took a long time to restart. First, starting new IOP sessions takes many minutes in these circumstances (why?). Hence restarting the teamster, which spawns a new IOP session, also takes a long time. Second, once the teamster's getGangs command does start, we see repeated murmur messages (starting at 09:45Z): 2003-03-10 09:58:27Z sdsshost TMS 225953 TEXTONLY Waiting for command response from id4 (crate 2, ICC 0), data server (client 1) Folklore says this should clear itself in ~30 minutes. Tonight, it did not. Folklore also says to run a ptvmeTest command, but this in turn requires an IOP session! However, tonight we got them, and they are copied below. Note that crate 2 does NOT appear to show any errors (to our untrained eyes). During this incident, we also got a watcher error in the Micro/imager section: server is not responding (Last packet: 2003-03-10 10:09:28Z) which I seem to recall was also reported on MJD52705's incident. Here are the ptvmeTest commands (sucess lies beyond all this stuff!): im> ptvmeTest -n 1 -a 0x18000000 PID: 235097 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 1 --> /dev/ptvme0n1 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n1", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 110 ms 10:00 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 B0 2C 3C 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <..,<............> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> im> ptvmeTest -n 2 -a 0x18000000 PID: 236498 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 2 --> /dev/ptvme0n2 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n2", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 62 ms 10:02 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 B0 2C 50 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <..,P............> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> im> ptvmeTest -n 3 -a 0x18000000 PID: 236421 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 3 --> /dev/ptvme0n3 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n3", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 33 ms 10:02 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 AE FB C0 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> im> ptvmeTest -n 4 -a 0x18000000 PID: 236043 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 4 --> /dev/ptvme0n4 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n4", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 4 ms 10:03 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 B0 2F 2C 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <../,............> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> im> PR 4924 opened. We then tried to soft-reset the crates: prn:sdsshost:~/ > setup ptvme prn:sdsshost:~/ > ptvmeReset ptvmeMode: INFO: controller 0 reset (/dev/ptvme0n0) ptvmeMode: INFO: cur debug mode: 0 NORMAL ptvmeMode: INFO: cur debug: 0x00000000 ptvmeMode: INFO: cur timeout: 900 ticks ptvmeMode: INFO: def T/I xfer mode: 0x22 A32:D32 Program, Non-Privileged / A32:D32 Program, Non-Privileged ptvmeMode: INFO: op mode: 1 SINGLE_IO ptvmeMode: INFO: VMEbus priority: 0x03 ptvmeMode: INFO: VMEbus vector: 0x01 ptvmeMode: INFO: intern sem val: -1 ptvmeMode: INFO: user sem val: 1 ptvmeMode: INFO: PT-VME940 node: 0 prn:sdsshost:~/ > date Mon Mar 10 03:28:49 MST 2003 This made no difference. We also tried forcing the restart of another teamster and this also had no effect on the problem. Eventually, after the drift for run 3712 was ended, we tried hard-reseting (with the button) crate 2, after removing the tapes from its drives. This had the effect of killing the teamster again with a very similar error message to the first time: 2003-03-10 11:58:28Z sdsshost TMS 252729 dsc_E_shareLost Access to shared data area at 0x18000000 lost 2003-03-10 11:58:29Z sdsshost TMS 252729 dsc_E_readFail Read failed from PT icc20 CSB=0x80420010 2003-03-10 11:58:29Z sdsshost TMS 252729 dsc_E_readFail Read failed from rmt=0x1808ABDC cnt=0x00000004 Eventually, we removed the tapes from photo crate 2 (id3 to id6) and pressed the reset button. When the crate had gotten going, we started a teamster anew and...gangs started transfering right away! We are concerned we are seeing the death throes of crate 2. All is not completely well: seeingRun stops after frame 187 instead of frame 250+. And to be able to run endNight, we had to eject ALL the tapes, then remount them, then run "initNight -tapeLoad", then run endNight.