APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday May 7, 2002 (MJD 52402) ---=== OBSERVING TEAM ===--- Dan Long (swing) Jurek Krzesinski Atsuko Nitta via phone: Connie Rockosi Eric Neilsen ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Good night of imaging. 32S 15 --> 57 done 32N 10 --> 46.5 done We had ptvme problem during a science drift. We recovered from it quickly, but then we could not get the teamster to do its job. We woke up Eric for help who fixed it. When we tried to do endDrift at the end of the night, the process hung. We recovered by killing the iop session the command was issued from and logging out the imager port. We are leaving endNight running and not wait until it finishes. runlog-3180.txt consists of no data. We tried making it manually by failed. ---=== OBSERVING LOG ===--- Dan tested iop v3_78_1 which has the fix for the critical error notification in case we run out of LN2 for spectros. Dan called Connie for testing this new iop. The test Connie first sent did not exactly do what it was suppose to do. Dan successfully tested the notification system after talking to Connie. We found out that cntrl-shift-p kills X on landru by accident. In order to recover, you have to kill fvwm2 owned by the user. We put the imager before sunset. The weather was so good that we thought it was worth trying imaging. I guess Sir Robert answered to our PR2678 on weather by sending us good weather! Thank you Robert. If I knew how effective it was to send weather PRs to Robert, I would have done it more often in the past. 2:46Z Run 3176 bias on the scope. 48 min after sunset -- so it should be a good bias. Run 3177 Stripe 32 S ----------------------- frame 27 Longitude 11.8 in focus Skippy showed that all was in spec. Frame 31 Longitude 12.5 Rampup begins. Frame 49 Longitude 15 Science begins. Frame 290 Longitude 51 Seeing became "acceptable" according to SoKent Frame 330 Longitude 57 Seeing became "bad" Frame 362 Longitude 62 Science ends. Frame 380 Longitude 64.7 Rampsdown ends/end of drift. Dust climbed slowly during this run. But since it was very dry (5% humidity), photometric sky and no evidence of ash from the forest fires or pollen, we kept taking data. Dan told us to watch out of surge of dust, but if it is a slow rise and relatively stable, then we can stay open even if it climbs above 3000 cnts/0.1cu.ft. BTW, Dan is in the process of updating more detail on the dust open/closure condition in the procedure. Run 3178 Stripe 32 N ---------------------- Started @ Lambda 7.5 Frame 19 Seeing is acceptable. Frame 20 Longitude 9.8 Seeing is bad. Frame 21 Longitude 10 Science begins. Frame 97 last frame. This run is NG. Gang transferred stopped during this run. We found out that there were no servers running and we could not start servers either. When we tried to end the drift, we found out that we had ptvme problem. See the problem section for more detail. Run 3179 Bias --------------- Making sure that all is well after we rebooted the crates. Run 3180 Strip 32 N --------------------- Started @ Lambda 7.5 In focus from the beginning of the run. We could not get teamster to start and ended up calling Eric for help. Please see problem section for detail. We finally got teamsters around Frame 78 of this run. Seeing run says the seeing was good from the beginning and skippy told us that we were in spec in offsets. We were very relieved since we were drifting without any knowledge of the seeing or rotator/pointing offsets. Frame 21 Longitude 10.0 Science begins 10:22Z moonrise according to the skycalc. 10:37 Frame 264 Longitude 46.5 twilight starts. 10:39 Frame 268 Longitude 47 end of science. Watcher still did not give us any sky warnings yet. Frame 268 last in focus frame Frame 273 focus=-736 (-300 outOfFocus) Frame 285 fpcis=-136 (+300 outOfFocus) The frames are saved in their usual place. 10:57Z We tried to do endDrift and only photo1, 2, 3 stopped. Photo 4, 5, 6 and the astrometrics did not stop drifting. We tried issuing this command from another iop session, but we got "ftelnet connection refused" message. Control-C didn't work (I know it is dangerous, but we had to try). We killed this iop session by kill command. Then we logged out the imager port which was still grabbed. Finally, we were able to talk to the imager again and issued another endDrift which worked! endNight is still running. We have sent email to JB and Dan that we will be going to bed before endNight rejoices. Which might not be the case since we see that runlog for the last run failed... runlog-3180.txt exists, but has nothing in it. I tried manually creating it, but failed by a simple "error" message. The command I tried is makeRunLog 52402 3180 1.5 /d3/home/ank bob.log filling out runlog for bob.log for run 3180. Error ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3176 02:41Z 03:02Z 100 O -24.66 -19.52 48 bias 3177 03:04Z 06:48Z 32 S 8.80 65.01 388 science 3178 06:53Z 07:40Z 32 N 7.92 19.71 92 science ptvmeProblem. ignore. 3179 07:56Z 08:01Z 100 O 54.30 55.55 22 ignore 3180 08:04Z 11:08Z 32 N 8.01 54.19 294 science ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 3177 13 4 55 -5.700 0.329 -4.600 0.763 -0.01476 39 58 3177 21 4 63 -5.600 0.183 -4.700 0.120 -0.01572 40 58 3177 56 4 53 -5.200 0.165 -4.900 0.123 -0.01625 44 56 3177 91 4 63 -5.000 0.196 -4.900 0.132 -0.01660 48 56 3177 125 4 48 -3.300 0.172 -4.800 0.119 -0.01565 53 55 3177 160 4 48 -5.000 0.195 -4.900 0.146 -0.01639 58 55 3177 194 4 66 -3.800 0.169 -4.900 0.106 -0.01570 64 55 3177 229 4 74 -3.800 0.191 -4.700 0.114 -0.01532 70 55 3177 264 4 80 -3.600 0.210 -4.800 0.136 -0.01487 76 55 3177 297 4 110 -4.200 0.220 -4.800 0.142 -0.01454 83 55 3177 332 4 129 -4.700 0.205 -5.200 0.399 -0.01321 90 55 3177 367 4 118 -5.100 0.193 -5.500 0.111 -0.01298 97 54 3178 9 4 48 5.200 0.100 -3.800 0.112 -0.01109 328 62 3178 18 4 49 5.800 0.103 -3.600 0.133 -0.01089 328 63 3178 41 4 50 8.000 0.124 -3.300 0.136 -0.01077 329 64 3180 14 4 52 6.800 0.098 -1.500 0.105 -0.01067 319 54 3180 55 4 48 7.600 0.081 -1.000 0.102 -0.01030 317 56 3180 104 4 50 8.600 0.408 0.400 0.656 -0.00921 313 59 3180 139 4 69 8.900 0.113 1.400 0.103 -0.01024 308 61 3180 174 4 37 8.500 0.093 2.500 0.066 -0.00959 302 62 3180 209 4 60 8.200 0.121 3.400 0.414 -0.01048 295 63 3180 243 4 83 7.500 0.111 4.500 0.599 -0.00874 288 63 ---=== LTMATCH RESULTS ===--- Run Field nFields nGood rowMean rowSig colMean colSig rot --------------------------------------------------------------- 3177 32 1 8 -0.135 0.464 3.334 0.353 -0.01121 3177 32 1 13 0.162 0.429 3.618 0.389 -0.01216 3177 32 1 5 0.965 0.384 3.569 0.209 -0.01200 3177 32 1 12 1.577 0.366 3.490 0.266 -0.01174 3177 32 1 8 1.845 0.308 3.772 0.304 -0.01268 3177 66 1 12 -0.841 0.295 4.473 0.241 -0.01504 3177 66 1 16 -0.409 0.256 4.530 0.180 -0.01523 3177 66 1 14 0.331 0.276 4.323 0.244 -0.01453 3177 66 1 12 0.682 0.272 4.280 0.222 -0.01439 3177 66 1 16 1.204 0.271 4.286 0.172 -0.01441 3177 101 1 10 -0.864 0.263 3.859 0.225 -0.01297 3177 101 1 14 -0.460 0.164 4.238 0.296 -0.01425 3177 101 1 19 0.067 0.245 4.047 0.243 -0.01361 3177 101 1 19 0.638 0.227 4.078 0.136 -0.01371 3177 101 1 13 0.990 0.199 4.205 0.155 -0.01414 3177 135 1 21 -0.524 0.120 3.814 0.283 -0.01282 3177 135 1 15 -0.043 0.164 3.726 0.222 -0.01253 3177 135 1 9 0.525 0.304 3.801 0.310 -0.01278 3177 135 1 11 1.034 0.346 3.774 0.249 -0.01269 3177 135 1 15 1.548 0.278 3.814 0.148 -0.01282 3177 170 1 17 -0.751 0.304 4.136 0.421 -0.01391 3177 170 1 10 -0.042 0.207 3.930 0.412 -0.01321 3177 170 1 20 0.411 0.354 4.151 0.404 -0.01396 3177 170 1 17 0.801 0.300 3.969 0.410 -0.01334 3177 170 1 18 1.193 0.288 3.977 0.481 -0.01337 3177 205 1 18 -1.114 0.371 3.819 0.220 -0.01284 3177 205 1 23 -0.533 0.339 3.868 0.282 -0.01300 3177 205 1 20 0.040 0.325 3.763 0.280 -0.01265 3177 205 1 16 0.757 0.267 3.730 0.188 -0.01254 3177 205 1 17 1.078 0.230 3.821 0.165 -0.01285 3177 240 1 20 -0.462 0.345 3.162 0.149 -0.01063 3177 240 1 25 0.261 0.356 3.122 0.133 -0.01050 3177 240 1 21 0.760 0.369 3.053 0.161 -0.01026 3177 240 1 17 1.364 0.202 3.105 0.163 -0.01044 3177 240 1 19 1.636 0.316 3.307 0.164 -0.01112 3177 275 1 26 -0.534 0.225 3.092 0.242 -0.01039 3177 275 1 26 -0.013 0.236 3.179 0.284 -0.01069 3177 275 1 33 0.361 0.353 3.206 0.366 -0.01078 3177 275 1 21 0.851 0.276 3.090 0.244 -0.01039 3177 275 1 19 1.321 0.141 3.234 0.288 -0.01087 3177 308 1 33 -0.398 0.557 2.830 0.291 -0.00951 3177 308 1 30 -0.104 0.651 2.843 0.293 -0.00956 3177 308 1 30 0.470 0.492 2.804 0.201 -0.00943 3177 308 1 31 1.074 0.436 2.818 0.189 -0.00948 3177 308 1 31 1.471 0.419 2.940 0.189 -0.00988 3177 343 1 32 -0.803 0.191 3.217 0.372 -0.01081 3177 343 1 36 -0.432 0.217 3.346 0.404 -0.01125 3177 343 1 30 -0.035 0.179 3.333 0.332 -0.01121 3177 343 1 29 0.599 0.282 3.334 0.163 -0.01121 3177 343 1 24 1.069 0.277 3.288 0.262 -0.01106 3178 28 1 11 1.434 0.219 3.493 0.388 -0.01175 3178 28 1 14 1.323 0.194 3.806 0.288 -0.01280 3178 28 1 10 1.598 0.176 3.795 0.339 -0.01276 3178 28 1 12 1.608 0.206 3.764 0.189 -0.01265 3178 28 1 9 1.732 0.288 3.736 0.182 -0.01256 3180 116 1 16 0.314 0.332 3.544 0.160 -0.01191 3180 116 1 22 0.456 0.304 3.645 0.123 -0.01226 3180 116 1 18 0.606 0.326 3.696 0.180 -0.01243 3180 116 1 13 0.686 0.124 3.621 0.087 -0.01218 3180 116 1 17 0.635 0.149 3.569 0.117 -0.01200 3180 149 1 11 0.398 0.277 3.285 0.214 -0.01104 3180 149 1 20 0.580 0.292 3.251 0.168 -0.01093 3180 149 1 14 0.755 0.322 3.351 0.145 -0.01127 3180 149 1 14 0.858 0.300 3.214 0.190 -0.01081 3180 149 1 10 0.696 0.269 3.442 0.137 -0.01157 3180 185 1 17 -0.115 0.152 3.787 0.164 -0.01273 3180 185 1 15 -0.053 0.163 3.838 0.156 -0.01290 3180 185 1 20 0.083 0.170 3.894 0.170 -0.01309 3180 185 1 17 0.184 0.153 3.884 0.135 -0.01306 3180 185 1 11 0.141 0.235 3.706 0.207 -0.01246 3180 220 1 21 -0.521 0.263 3.399 0.173 -0.01143 3180 220 1 30 -0.389 0.201 3.506 0.281 -0.01179 3180 220 1 31 -0.494 0.229 3.395 0.228 -0.01141 3180 220 1 22 -0.308 0.173 3.234 0.175 -0.01087 3180 220 1 29 -0.205 0.119 3.350 0.151 -0.01126 3180 254 1 25 -0.604 0.140 3.150 0.152 -0.01059 3180 254 1 32 -0.574 0.111 3.290 0.164 -0.01106 3180 254 1 28 -0.528 0.154 3.162 0.135 -0.01063 3180 254 1 27 -0.376 0.115 3.158 0.089 -0.01062 3180 254 1 27 -0.394 0.180 3.326 0.142 -0.01118 ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 00:03 14946 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 03:40Z imager 134.69 0.0007 56.20 0.0016 220.93 -0.0040 1.000000 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 03:40Z imager 1.00000 0 -592 -490 135 56.2 11.8 17 267 23 1.28 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:21Z 64 -6 26 214 (SW) 1696 - 22:57Z 65 -6 19 235 (SW) 1631 - 23:29Z 61 -7 18 264 (W) 1640 - 00:01Z 59 -9 19 266 (W) 1505 - 00:34Z 57 -11 18 264 (W) 1610 - 01:06Z 57 -11 12 255 (WSW) 1613 - 01:38Z 54 -14 16 259 (W) 1850 - 02:11Z 52 -16 17 230 (SW) 2042 - 02:43Z 54 -14 14 260 (W) 2079 - 03:15Z 53 -13 22 279 (W) 2357 - 03:47Z 53 -13 16 263 (W) 2425 - 04:19Z 53 -13 16 259 (W) 2523 - 04:52Z 52 -13 19 255 (WSW) 2674 - 05:24Z 51 -13 15 240 (WSW) 2606 - 05:56Z 50 -12 20 271 (W) 2805 - 06:28Z 50 -12 13 264 (W) 3276 - 07:01Z 50 -12 11 282 (WNW) 3694 - 07:33Z 49 -10 12 263 (W) 4095 - 08:05Z 48 -11 10 263 (W) 4157 - 08:38Z 48 -11 14 273 (W) 4276 - 09:10Z 48 -10 15 267 (W) 3815 - 09:42Z 47 -8 12 273 (W) 3608 - 10:14Z 47 -7 14 272 (W) 3617 - 10:47Z 47 -6 15 268 (W) 3355 - 11:19Z 46 -6 13 273 (W) 3433 - 11:52Z 46 -6 13 267 (W) 3308 - ---=== TELESCOPE STATUS ===--- Fans on:1:30Z Opened:2:45Z Closed and fans off:11:20Z Telescope at stow with imager. Spectro autofill connected. Slip detection bypassed. imager@103lb spectro@108lb ---=== SOFTWARE USED ===--- IOP/SOP: v3_78_1 Watcher: v2_17_0 MCP: v5_11_3 TPM: v2_8_0 AstroDa: v14_47 TCC: 2.6.2 sdssProcedures: 1_44 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0760 0.0760 Axial B 0.7970 0.7990 Axial C 0.0750 0.0810 Trans D -1.5250 -1.5180 Lateral E 1.8330 1.8240 Lateral F 1.4010 1.3920 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 17.0 Alt.: 30.000767 MIGS TONIGHT NOMINAL Axial A 1.5500 1.5500 Axial B 1.2210 1.2220 Axial C 1.2150 1.2140 Trans D -0.5810 -0.5770 GALILS Commanded: 1599687. 1565230. 1586421. -10400. -9750. Actual: 1599600. 1565250. 1586350. -10400. -9750. SETMIR VALUES SecDesOrient: 1257.00 -3.40 -10.00 0.00 133.82 SecOrient: 1257.03 -3.42 -10.02 -0.25 133.71 PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0760 0.0760 Axial B 0.7970 0.7990 Axial C 0.0750 0.0810 Trans D -1.5280 -1.5180 Lateral E 1.8360 1.8240 Lateral F 1.4020 1.3920 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 13.7 Alt.: 30.000969 MIGS TONIGHT NOMINAL Axial A 1.5490 1.5500 Axial B 1.2210 1.2220 Axial C 1.2140 1.2140 Trans D -0.5810 -0.5770 GALILS Commanded: 1599662. 1565255. 1586377. -10400. -9750. Actual: 1599600. 1565250. 1586350. -10400. -9750. SETMIR VALUES SecDesOrient: 1257.00 -3.40 -10.00 0.00 133.82 SecOrient: 1257.02 -3.40 -10.01 -0.25 133.70 ---=== PROBLEMS IN DETAIL ===--- Around 7:20Z, we noticed that we were no longer getting updates on seeing info. Shortly after that, we noticed that the servers were no longer running. We tried restarting the servers, but the windows popped up and then disappeared. We saw followings in the Murmurlog. 2002-05-08 07:24:37Z sdsshost TMS 9573 TEXTONLY getGangs failed: ICC sd1 (crate 4, board 0) is not available Bad ICC 2002-05-08 07:25:37Z sdsshost TMS 9573 TEXTONLY transfering frames to unix We then tried the ptvmeTest commands. There were non-zero errono. The output of the three ptvmeTest commands are as follows. ~ Host[1:28]ptvmeTest -n 3 -a 0x18000000 PID: 261989 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 3 --> /dev/ptvme0n3 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n3", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 5: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0xFFFFFFFF (I/O error) . 0 2 ms 01:28 errno = 16: read (fd=3, 0x7F7F2CF0, 0x00000004) = -1 (Resource busy) CSB status: 0x00000000 AUX = (unknown) CMP = (Never Used) INNA = 0 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ~ Host[1:28]ptvmeTest -n 2 -a 0x18000000 PID: 262539 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 2 --> /dev/ptvme0n2 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n2", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 5: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0xFFFFFFFF (I/O error) . 0 37 ms 01:28 errno = 16: read (fd=3, 0x7F7F2CF0, 0x00000004) = -1 (Resource busy) CSB status: 0x00000000 AUX = (unknown) CMP = (Never Used) INNA = 0 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ~ Host[1:28]ptvmeTest -n 1 -a 0x18000000 PID: 262159 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 1 --> /dev/ptvme0n1 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n1", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 5: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0xFFFFFFFF (I/O error) . 0 2 ms 01:28 errno = 16: read (fd=3, 0x7F7F2CF0, 0x00000004) = -1 (Resource busy) CSB status: 0x00000000 AUX = (unknown) CMP = (Never Used) INNA = 0 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> We followed Dan's procedure on how to recover the ptvme failure. Jurek tried to endDrift before issuing any ptvmeReset commands, but after he issued endDrift command and answered questions, his iop session died and he had the familiar ptvme failure message. The endDrift Jurek sent did not completely stop the drift, but we no longer saw stars. Maybe it closed the shutter, but didn't end the drifting?? The monitors didn't change color after ptvmeReset, we saw the drift continue. We were able to restart the servers after the ptvmeReset. We ended the drift sucessfully after another endDrift. But now we could not start a new drift, we got "waiting for ad1" messages. So we did another set of ptvmeTest commands which came back with all errno=0. We checked ad1 by issuing " iccExec ad4 iciShow" just in case and all was well. .inprog was set to zero as it should. We sent another ptvmeRest. The output of ptvmeReset are as follows: iop> ptvmeReset ptvmeMode: INFO: controller 0 reset (/dev/ptvme0n0) ptvmeMode: INFO: cur debug mode: 0 NORMAL ptvmeMode: INFO: cur debug: 0x00000000 ptvmeMode: INFO: cur timeout: 900 ticks ptvmeMode: INFO: def T/I xfer mode: 0x22 A32:D32 Program, Non-Privileged / A32:D32 Program, Non-Privileged ptvmeMode: INFO: op mode: 1 SINGLE_IO ptvmeMode: INFO: VMEbus priority: 0x03 ptvmeMode: INFO: VMEbus vector: 0x01 ptvmeMode: INFO: intern sem val: 1 ptvmeMode: INFO: user sem val: 1 ptvmeMode: INFO: PT-VME940 node: 0 We waited but the screens did not change color. We decided to do reboot the crates before wasting any more moonless photometric time. Now we were able to start servers and do goDrift. We did a bias drift before starting the science drift. Once we started the science drift, we noticed that gangs are still not getting transferred. We tried to force start a new one. Jurek issued the command, but then it hung.He tried it from another iop session. This time, it said "moving the old teamster to meadowland" but then "Failed to kill old teamster" message followed. We tried killing this job with "kill -9" both as Jurek and as ROOT, but the job won't die. At this time, we decided to wake up Eric who knows UNIX and IOP much better than us. Eric managed to get us out of trouble by kiilling the parent process of the teamster process. Thank you Eric. Eric told us that the 3rd column in the "ps -ef" shows the parent process ID number. So if we ever need to kill the parent process, that is how you find out about their IDs. We immidiately saw gangs getting transferred when we restarted the teamster. By this time, we were on Run 3180 Frame 77 and the gangs were being transferred from Run 3178. We had to wait for a while to find out if our current run is good or not. TCC watcher error messages: TText="Could not queue a write to user:message output dies": (2002-05-08 08:36:11Z) Text="Write to user failed:output terminates to user13": (2002-05-08 08:36:11