APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday March 25, 2003 (MJD 52724) ---=== OBSERVING TEAM ===--- Swing: Atsuko Nitta Night: Dan Long Pete Newman ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Amid a host of problems we got four hours of imaging on 28 S ( lambda -23 to 40.9 ramps not subtracted ), and three plates done ( 1041, 1195, 1234 ). Blank guider frames: ID1 archiver hang: Spectro ptvme failure: Due to archiver troubles endNight was delayed getting started so it had not finished by the time we left. ---=== OBSERVING LOG ===--- Afternoon: 22:54Z My first iop session core dumped before successfully starting. /p/astroda/v14_47/bin/dscRun[429]: 7032 Memory fault(coredump) While doing afternoon doghouse bias, we got "wrong number of pixel" error in watcher as well as rowsSkipped error. See PROBLEM section for sample error messages. Spectrographs are in focus. Night: Cartridge 7 1042 17.3 18.0 22.4 23.7 Done Guidestars on slew (went to FK5 since sky was still bright, but it was in the fiber). First guider frame after returning from the FK5 star was blank. They were centered in the next one. Could this be the same problem we occasionally see where the first guider exposure after preCals is blank? 02:28Z First science exposure started Curiously SoS asked for postCals starting with the first science exposure. Seeing deteriorated as wind switched to NE. Second and third exposures had fwhm close to 2" 03:43Z Cartridge 2 1195 22.6 21.1 21.5 19.1 Done Guidestars on slew. Seeing variable between 1.6" and 2.0" Guide Fiber 6 shown as giant donut. Much better signal on this plate for the same 900s exposure time and seeing. Exposure time reduced to 700s for the next two. 04:35Z Clouds visible up to about 15 deg in the west 04:50Z Clouds leave. Seeing improving to 1.2 - 1.4 05:00Z We decide to switch to imaging Run 3818 28 S -23 to 40.9* *useable range not factoring in ramp-up/down. if knee in sky brightness due to impending moonrise is a problem, end lambda would be 36.4 skippy good from start in focus by frame 20, lambda -23 Required science start from plan is lambda -20 or frame 38 of this run. Everything is stable some 3 degrees of lambda prior to that. Seeing varying between good and excellent ( 0.8 to 1.3 arcseconds ) with a 10 frame excursion to yecch around frame 75 ID1 tape drives stopped writing. The archiver claimed to have disabled only the second drive, and should have autorecovered and continued writing to ID1-1. It didn't. ID1 seemed stalled. No change in state when paused. No increase in queue depth despite frames continuing to come in from the imager. It became clear that ID1 archiver was unresponsive, so we followed the recovery procedures, and everything started responding again. That is until we started archiving again, when everything claimed to be happy and ready, but it never wrote anything. It then stayed stuck not writing anything. We couldn't pause it. The queue depth froze at 87 ( where it was when we did archResume ). At this point we decided to leave things as they were until the run finished and we were back to spectroscopy. Moonrise frame 428, lambda 38.3 Sky level rise visible from frame 415, lambda 36.4 Ended drift at frame 445, lambda 40.9 Run 3819 Moonlight Donut Run Frame In Focus 10 +400 14 -400 18 Frames in /data/photo/optics/52724 10:00Z Switching back to spectro now that the moon is up. 10:18Z Cartridge 5 1041 17.7 15.7 19.1 15.9 Done Almost totally unilluminated preCal guiderFlat ( gimg0233 ) FF lamps were on. Could it be these blank frames we are seeing are due to an occasionally sticky shutter on the guide camera? Back to the ID1 problem. With the imager no longer taking data we go through the unresponsive archiver procedure again, this time rebooting the crate instead of just issuing archInit. This action cleared the problem with ID1. As soon as we issued a catasRecovery ID1 started writing. With over 200 frames to archive, this is going to take a while. Spectro ptvme failure during smear. Nothing written to disk. Too bright to try again by the time we recovered. This plate will not have a smear, since it is otherwise done. A bias taken at the end of the night had a exposure number mismatch which was caught and corrected by SOP. We close for the night. ID1 stopped again. Crate rebooted again. Archiving resumed. It completed this time. endNight was started at about 13:40Z, successfully navigated the frame and gang check and was happily writing the gang tapes when we quit for the night. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3816 23:08Z 23:32Z 100 O -120.64 -114.64 54 ignore 3817 05:09Z 05:24Z 100 O -30.30 -26.53 39 bias on telescope 3818 05:25Z 09:48Z 28 S -24.70 41.11 453 science 28 S 3819 09:49Z 10:00Z 28 S 41.55 44.06 27 engineering donuts ---=== IMAGING RUN DETAILS ===--- Run 3818 28 S -23 to 40.9* *useable range not factoring in ramp-up/down. if knee in sky brightness due to impending moonrise is a problem, end lambda would be 36.4 skippy good from start in focus by frame 20, lambda -23 Required science start from plan is lambda -20 or frame 38 of this run. Everything is stable some 3 degrees of lambda prior to that. Seeing varying between good and excellent ( 0.8 to 1.3 arcseconds ) with a 10 frame excursion to yecch around frame 75 Moonrise frame 428, lambda 38.3 Sky level rise visible from frame 415, lambda 36.4 Ended drift at frame 445, lambda 40.9 Run 3819 Moonlight Donut Run Frame In Focus 10 +400 14 -400 18 Frames in /data/photo/optics/52724 ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 3818 9 4 49 3.000 0.209 5.900 0.108 0.00301 324 81 3818 22 4 53 2.500 0.182 6.300 0.108 0.00168 329 81 3818 76 4 55 2.800 0.232 6.100 0.112 -0.00210 344 79 3818 126 4 17 4.100 0.148 5.500 0.070 -0.00367 355 78 3818 180 4 38 4.300 0.184 5.200 0.075 -0.00426 6 78 3818 235 4 36 5.400 0.156 4.300 0.094 -0.00490 18 78 3818 286 4 40 4.200 0.157 3.100 0.118 -0.00365 30 78 3818 338 4 63 5.700 0.445 1.900 0.721 -0.00316 45 79 3818 391 4 56 6.000 0.594 0.200 0.139 0.00176 64 80 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 3818 34 1 80 206 15 0.396 0.239 0.272 0.114 -0.00092 3818 34 1 80 206 18 0.727 0.189 0.374 0.120 -0.00126 3818 34 1 80 206 14 1.217 0.214 0.342 0.131 -0.00115 3818 34 1 80 206 12 1.989 0.284 0.424 0.114 -0.00143 3818 34 1 80 206 15 2.451 0.207 0.509 0.099 -0.00171 3818 51 1 80 201 16 -0.730 0.319 0.237 0.204 -0.00080 3818 51 1 80 201 20 -0.271 0.134 0.448 0.106 -0.00151 3818 51 1 80 201 15 0.237 0.274 0.400 0.176 -0.00134 3818 51 1 80 201 14 0.601 0.338 0.314 0.187 -0.00106 3818 51 1 80 201 19 1.186 0.334 0.326 0.161 -0.00110 3818 85 1 79 193 14 -0.422 0.161 -0.077 0.162 0.00026 3818 85 1 79 193 5 0.134 0.228 0.254 0.117 -0.00085 3818 85 1 79 193 7 0.743 0.224 0.127 0.156 -0.00043 3818 85 1 79 193 10 1.278 0.176 0.136 0.145 -0.00046 3818 85 1 79 193 15 1.792 0.284 0.208 0.186 -0.00070 3818 103 1 79 189 10 -0.604 0.238 0.463 0.175 -0.00156 3818 103 1 79 189 19 -0.154 0.232 0.659 0.214 -0.00222 3818 103 1 79 189 12 0.396 0.204 0.709 0.267 -0.00238 3818 103 1 79 189 13 1.065 0.216 0.717 0.261 -0.00241 3818 103 1 79 189 12 1.448 0.211 0.532 0.317 -0.00179 3818 138 1 78 181 8 -0.383 0.188 0.784 0.110 -0.00263 3818 138 1 78 181 5 0.148 0.103 0.991 0.028 -0.00333 3818 138 1 78 181 13 0.637 0.119 1.084 0.158 -0.00364 3818 138 1 78 181 11 1.276 0.188 1.019 0.097 -0.00343 3818 138 1 78 181 12 1.664 0.127 1.134 0.152 -0.00381 3818 155 1 78 178 10 -0.448 0.078 0.979 0.248 -0.00329 3818 155 1 78 178 9 0.025 0.103 1.049 0.228 -0.00353 3818 155 1 78 178 9 0.487 0.214 0.917 0.177 -0.00308 3818 155 1 78 178 12 1.101 0.138 1.113 0.098 -0.00374 3818 155 1 78 178 12 1.662 0.098 1.135 0.154 -0.00382 3818 192 1 78 170 5 -0.009 0.097 0.678 0.271 -0.00228 3818 192 1 78 170 9 0.615 0.221 0.888 0.179 -0.00298 3818 192 1 78 170 16 1.193 0.137 0.929 0.198 -0.00312 3818 192 1 78 170 12 1.657 0.139 0.947 0.196 -0.00318 3818 192 1 78 170 17 2.114 0.188 0.969 0.201 -0.00326 3818 209 1 78 166 9 -0.088 0.097 0.757 0.096 -0.00254 3818 209 1 78 166 8 0.425 0.079 0.823 0.123 -0.00277 3818 209 1 78 166 8 0.866 0.079 0.748 0.169 -0.00251 3818 209 1 78 166 11 1.360 0.144 0.841 0.109 -0.00283 3818 209 1 78 166 7 1.773 0.080 0.693 0.134 -0.00233 3818 245 1 78 158 11 -0.382 0.207 0.228 0.156 -0.00077 3818 245 1 78 158 16 0.028 0.153 0.386 0.124 -0.00130 3818 245 1 78 158 23 0.529 0.168 0.368 0.177 -0.00124 3818 245 1 78 158 18 1.097 0.171 0.390 0.125 -0.00131 3818 245 1 78 158 12 1.563 0.109 0.252 0.154 -0.00085 3818 263 1 78 154 12 -0.538 0.136 1.247 0.112 -0.00419 3818 263 1 78 154 13 -0.030 0.100 1.474 0.201 -0.00496 3818 263 1 78 154 15 0.558 0.135 1.527 0.201 -0.00513 3818 263 1 78 154 4 1.030 0.081 1.485 0.262 -0.00499 3818 263 1 78 154 19 1.548 0.106 1.493 0.252 -0.00502 3818 297 1 79 145 14 -0.870 0.163 -0.251 0.119 0.00085 3818 297 1 79 145 18 -0.580 0.142 -0.310 0.160 0.00104 3818 297 1 79 145 11 0.064 0.182 -0.324 0.200 0.00109 3818 297 1 79 145 12 0.596 0.175 -0.259 0.110 0.00087 3818 297 1 79 145 14 1.025 0.114 -0.291 0.219 0.00098 3818 315 1 79 140 10 -0.355 0.141 0.130 0.289 -0.00044 3818 315 1 79 140 10 0.093 0.094 0.398 0.102 -0.00134 3818 315 1 79 140 19 0.594 0.115 0.425 0.184 -0.00143 3818 315 1 79 140 17 1.141 0.167 0.403 0.124 -0.00136 3818 315 1 79 140 16 1.534 0.088 0.389 0.137 -0.00131 3818 349 1 80 129 21 0.111 0.151 -0.777 0.145 0.00261 3818 349 1 80 129 25 0.453 0.134 -0.714 0.163 0.00240 3818 349 1 80 129 13 1.003 0.208 -0.801 0.267 0.00269 3818 349 1 80 129 11 1.535 0.143 -0.581 0.084 0.00196 3818 349 1 80 129 29 1.891 0.155 -0.596 0.162 0.00200 3818 365 1 80 123 16 0.263 0.135 -1.021 0.183 0.00343 3818 365 1 80 123 20 0.688 0.106 -0.797 0.145 0.00268 3818 365 1 80 123 13 1.119 0.189 -0.790 0.138 0.00266 3818 365 1 80 123 17 1.698 0.126 -0.595 0.068 0.00200 3818 365 1 80 123 22 2.049 0.133 -0.658 0.156 0.00221 3818 400 1 80 109 14 0.709 0.100 -1.988 0.136 0.00668 3818 400 1 80 109 21 1.050 0.113 -1.799 0.121 0.00605 3818 400 1 80 109 19 1.566 0.127 -1.696 0.117 0.00570 3818 400 1 80 109 15 1.954 0.146 -1.647 0.109 0.00554 3818 400 1 80 109 26 2.434 0.150 -1.679 0.151 0.00564 3818 419 1 80 100 17 0.323 0.174 -3.261 0.174 0.01097 3818 419 1 80 100 28 0.630 0.155 -3.090 0.125 0.01039 3818 419 1 80 100 23 1.146 0.172 -2.998 0.150 0.01008 3818 419 1 80 100 12 1.598 0.175 -2.869 0.133 0.00965 3818 419 1 80 100 22 2.115 0.155 -2.890 0.104 0.00971 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n):yes QA Procedures Done (y/n):yes UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 20010, plate -9999 ------- 23:01 20010 0.0 bias ----- sequence 20011, plate 1234 ------- 23:12 20011 10.0 flat 23:14 20012 2.0 arc 23:20 20013 2.1 arc 23:23 20014 2.1 arc ----- sequence 20015, plate -9999 ------- 00:10 20015 0.0 bias ----- sequence 20016, plate 1234 ------- 02:20 20016 10.0 flat 02:23 20017 2.0 arc ----- sequence 20016, plate 1234 ------- 17.3 18.0 22.4 23.7 Done 02:45 20018 900.1 target 03:03 20019 900.1 target 03:21 20020 900.1 target 03:29 20021 240.0 smear 03:35 20022 10.0 flat 03:37 20023 2.0 arc ----- sequence 20024, plate 1195 ------- 22.6 21.1 21.5 19.1 Done 03:55 20024 10.0 flat 03:58 20025 2.0 arc 04:16 20026 900.1 target 04:31 20027 700.1 target 04:45 20028 700.1 target 04:54 20029 240.0 smear ----- sequence 20030, plate 1041 ------- 17.7 15.7 19.1 15.9 Done 10:33 20030 10.0 flat 10:36 20031 2.0 arc 10:57 20032 900.1 target 11:15 20033 900.1 target 11:33 20034 900.1 target ----- sequence 20036, plate -9999 ------- 11:51 20036 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 02:28Z 1 1234 56.46 0.0033 51.48 0.0000 224.95 0.0107 1.000120 03:54Z 2 1195 11.24 0.0033 64.07 0.0000 189.45 0.0107 1.000180 10:32Z 5 1041 218.43 0.0025 58.05 0.0017 62.38 0.0000 0.999890 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Assuming archiving is successful the following tapes will go. If not, other tapes will join the list. Goes: JL6162-67,76 Stays: JL6168-73,77 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 02:28Z 1 1234 1.00012 -1184 61 -135 57 51.5 3.7 14 17 - 1.6 03:55Z 2 1195 1.00018 -1776 -478 -150 11 64.1 3.9 5 1 - 1.6 05:35Z imager 1.00000 0 -559 -412 211 80.6 3.2 5 4 23-i' 0.94 06:31Z imager 1.00000 0 -538 -393 186 78.3 3.6 8 316 23-i' 1.32 07:48Z imager 1.00000 0 -566 -421 159 78.0 3.6 9 301 23-i' 1.07 09:10Z imager 1.00000 0 -588 -441 118 80.1 3.7 9 302 23-i' 0.89 10:31Z 5 1041 0.99989 1085 1986 -70 218 58.1 2.2 9 350 - 1.5 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:45Z 39 30 0 100 (E) 185 - Lt Snow 23:18Z 38 28 9 354 (NNW) 407 - Mostly Cloudy 23:50Z 40 27 3 292 (WNW) 468 - Mostly Cloudy 00:34Z 39 26 6 348 (NNW) 596 - Partly Cloudy 01:06Z 38 27 5 353 (NNW) 475 - Partly Cloudy 01:36Z 37 27 8 17 (NNE) 478 - Clear 02:07Z 37 19 18 25 (NNE) 540 - Clear 02:40Z 39 0 17 36 (NE) 521 - Clear 03:11Z 41 -3 15 18 (NNE) 390 - Clear 03:41Z 39 2 12 358 (NNW) 440 - Clear 04:12Z 38 6 13 18 (NNE) 473 - Clear 04:44Z 38 7 14 26 (NNE) 455 - Clear 05:14Z 39 7 3 357 (NNW) 497 - Clear 05:47Z 38 9 9 326 (NW) 527 - Clear 06:19Z 37 12 8 348 (NNW) 550 - Clear 06:49Z 38 11 9 308 (NW) 565 - Clear 07:22Z 38 12 5 279 (W) 569 - Clear 07:54Z 38 13 9 298 (WNW) 553 - Clear 08:26Z 39 15 14 306 (NW) 550 - Clear 09:00Z 38 14 11 317 (NW) 594 - Clear 09:31Z 38 10 10 315 (NW) 610 - Clear 10:02Z 37 12 9 307 (NW) 639 - Clear 10:32Z 36 17 2 342 (NNW) 622 - Clear 11:05Z 36 21 9 12 (NNE) 642 - Clear 11:37Z 35 22 6 3 (N) 613 - Clear 12:11Z 35 23 14 354 (NNW) 577 - Clear 12:43Z 36 23 16 4 (N) 516 - Clear 13:19Z 36 16 10 1 (N) 594 - Clear 13:51Z 37 15 13 2 (N) 592 - Clear ---=== TELESCOPE STATUS ===--- 00:05Z fans on, doors open 01:20Z Enclosure open 12:10Z Enclosure closed, fans off Stowed at 30 deg alt, Cartridge 3 installed. Spectro auto-fill connected. Spectrograph dewar 13:30Z Dewar wt 214 lb Imager dewar 13:30Z Dewar wt 66 lb No interlocks bypassed ---=== SOFTWARE USED ===--- IOP/SOP: v3_113_0 Watcher: v2_22_0 MCP: v5_18_0 TPM: tpm_v2_28_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_67 SoS: v4_9_13 hoggPT: v1_6_7 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0890 0.0790 Axial B 0.7980 0.7970 Axial C 0.8230 0.8240 Trans D -9.0630 -9.1130 Lateral E 1.8710 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5377. -3699. 923. -208. 31535. 30647. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 5.4 Alt.: 29.997535 MIGS TONIGHT NOMINAL Axial A 1.5170 1.5170 Axial B 1.1570 1.1560 Axial C 1.1010 1.0990 Trans D -0.6990 -0.7070 GALILS Commanded: 1622358. 1554175. 1576203. -3400. -7900. Actual: 1619636. 1555273. 1576326. -3383. -7925. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1257.00 -0.01 -20.00 -0.48 130.89 ---=== PROBLEMS IN DETAIL ===--- Wrong Number of Pixels: Beginning of the night VME errors (on all nodes) regarding num of pix --- the bias was running when this happened, but we saw no pink column or anything indicating this was the case: dsc E trcErr (throttle) WRONG num of pix?? 2130 should be 2128 (dsc_E_trcErr (throttle) WRONG num of pix?? 2130 should be 2128 : 2003-03-25 23:17:37Z ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000004 rowIdx:566 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000004 rowIdx:566 : 2003-03-25 23:17:37Z ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000003 rowIdx:566 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000003 rowIdx:566 : 2003-03-25 23:17:37Z ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000002 rowIdx:566 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000002 rowIdx:566 : 2003-03-25 23:17:37Z ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000001 rowIdx:566 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000001 rowIdx:566 : 2003-03-25 23:17:37Z ) dsc E trcErr WRONG num of pix?? 2130 should be 2128 (dsc_E_trcErr WRONG num of pix?? 2130 should be 2128 : 2003-03-25 23:17:37Z ) dsc E trcErr throttled 0 -- WRONG num of pix?? 2130 should be 2128 (dsc_E_trcErr throttled 0 -- WRONG num of pix?? 2130 should be 2128 : 2003-03-25 23:17:37Z ) dsdsc E trcErr throttled 0 -- WRONG num of pix?? for (sr<<16)|ccd:0x0000000 (dsc_E_trcErr throttled 0 -- WRONG num of pix?? for (sr<<16)|ccd:0x0000000 : 2003-03-25 23:17:37Z ) c E trcErr throttled 0 -- WRONG num of pix?? for (sr<<16)|ccd:0x0000000 (dsc_E_trcErr throttled 0 -- WRONG num of pix?? for (sr<<16)|ccd:0x0000000 : 2003-03-25 23:17:37Z ) dsc E trcErr (throttle) WRONG num of pix?? for (sr<<16)|ccd:0x00000000 ro (dsc_E_trcErr (throttle) WRONG num of pix?? for (sr<<16)|ccd:0x00000000 ro : 2003-03-25 23:17:37Z ) While clicking away the watcher error message, (ad1 errors on the VME errors), I got a popup window saying "error:can't read "errorFlags(da, vme, ad1, dsc_E_trcErr(throttle) WRONG num of pixx?? 22130 should be 2128)":no such element in array. The stack trace said: can't read "errorFlags(da,vme,ad1,dsc_E_trcErr (throttle) WRONG num of pix?? 2130 should be 2128 )": no such element in array while executing "expr $errorWords($sys,$sub,$comp) & ~$errorFlags($sys,$sub,$comp,$sc)..." invoked from within "set errorWords($sys,$sub,$comp) [expr $errorWords($sys,$sub,$comp) & ~$errorFlags($sys,$sub,$comp,$sc)]..." invoked from within "if {$sc != ""} { # there's a sub-component set errorWords($sys,$sub,$comp) [expr $errorWords($sys,$sub,$comp) & ~$errorFlags($sys,$sub,$comp, ..." (procedure "clear_flag" line 4) invoked from within "clear_flag $sys $sub $comp $sc" (procedure "clear_status" line 6) invoked from within "clear_status ._ad1_1048634322_dialog.frame.values.row2.left da vme ad1 {dsc_E_trcErr (throttle) WRONG num of pix?? 2130 should be 2128 } daError" ("eval" body line 1) invoked from within "eval clear_status ._ad1_1048634322_dialog.frame.values.row2.left da vme ad1 {{dsc_E_trcErr (throttle) WRONG num of pix?? 2130 should be 2128 }} daErro ..." invoked from within "._ad1_1048634322_dialog.frame.values.row2.left invoke" ("uplevel" body line 1) invoked from within "uplevel #0 [list $w invoke]" invoked from within "if {($w == $tkPriv(window)) && ([$w cget -state] != "disabled")} { uplevel #0 [list $w invoke] }" invoked from within "if {$w == $tkPriv(buttonWindow)} { set tkPriv(buttonWindow) "" $w config -relief $tkPriv(relief) if {($w == $tkPriv(window)) && ([$w cget -state] ..." (procedure "tkButtonUp" line 3) invoked from within "tkButtonUp ._ad1_1048634322_dialog.frame.values.row2.left" (command bound to event) Then watcher went red on "rowsSkipped" for CCD71-76, CCD91-95. ============================================================================ Blank guider frame: With increasing frequency, some guider frames are showing essentially a bias level when it should have had signal. This has been most common with the first guider exposure after a preCal. However it has also been seen on the first frame after returning from an FK5 star. It was also seen tonight for the guider flat in a preCal (gimg0233). Possible candidates for this problem are the guider shutter, and the flat field screens. No errors have been reported during these events, so whatever is failing is doing so silently. ============================================================================ ID1 archiver hang: Three times tonight the ID1 node stopped archiving. It was complaining about being unable to check a tape ID. TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 0 (TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 0 : 2003-03-26 08:05:00Z ) archUseDrv (drive 2: archUseDrv is 0: 2003-03-26 08:05:16Z) archDrvSts (drive 2: Disabled: 2003-03-26 08:05:16Z) TEXTONLY *** WARNING *** dscArch: Couldn't check tape tape ID 0 (TEXTONLY *** WARNING *** dscArch: Couldn't check_tape tape ID 0 : 2003-03-26 08:05:00Z ) ID1 was completely unresponsive. Eventhough it wasn't archiving frames, the queue depth remained constant as the imager was reading. Following the unresponsive archiver procedures using archInit we got the archiver talking again, but as soon as we asked it to write frames it hung again. Waiting until the end of the imaging run, we then went through the bigger hammer version of the recovery procedures and rebooted the crates. This time it started writing frames when we ran catasRecovery. It got about a third of the way through the backlog of 200+ frames and hung again. At this point it was the end of the night. We ran the recovery with reboot one more time and it finally finished archiving about 13:40Z, at which point we started endNight at left. ============================================================================ Spectro ptvme failure: On the last exposure of the night a ptvme failure booted Pete out of his sop session during the smear. After a ptvmeReset there was an exposure number mismatch on a test bias which was corrected in sop ( 20035 ). Here are excerpts from Pete's sop session. SOPGUI: smear mcpPut: SYSTEM.STATUS Setting up for the smears Opening Flat Field Screen, just in case mcpPut: ffs.select 3 ffs.open sp1 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 sp2 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 Flushing charge for 15 sec mcpPut: SYSTEM.STATUS appending BEGIN Offseting Az: -1.250 Alt: 3.000 Arcseconds Starting smear 0 Exposed for 121.269 seconds Offseting Az: 1.250 Alt: 3.000 Arcseconds Starting smear 1 Exposed for 123.189 seconds Finishing up, yippee appending END EXPOSURE TIME is: 240.0 sp1 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 sp2 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 sp1 camstat {} t-g-sdss-2.apo.nmsu.edu 2100 sp2 camstat {} t-g-sdss-2.apo.nmsu.edu 2200 Access to shared data area at 0x18000000 lost (errno=0x10 Resource busy) Read failed from PT icc40 CSB=0x00000000 rmt=0x180866B0 cnt=0x00000004 ******************************************************************************** * * * Backplane communications between the ICC/OAC are no longer possible. * * * * Perform the following steps until operations are possible. * * * * 1. Check connections between the machines. * * 2. Reboot (reset) the ICC. * * 3. Restart this (OAC) program. * * 4. Reset the local PT-VME940 (ptvmeMode -R). * * 5. Power-cycle the VMEbus crates. * * 6. Reboot the local machine (host). * * * ******************************************************************************** prn:sdsshost:~/ > setup ptvme prn:sdsshost:~/ > ptvmeReset ptvmeMode: INFO: controller 0 reset (/dev/ptvme0n0) ptvmeMode: INFO: cur debug mode: 0 NORMAL ptvmeMode: INFO: cur debug: 0x00000000 ptvmeMode: INFO: cur timeout: 900 ticks ptvmeMode: INFO: def T/I xfer mode: 0x22 A32:D32 Program, Non-Privileged / A32:D32 Program, Non-Privileged ptvmeMode: INFO: op mode: 1 SINGLE_IO ptvmeMode: INFO: VMEbus priority: 0x03 ptvmeMode: INFO: VMEbus vector: 0x01 ptvmeMode: INFO: intern sem val: 1 ptvmeMode: INFO: user sem val: 1 ptvmeMode: INFO: PT-VME940 node: 0 prn:sdsshost:~/ > then restarted SOP and did: sp1> endStare -abort -force appending ABORT Dropping at exit Dropping at exit Do not schedule startFlushCharge sp1 200 Do not schedule startFlushCharge sp2 200 sp1> grabInst -handlers 123 IOP running as Backup Relay Server requestListen: listening 2003-03-26 11:49:33Z 1 sp1 camcheck 2003-03-26 11:49:36Z 1 sp1 ln2stat 2003-03-26 11:49:36Z 1 sp2 camcheck 2003-03-26 11:49:39Z 1 sp2 ln2stat sp1> 2003-03-26 11:49:41Z: registered (addr1) listener sdsshost:observer:82713 sp1> goStare -bias PlateId [-9999]: ***************************************** WARNING: SOP and the DA have conflicting records of the current frame number. SOP's value: 20035 DA's value: 20034 You can: 1. Type "1" to have goStare set both to 20035 and continue 2. Type "2" to abort goStare and fix it yourself goStare> 1 sp1 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 sp2 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 Flushing charge for 15 sec mcpPut: SYSTEM.STATUS mcpPut: version mcpPut: SYSTEM.STATUS mcpPut: USER.ID prn@sdsshost 375160 appending BEGIN appending END EXPOSURE TIME is: 0 sp1 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 sp2 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 sp1 camstat {} t-g-sdss-2.apo.nmsu.edu 2100 sp2 camstat {} t-g-sdss-2.apo.nmsu.edu 2200 1048679453 1048679459 2003-03-26 11:51:13Z: scheduling gangs2Unix transfers The old teamster, PID 118061, appears to have died; restarting 2003-03-26 11:51:14Z: finished with iccExec -riskyOn data -async sd1 readoutBegin sp1 cameraStart {} t-g-sdss-2.apo.nmsu.edu 2100 sp2 cameraStart {} t-g-sdss-2.apo.nmsu.edu 2200 Waiting for read to finish Seq 20036 Exp 20036 Frame 20036 Do not schedule startFlushCharge sp1 200 Do not schedule startFlushCharge sp2 200 sp1> and then all seems well.