APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday January 12, 2002 (MJD 52287) ---=== OBSERVING TEAM ===--- Daniel Long Mike Harvanek By Phone: Jon Brinkmann French Leger Craig Loomis Ron Rechenmacher ---=== OBSERVING PLAN ===--- Imaging if conditions allow, spectroscopy otherwise. ---=== OBSERVING SUMMARY ===--- An all imaging night. Two relatively long scans done. Run 2873 covered 82 N between -162 and -123. Run 2883 convered 31 N between -29 and 8. Lost about 3 hours in the middle of the night due to some problem involving the crates and/or sdsshost. ---=== OBSERVING LOG ===--- During the afternoon check we got errors trying to take spectro exposures. We got a message about a teamster being too old and attempting to restart. However it couldn't be killed. We got ahold of Craig and he found the parent process also needed to be killed in order for the process to die. The first scan, 82 N, went smoothly. Donuts were taken at the start of the scan and are saved in /data/photo/optics/52287. Things began to degenerate rapidly upon going to the next stripe, 30 S. The altitude aborted shortly after arrival, verified by streaks in the drift. altitude (MGT: bad alt state ABORT_EVENT: ID_ERROR_LIMIT (Jan 12 21:31:26): Jan 12 21:31:42 MST ) An axis stop and a couple axis init's got it back. Joy was short lived however. The next goDrift was greeted with a slew of errors, one from each id node complaining that it couldn't read /p/astrobase/fitsHeaders.tcl. For about the next two and a half hours the problem was debugged. Jon Brinkmann, Craig Loomis, and Ron Rechenmacher all looked at the problem. There apparently was some resource issue on host which caused the id nodes to lose the ability to see the host disk via nfs. Things continued to get worse on host to the point where commands stopped working and could not be ctrl-C'd out of or killed. In the process of cleaning up iops preparing to reboot host, things started working again upon quitting a sop session ( by destroying the window ). In this sop session an iccSoundOff was issued which had hung. The crates still needed to be rebooted to fix the problem of reading files on host. Nearly 3 hours after the problem first started we issued a goDrift -bias drift.... and saw stars! This may mean the shutters were open the entire time we were dealing with the problem. We stopped that run, and started a new bias drift. No stars this time. We let this one run for a while to clear the chips and to let astroline recover from the high levels encountered at the start of the previous bias run. At this point we decided to try 31 N. We got setup for this stripe and shortly after the scan started, clouds appeared in the NE. They weren't where we were pointed, but were large enough and near enough that there is some chance that the data is not photometric. As the satellite showed the clouds would pass by, we kept imaging. Once it was photometric again, we decided to restart the 31 N scan to be absolutely sure the data would be good. This scan went for the entire desired lambda range. Since the plan indicated it could go beyond the end specified we let it continue for the rest of the night. However the seeing worsened after we left the desired scan range and did not return to acceptable values. We ended the drift and put the telescope away and started endNight as it was nearing 12:30 anyway. At the end of the night when putting the telescope away, the laptop (sirrobert) had the cursor displacement problem again. A PR has been filed against unknown hardware. endNight didn't quite cross the finish line. It bailed because it couldn't find files associated with runs attempted during the problems we had with host and the id nodes. It was attempting to mail the run summaries, but it didn't do it for runs 2882 and 2883 since these came after the missing runs. All the important steps completed ( gang tapes written for all runs and tpm ). ---=== IMAGING RUN SUMMARY ===--- Run Stripe Flavor Lambda Last Begin End Frame Comments ------------------------------------------------------------------ 2871 100 O ignore 160.63 170.64 78 2872 100 O bias -172.59 -168.32 43 Real Bias 2873 82 N science ~-162 -123 ~300 No problem here 2880 100 O ignore -65.11 -30.04 15 2881 100 O bias -63.28 -61.06 22 Probably not useful bias 2882 31 N science -31.48 -17.40 101 Clouds nearby 2883 31 N science -29.50 20.80 342 Clear, bad seeing > 8 ---=== IMAGING RUN DETAILS ===--- 82 N, Run 2873: --------------- Pointing in spec from the start. Donuts taken during twilight. frame 24 In focus frame 27 +300 frame 31 -300 saved to /data/photo/optics/52287 Frame 65 is start of survey area. Data good from frame 35 ( except twilight didn't end until about frame 55 ). 30 S, Run 2874: --------------- Alt abort shortly after arriving at next stripe so Run 2874 is a bunch of streaks. 31 N, Run 2882: --------------- Pointing in spec from the start. An isolated but fairly large cloud in the NE was present shortly after the run started. The cloud was always 30 degrees or more away from where we were pointed. We aren't absolutely certain this data can be called photometric, so this run will be repeated. It was decided to let the drift continue an wait out the cloud rather than stop the scan. 31 N, Run 2883: --------------- Pointing good from the start. Seeing went bad between frames 250 and 270. It didn't recover. ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot --------------------------------------------------------------- 2873 10 4 42 3.900 0.932 -4.400 0.810 0.00912 2873 44 4 46 2.400 0.653 -3.700 0.241 0.00811 2873 79 4 48 2.900 0.161 -3.100 0.146 0.00746 2873 114 4 50 4.500 0.132 -2.500 0.111 0.00735 2873 148 4 40 4.200 0.114 -1.900 0.126 0.00762 2873 183 4 48 4.900 0.456 -2.100 0.554 0.00836 2873 217 4 53 3.300 0.570 -2.100 0.149 0.00732 2873 251 4 65 4.200 0.137 -1.900 0.126 0.00757 2882 10 4 60 0.500 0.141 -0.300 0.105 -0.00254 2882 45 4 50 1.900 0.157 0.400 0.096 -0.00343 2882 80 4 65 1.200 0.145 0.600 0.130 -0.00396 2883 17 4 48 1.500 0.095 1.500 0.127 -0.00967 2883 51 4 50 3.600 0.630 1.900 0.147 -0.00816 2883 86 4 59 2.200 0.145 2.200 0.107 -0.00756 2883 121 4 48 3.200 0.322 2.500 0.113 -0.00760 2883 155 4 48 1.400 0.634 2.500 0.559 -0.00597 2883 191 4 52 1.500 0.109 3.100 0.094 -0.00663 2883 224 4 51 2.100 0.130 3.400 0.098 -0.00604 2883 259 4 51 3.500 0.108 3.300 0.094 -0.00576 2883 294 4 52 3.200 0.099 3.300 0.136 -0.00494 2883 328 4 39 2.500 0.130 3.100 0.120 -0.00405 2883 338 4 68 3.300 0.122 2.900 0.141 -0.00456 2883 338 4 68 3.300 0.122 2.900 0.141 -0.00456 2883 338 4 68 3.300 0.122 2.900 0.141 -0.00456 2883 338 4 68 3.300 0.122 2.900 0.141 -0.00456 2883 338 4 68 3.300 0.122 2.900 0.141 -0.00456 ---=== LTMATCH RESULTS ===--- Run Field nFields nGood rowMean rowSig colMean colSig rot --------------------------------------------------------------- 2873 56 1 16 -0.175 0.630 -1.591 1.606 0.00535 2873 56 1 13 -0.580 0.667 -1.356 1.367 0.00456 2873 56 1 20 -0.988 1.095 -1.536 1.552 0.00517 2873 56 1 12 -1.336 1.391 -1.777 1.789 0.00598 2873 56 1 14 -1.478 1.496 -1.510 1.539 0.00508 2873 90 1 10 0.594 0.625 -2.131 2.159 0.00716 2873 90 1 7 0.328 0.418 -2.496 2.498 0.00839 2873 90 1 14 0.193 0.345 -2.275 2.282 0.00765 2873 90 1 15 -0.422 0.502 -2.424 2.428 0.00815 2873 90 1 8 -0.515 0.525 -2.513 2.516 0.00845 2873 124 1 13 0.039 0.098 -1.962 1.970 0.00660 2873 124 1 11 -0.058 0.114 -2.165 2.173 0.00728 2873 124 1 17 -0.259 0.273 -2.089 2.100 0.00702 2873 124 1 14 -0.370 0.389 -1.978 1.983 0.00665 2873 124 1 15 -0.676 0.682 -2.132 2.149 0.00717 2873 157 1 16 -0.132 0.171 -2.773 2.783 0.00932 2873 157 1 18 -0.355 0.411 -2.990 2.992 0.01005 2873 157 1 9 -0.684 0.704 -3.083 3.084 0.01036 2873 157 1 14 -0.794 0.800 -3.067 3.070 0.01031 2873 157 1 12 -1.102 1.105 -3.069 3.075 0.01032 2873 193 1 15 -0.148 0.254 -3.063 3.075 0.01030 2873 193 1 26 -0.328 0.370 -3.161 3.179 0.01063 2873 193 1 14 -0.603 0.622 -3.403 3.416 0.01144 2873 193 1 12 -0.867 0.872 -3.241 3.247 0.01090 2873 193 1 17 -0.984 0.995 -3.286 3.294 0.01105 2873 228 1 15 -0.162 0.243 -2.654 2.657 0.00892 2873 228 1 17 -0.204 0.308 -2.700 2.702 0.00908 2873 228 1 19 -0.531 0.583 -2.945 2.955 0.00990 2873 228 1 10 -0.808 0.825 -2.842 2.844 0.00955 2873 228 1 21 -1.012 1.027 -2.938 2.945 0.00988 2873 262 1 23 -0.113 0.231 -2.937 2.946 0.00988 2873 262 1 18 -0.229 0.320 -3.124 3.137 0.01050 2873 262 1 10 -0.404 0.458 -3.132 3.139 0.01053 2873 262 1 13 -0.871 0.887 -3.391 3.397 0.01140 2873 262 1 23 -0.935 0.954 -3.352 3.361 0.01127 2882 21 1 10 0.343 0.630 1.317 1.325 -0.00443 2882 21 1 36 0.173 0.353 1.297 1.299 -0.00436 2882 21 1 23 -0.227 0.472 1.213 1.221 -0.00408 2882 21 1 9 -0.282 0.406 1.366 1.371 -0.00459 2882 21 1 20 -0.480 0.590 1.356 1.363 -0.00456 2882 57 1 11 0.478 0.509 1.734 1.738 -0.00583 2882 57 1 14 0.393 0.425 1.809 1.813 -0.00608 2882 57 1 17 0.047 0.267 1.619 1.626 -0.00544 2882 57 1 13 -0.309 0.367 1.676 1.678 -0.00564 2882 57 1 18 -0.390 0.443 1.538 1.541 -0.00517 2882 89 1 12 1.137 1.159 1.333 1.351 -0.00448 2882 89 1 20 0.931 0.964 1.385 1.410 -0.00466 2882 89 1 14 0.707 0.724 1.598 1.630 -0.00537 2882 89 1 10 0.460 0.513 1.396 1.440 -0.00469 2882 89 1 12 0.170 0.213 1.450 1.548 -0.00487 2883 28 1 14 0.849 0.868 2.721 2.724 -0.00915 2883 28 1 10 0.564 0.566 2.745 2.749 -0.00923 2883 28 1 17 0.188 0.284 2.593 2.602 -0.00872 2883 28 1 10 -0.224 0.323 2.776 2.785 -0.00933 2883 28 1 14 -0.388 0.453 2.850 2.866 -0.00958 2883 63 1 7 0.265 0.324 2.485 2.512 -0.00836 2883 63 1 22 -0.229 0.307 2.680 2.690 -0.00901 2883 63 1 15 -0.446 0.504 2.557 2.568 -0.00860 2883 63 1 15 -0.708 0.741 2.707 2.709 -0.00910 2883 63 1 12 -0.816 0.834 2.822 2.824 -0.00949 2883 98 1 17 0.478 0.505 2.798 2.804 -0.00941 2883 98 1 10 0.219 0.277 2.866 2.871 -0.00963 2883 98 1 11 -0.114 0.165 2.779 2.786 -0.00934 2883 98 1 9 -0.245 0.288 2.834 2.840 -0.00953 2883 98 1 8 -0.490 0.497 2.783 2.789 -0.00936 2883 132 1 11 -0.022 0.087 2.429 2.445 -0.00817 2883 132 1 11 -0.168 0.193 2.447 2.450 -0.00823 2883 132 1 16 -0.548 0.574 2.347 2.353 -0.00789 2883 132 1 6 -0.590 0.595 2.530 2.531 -0.00851 2883 132 1 18 -0.843 0.861 2.507 2.511 -0.00843 2883 165 1 15 0.017 0.129 1.918 1.928 -0.00645 2883 165 1 17 -0.159 0.235 1.736 1.750 -0.00584 2883 165 1 10 -0.480 0.508 1.786 1.798 -0.00601 2883 165 1 14 -0.634 0.674 1.587 1.597 -0.00534 2883 165 1 22 -0.821 0.836 1.670 1.680 -0.00562 2883 200 1 14 0.886 0.940 1.958 1.980 -0.00658 2883 200 1 12 0.650 0.706 1.983 2.005 -0.00667 2883 200 1 15 0.306 0.437 1.875 1.894 -0.00630 2883 200 1 9 0.064 0.381 1.948 1.971 -0.00655 2883 200 1 10 -0.071 0.337 1.920 1.949 -0.00645 2883 235 1 13 0.987 1.006 1.366 1.371 -0.00459 2883 235 1 15 0.713 0.731 1.499 1.518 -0.00504 2883 235 1 11 0.332 0.388 1.299 1.310 -0.00437 2883 235 1 9 -0.092 0.205 1.348 1.359 -0.00453 2883 235 1 10 -0.210 0.277 1.548 1.565 -0.00521 2883 269 1 5 0.452 0.501 0.548 0.577 -0.00184 2883 269 1 13 0.022 0.265 0.508 0.548 -0.00171 2883 269 1 9 -0.159 0.231 0.670 0.719 -0.00225 2883 269 1 6 -0.458 0.494 0.843 0.938 -0.00283 2883 269 1 4 -0.687 0.700 0.982 1.017 -0.00330 2883 304 1 7 0.819 0.835 0.014 0.315 -0.00005 2883 304 1 10 0.586 0.605 0.478 0.605 -0.00161 2883 304 1 8 0.097 0.327 0.107 0.274 -0.00036 2883 304 1 7 -0.018 0.306 0.433 0.652 -0.00145 2883 304 1 10 -0.213 0.374 0.489 0.594 -0.00164 2883 331 1 12 0.124 0.304 0.671 0.834 -0.00226 2883 331 1 10 0.160 0.227 0.726 0.789 -0.00244 2883 331 1 10 -0.517 0.532 0.575 0.800 -0.00193 2883 331 1 6 -0.355 0.432 0.723 0.836 -0.00243 2883 331 1 9 -0.605 0.659 0.028 0.628 -0.00009 2883 331 1 12 0.124 0.304 0.671 0.834 -0.00226 2883 331 1 10 0.160 0.227 0.726 0.789 -0.00244 2883 331 1 10 -0.517 0.532 0.575 0.800 -0.00193 2883 331 1 6 -0.355 0.432 0.723 0.836 -0.00243 2883 331 1 9 -0.605 0.659 0.028 0.628 -0.00009 2883 331 1 12 0.124 0.304 0.671 0.834 -0.00226 2883 331 1 10 0.160 0.227 0.726 0.789 -0.00244 2883 331 1 10 -0.517 0.532 0.575 0.800 -0.00193 2883 331 1 6 -0.355 0.432 0.723 0.836 -0.00243 2883 331 1 9 -0.605 0.659 0.028 0.628 -0.00009 2883 331 1 12 0.124 0.304 0.671 0.834 -0.00226 2883 331 1 10 0.160 0.227 0.726 0.789 -0.00244 2883 331 1 10 -0.517 0.532 0.575 0.800 -0.00193 2883 331 1 6 -0.355 0.432 0.723 0.836 -0.00243 2883 331 1 9 -0.605 0.659 0.028 0.628 -0.00009 2883 331 1 12 0.124 0.304 0.671 0.834 -0.00226 2883 331 1 10 0.160 0.227 0.726 0.789 -0.00244 2883 331 1 10 -0.517 0.532 0.575 0.800 -0.00193 2883 331 1 6 -0.355 0.432 0.723 0.836 -0.00243 2883 331 1 9 -0.605 0.659 0.028 0.628 -0.00009 2883 331 1 12 0.124 0.304 0.671 0.834 -0.00226 2883 331 1 10 0.160 0.227 0.726 0.789 -0.00244 2883 331 1 10 -0.517 0.532 0.575 0.800 -0.00193 2883 331 1 6 -0.355 0.432 0.723 0.836 -0.00243 2883 331 1 9 -0.605 0.659 0.028 0.628 -0.00009 ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 23:13 12627 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument getclock Az Alt Rot Scale ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL4907-4912,4919 Stays: JL4913-4918,4920 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 19:48 imager 1.00000 0 -364 -264 -21 55.6 1.2 13 297 23-i' 1.2 20:58 imager 1.00000 0 -401 -300 -21 55.6 0.6 8 298 23-i' 1.2 01:17 imager 1.00000 0 -462 -325 142 72.2 1.0 8 288 23-i' 1.4 02:05 imager 1.00000 0 -476 -332 171 77.0 0.2 11 307 23-i' 1.1 03:53 imager 1.00000 0 -503 -375 154 66.9 1.4 14 268 23-i' 1.2 ---=== WEATHER LOG ===--- Weather data missing bewteen 04:30 and 07:30. The sky was clear except for a few high clouds between 05:30 and 06:30. Clouds returned for a while between 07:50 and 08:50. Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:16 33 5 4 331 (NNW) 77 - Clear 23:47 34 0 4 293 (WNW) 40 - Clear 00:19 33 -5 2 357 (NNW) 13 - Clear 00:49 34 -4 7 330 (NNW) 20 - Clear 01:19 34 -5 8 318 (NW) 19 - Clear 01:49 33 -5 8 323 (NW) 21 - Clear 02:19 33 -4 12 325 (NW) 24 - Clear 02:50 34 -2 11 310 (NW) 21 - Clear 03:20 33 1 5 270 (W) 35 - Clear 03:50 32 1 9 297 (WNW) 41 - Clear 07:44 33 0 10 296 (WNW) 63 - Mostly Clear 08:16 33 2 18 286 (WNW) 77 - Mostly Clear 08:48 33 7 8 292 (WNW) 97 - Clear 09:20 32 9 7 291 (WNW) 75 - Clear 09:53 32 9 9 268 (W) 80 - Clear 10:27 34 9 12 274 (W) 78 - Clear 10:59 34 10 15 253 (WSW) 49 - Clear 11:31 34 8 11 268 (W) 57 - Clear 12:02 33 8 15 271 (W) 55 - Clear 12:34 32 8 12 255 (WSW) 69 - Clear 13:06 32 8 14 266 (W) 57 - Clear 06:38 32 5 15 270 (W) 64 - ---=== TELESCOPE STATUS ===--- 23:20 fans on doors open 00:20 doors closed for imager bias ( fans still on ) 00:45 Enclosure open, fans on 12:30 Enclosure closed, fans off Stowed at 30 deg alt, imager installed. Spectro auto-fill connected. Spectrograph dewar 13:30 Dewar wt 26 lb Imager dewar 13:30 Dewar wt 116 lb Interlocks bypassed: Building, Slip trip ---=== SOFTWARE USED ===--- IOP/SOP: v3_64_3c Watcher: v2_13_2a MCP: v5_8_7 TPM: v2_4_5 AstroDa: v14_40 TCC:2.5.5 sdssProcedures: v1_37 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0750 0.0750 Axial B 0.8030 0.8020 Axial C 0.0710 0.0710 Trans D -1.4900 -1.4860 Lateral E 1.8350 1.8540 Lateral F 1.4150 1.4240 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 1.1 Alt.: 30.000000 MIGS TONIGHT NOMINAL Axial A 1.5460 1.5500 Axial B 1.2250 1.2260 Axial C 1.2470 1.2480 Trans D 0.2110 0.2320 GALILS Commanded: 1599643. 1565255. 1586307. -10395. -9766. Actual: 1599620. 1565254. 1586350. -10395. -9766. SETMIR VALUES SecDesOrient: 1257.00 -3.40 -10.00 0.00 133.82 SecOrient: 1256.99 -3.38 -10.01 0.00 133.81 ---=== PROBLEMS IN DETAIL ===--- goDrift error: -------------- Jan 12 21:45:01 sdsshost IDA 276058 TEXTONLY beginData has failed: {id1 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {id2 {couldn't read file "/p/astrobase/fitsHe... Jan 12 21:45:01 sdsshost IDA 276058 TEXTONLY ...aders.tcl"}} {id3 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {id4 {couldn't read file "/p/astrobase/fitsHeaders.tcl... Jan 12 21:45:01 sdsshost IDA 276058 TEXTONLY ..."}} {id5 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {id6 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {ad1 ... Jan 12 21:45:01 sdsshost IDA 276058 TEXTONLY ...{}} {ad2 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {ad3 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {ad4 ... Jan 12 21:45:01 sdsshost IDA 276058 TEXTONLY ...{}} Jon Brinkmann, Craig Loomis, and Ron Rechenmacher all got in on this one. Jon suspected an nfs problem, but thought Craig might know more. Craig took a long hard look at the id nodes, discovering that while the nfs device ( given by nfsDeviceShow ) was correct, an ls would not show the directory structure. The file was visible when looked at from host. There was no problem with read permission. Only when an attempt was made to look at it from an id node, it couldn't see anything in /p. Ron got an error from an ls on the node. The error number was translated to too many files open. Ron rebooted the id node boards, and that seemed to clear the problem. That is the nodes could again see the directory structure on host. At this point though, we lost the ability to execute most commands involving the crates and new iop sessions. ptvme reset wouldn't work either. You couldn't even ctrl-C to get out. The processes causing the hang couldn't be killed. It was decided to reboot sdsshost. In the process of cleaning up before the reboot ( getting out of iops etc ), the problem cleared. Getting rid of a sop session cleared the problem. In this sop session an iccSoundOff was issued which had hung. Ron found a command which will tell you if any of the nodes is having this read problem. If everything is OK it will simply return the node names. If there is an error, it should pass back the error. iccExec -riskyOn control {id1 id2 id3 id4 id5 id6 ad1 ad2 ad3 ad4 } exec ls /p We found we had to reboot the ad nodes as well for the same inability to read /p/astrobase/fitsHeader.tcl. At this point we successfully initiated a drift. Have another error: AxisDTime:az (-1.1: Jan 13 00:33:02 MST) Teamster Problem: ----------------- During the afternoon check, the following was seen when trying to take a bias frame with the spectrographs. sp1> goStare -init -bias PlateId [-9999]: Flushing charge for 15 sec appending BEGIN appending END EXPOSURE TIME is: 0 1010875685 1010875685 The running teamster is too old. Attempting to restart Starting a new teamster; moving old one to the Meadowlands beginData has failed: Failed to kill old teamster, 28524716 Error sp1> A grep shows the process. sdsshost> ps -elf |grep 28524716 0 S long 28524716 26244783 0 20 20 * 1309:449 53b64b8 17:08:44 pts/33 3:05 iop.bin -command getGangs 60 spectr After attempting to kill the process, first as long, second as root, third as root using kill -9, the grep showed the same thing. The same response was seen from sop after all the kill attempts. sdsshost> ps -elf |grep 28524716 0 Z long 28524716 26244783 0 0 - - - - - - - 0:00 A call to Craig got us back on track. He killed the parent process ( the second pid in the grep ). Unfortunately we don't know how the problem arose in the first place. endNight failure message: ----------------------- endNight mail imager run summaries: Processing 2873Open /CPS Close /CPS Open /CPS Close /CPS 2875Open /CPS Cannot find idAstroline-c3-002875.par in any of /astrolog/52287 /usrdevel/observer/ppd/52287 Open /CPS Cannot find idAstroline-c1-002875.par in any of /astrolog/52287 /usrdevel/observer/ppd/52287 2876Open /CPS Cannot find idAstroline-c3-002876.par in any of /astrolog/52287 /usrdevel/observer/ppd/52287 Open /CPS Cannot find idAstroline-c1-002876.par in any of /astrolog/52287 /usrdevel/observer/ppd/52287 Error: couldn't open "/tmp/fwhm27605812.ps": No such file or directory