APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday November 12, 2005 (MJD 53687) ---=== OBSERVING TEAM ===--- Swing: Olena Malanushenko Viktor Malanushenko Kaike Pan Night: John Barentine Support: John Marriner (FNAL) Eric Neilsen (by phone) Kurt Biery (by phone) ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- We lost about 2 hours of time to a fatal problem getting the servers to start. See Problems section for details. Despite this, we lost only about 45 minutes of actual post-twilight dark sky time to the problem. We imaged superNova stripe 82 S from 111.5 < lambda < -125, excluding 40 degrees near the moon from 167.5 < lambda < -152.5 (not including ramps). With that extra 40 degrees we then went back and imaged 82 N from -161.2 < lambda < -122.5. This was a sub-par night in terms of seeing and photometricity, although weather never threatened us with closure. For the remaining ~1h15m of the night, we switched to spectro and accumulated a minimum (S/N)^2=4.5 on Plate 2285. In addition to thanking Kurt and Eric for their superDuper weekend afternoon phone support during the DA problem, I would like to send a special shout-out to Viktor, Olena and Kaike for their hard work and dedication to getting us on the sky as soon as possible tonight. I couldn't have pulled this off without their help. All these folks helped avoid near-certain disaster and assured a reasonably productive night of science. ---=== OBSERVING LOG ===--- Afternoon --------- We lost the afternoon setup time to a problem getting the servers up and running. See Problems section. Checkout did not begin in earnest until after senset at 00:30Z. We did not check the spectro focus in the interest of getting on the sky as early as possible. Night ----- We took an on-telescope bias (Run 5869) in late twilight while waiting for the sky to get dark. 02:11Z Run 5870 (SN Stripe 82 S, startLambda=+111.5) For the second night in a row, the telescope bailed out almost immediately on the scanstripe, all three axes halting suddenly. Reinitialized the axes, issued the scanStripe again and the slew completed properly this time. Skippy was happy and we were in focus from the beginning of this run, although we were getting comparatively few stars in skippy, only around 10. Depending on which starting frame and for how many frames a skippy run was started, results weither suggested big THETAF errors or entirely reasonable errors, whose magnitudes were clearly related directly to the value of NSTARS. We think the trouble may have been due to bright moonlight interference and not getting enough catalog stars. I decided in fact that no correction was necessary in rotation and later on got much more reasonable numbers in skippy runs. Seeing was good throughout this run and the skies stayed clear til toward the end when we had some light cirrus that made our sky values jump around Frame 345 (lambda=162, RA=347). Began the ramp down at lambda=167.5 for 2.5 degrees (Frame 381, RA=352, 05:55Z) and ded the drift at lambda=170, RA=355, Frame 394, 06:04Z. We then moved to a point along the stripe 20 degrees east of the Moon's RA, or a total of 40 degrees from where science ended on the last run and picked up the remainder of the stripe. 06:10Z Run 5871 (SN Stripe 82 S, startLambda=-155). Skippy was happy from the get-go on this run and we were in focus from the beginning. Science started at lambda=-152.5, RA=33,5 Frame 21. The teamster died around Frame 14 and I had to force a restart, at which frames started being transferred again normally. Clouds interfered around Frame 35 as a rather puffly jet contrail passed over us, curiously parallel to Stripe 82 and at the right declination! Okay again by Frame 63 (lambda=-146). Seeing took a dive during this time and went very much out of survey spec, remaining poor for the rest of the drift. The ramp down began at lambda=-125 (Frame 204, RA=60, 08:06Z). We ramped for 2.5 degrees to end the drift at lambda=-122.5 (RA=62.5, Frame 221, 08:16Z). John and I decided to use the rest of the imaging time to do approximately 40 degrees' worth of 82 N, to use most effectively the time (~=2.6 hours) we skipped over in 82 S to avoid the moon. 08:21Z Run 5872 (SN Stripe 82 N, startLambda=-161). Another 2.5 degree ramp into science. Skippy was happy and we were in focus from the start of this run. Some light clouds passed over from the west around Frame 30. Those clouds persisted for most of thisdrift, and seeing only got worse since the run before, averaging somewhere around 3 arcsec with a stiff east wind fueling the fury. The ramp down began at lambda=-125 (RA=61, Frame 247, 10:46Z). The drift ended at lambda=-122.5, RA=62.5, Frame 263, 10:55Z. At the end of this drift, only about 75 minutes remained before the start of twilight. Seeing was still quite bad and the sky decidedly nonphotometric, but the moon had just recently set. John helped me to remove the imager from the telescope and we switched to spectroscopy in hopes of perhaps finishing one plate before the end of the night. 11:24Z Cart 6, Plate 2285 (tile 1651, chunk 122). I decided to maximize the dark sky on this plate and skip preCalibs in order to get right on the target. I made a reasonably good guess at focus, based on guider images, and dedided to similarly eschew a proper focus loop, opting instead to make manual adjustments once science exposures were underway. Given the poor seeing, this turned out to be a reasonable approach. The requested scale change was rather large (+0.0032) but I made the correction before starting science. Got two 900 s exposures on this plate but deciced to bail on a third because it would have ended almost 25 minutes past the start of twilight. Guiding stopped on frame 1301 for no evident reason: it simply stopped sending new gcam doread commands to the TCC. I also sent a small scale change command at this time and the command timed out, so I tried another 'startDa -tcc' to attempt a new connection to the TCC. This worked and I could guide again. I only got as far as QA on the biases while these exposures were going on, and missed the offsets and focus. Twilight began at 12:08Z, a few minutes into the second science exposure. After this exposure ended I took a postCalib and did an sos_redo to force another attempt at the reduction of the science. The low (S/N)^2 is 4.5, not bad considering how low the efficiency was during the observations. For this reason I am going to request that this plate remain plugged over the rather short interval until the next dark run. Closed up (thankfully) for the night at 12:35Z. endNight prompted me for an SSH passphrase on attempting to copy over the cloudcam images: Enter passphrase for RSA key '/home/observer/.ssh/identity': Enter passphrase for RSA key '/home/observer/.ssh/identity': Enter passphrase for RSA key '/home/observer/.ssh/identity': observer@hoggpt.apo.nmsu.edu's password: This password is evidently not the usual observer password and endNight complained that it couldn't copy ove the cloud cam images -- "Perhaps there were none tonight." I told it to continue anyway but this is an FYI. It otherwise rejoiced. Good night, and good luck. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 5868 00:31Z 01:07Z 100 O 129.14 138.12 74 ignore 5869 01:07Z 01:25Z 100 O 138.36 142.70 43 bias On telescope 5870 02:11Z 06:04Z 82 S 111.80 170.24 404 science 5871 06:09Z 08:17Z 82 S -154.50 -122.32 229 science 5872 08:23Z 10:56Z 82 N -160.75 -122.30 271 science ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 5870 9 4 5 -15.000 1.312 -3.000 1.567 0.00157 239 38 5870 62 4 85 3.500 0.172 -5.900 0.156 -0.01473 239 38 5870 116 4 130 3.500 0.142 -6.300 0.125 -0.01437 239 38 5870 144 4 109 3.500 0.169 -6.600 0.126 -0.01409 239 38 5870 162 4 83 3.700 0.162 -6.700 0.148 -0.01435 239 38 5870 168 4 93 3.500 0.156 -6.600 0.136 -0.01465 239 38 5870 220 4 70 4.300 0.169 -6.700 0.128 -0.01422 239 38 5870 272 4 64 4.100 0.174 -7.000 0.136 -0.01432 239 38 5870 287 4 67 4.100 0.181 -7.000 0.143 -0.01389 239 38 5870 289 4 60 3.900 0.188 -7.000 0.140 -0.01414 239 38 5870 324 4 43 3.900 0.155 -7.200 0.131 -0.01331 239 38 5870 377 4 44 3.700 0.151 -7.300 0.100 -0.01379 239 38 5871 11 4 49 3.400 0.128 -1.400 0.109 0.00125 195 56 5871 66 4 41 3.400 0.151 -0.900 0.119 0.00177 195 56 5871 116 4 49 2.600 0.168 -0.800 0.115 0.00153 195 56 5871 168 4 42 2.800 0.170 -0.500 0.147 0.00108 195 56 5871 217 4 25 2.900 0.142 -0.700 0.143 0.00203 195 56 5872 14 4 5 3.300 0.089 -5.900 0.052 -0.01836 244 34 5872 64 4 4 3.500 0.207 -6.200 0.118 -0.01836 244 34 5872 116 4 37 2.700 0.193 -6.800 0.163 -0.01869 244 34 5872 166 4 52 2.600 0.142 -7.100 0.141 -0.01850 244 34 5872 218 4 43 2.100 0.144 -7.500 0.123 -0.01869 244 34 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 5870 21 1 38 301 71 -0.965 0.173 0.715 0.216 -0.00240 5870 21 1 38 301 70 -0.659 0.196 0.756 0.182 -0.00254 5870 21 1 38 301 71 -0.275 0.239 0.988 0.227 -0.00332 5870 21 1 38 301 73 0.081 0.241 1.120 0.180 -0.00377 5870 21 1 38 301 75 0.409 0.293 1.283 0.251 -0.00431 5870 38 1 38 301 84 -1.441 0.150 1.107 0.221 -0.00372 5870 38 1 38 301 72 -1.122 0.142 1.272 0.281 -0.00428 5870 38 1 38 301 72 -0.751 0.189 1.201 0.268 -0.00404 5870 38 1 38 301 74 -0.398 0.175 1.354 0.149 -0.00455 5870 38 1 38 301 76 -0.005 0.224 1.434 0.232 -0.00482 5870 73 1 38 301 64 -1.468 0.161 1.293 0.328 -0.00435 5870 73 1 38 301 77 -1.078 0.226 1.416 0.324 -0.00476 5870 73 1 38 301 72 -0.777 0.173 1.372 0.244 -0.00461 5870 73 1 38 301 80 -0.324 0.152 1.666 0.200 -0.00560 5870 73 1 38 301 74 0.021 0.176 1.596 0.157 -0.00537 5870 90 1 38 301 66 -1.860 0.240 1.115 0.376 -0.00375 5870 90 1 38 301 77 -1.251 0.227 1.409 0.267 -0.00474 5870 90 1 38 301 75 -0.751 0.236 1.762 0.286 -0.00592 5870 90 1 38 301 70 -0.335 0.253 1.851 0.175 -0.00622 5870 90 1 38 301 68 -0.276 0.245 1.610 0.225 -0.00541 5870 125 1 38 301 75 -0.749 0.241 1.243 0.503 -0.00418 5870 125 1 38 301 63 -0.511 0.170 1.360 0.419 -0.00457 5870 125 1 38 301 76 0.058 0.221 1.454 0.452 -0.00489 5870 125 1 38 301 57 0.365 0.192 1.320 0.552 -0.00444 5870 125 1 38 301 68 0.791 0.266 1.534 0.590 -0.00516 5870 144 1 38 301 43 -1.358 0.145 0.917 0.156 -0.00309 5870 144 1 38 301 64 -1.095 0.173 1.100 0.125 -0.00370 5870 144 1 38 301 64 -0.497 0.205 1.387 0.144 -0.00466 5870 144 1 38 301 49 -0.146 0.142 1.673 0.130 -0.00562 5870 144 1 38 301 52 0.138 0.155 1.792 0.131 -0.00603 5870 178 1 38 301 42 -1.004 0.303 1.384 0.246 -0.00466 5870 178 1 38 301 39 -0.736 0.278 1.614 0.195 -0.00543 5870 178 1 38 301 41 -0.187 0.279 1.769 0.323 -0.00595 5870 178 1 38 301 36 0.118 0.255 1.939 0.248 -0.00652 5870 178 1 38 301 46 0.506 0.250 2.234 0.230 -0.00751 5870 196 1 38 301 23 -0.689 0.249 0.273 0.209 -0.00092 5870 196 1 38 301 35 -0.452 0.288 0.308 0.240 -0.00104 5870 196 1 38 301 33 0.055 0.371 0.713 0.316 -0.00240 5870 196 1 38 301 25 0.429 0.380 0.764 0.180 -0.00257 5870 196 1 38 301 33 0.932 0.325 1.036 0.276 -0.00348 5870 231 1 38 301 31 -1.343 0.433 0.560 0.387 -0.00188 5870 231 1 38 301 32 -0.980 0.448 0.924 0.384 -0.00311 5870 231 1 38 301 26 -0.401 0.458 0.952 0.411 -0.00320 5870 231 1 38 301 35 -0.214 0.304 1.206 0.391 -0.00405 5870 231 1 38 301 31 0.239 0.416 1.419 0.363 -0.00477 5870 248 1 38 301 35 -0.631 0.091 0.990 0.210 -0.00333 5870 248 1 38 301 47 -0.291 0.126 1.143 0.307 -0.00384 5870 248 1 38 301 31 0.183 0.087 1.535 0.267 -0.00516 5870 248 1 38 301 25 0.560 0.126 1.657 0.180 -0.00557 5870 248 1 38 301 18 1.039 0.139 1.628 0.230 -0.00547 5870 283 1 38 301 19 -1.295 0.299 0.898 0.412 -0.00302 5870 283 1 38 301 26 -1.046 0.315 1.198 0.498 -0.00403 5870 283 1 38 301 27 -0.465 0.332 1.464 0.411 -0.00492 5870 283 1 38 301 24 -0.070 0.188 1.564 0.365 -0.00526 5870 283 1 38 301 19 0.157 0.195 1.893 0.390 -0.00637 5870 301 1 38 301 25 -1.767 0.293 0.779 0.426 -0.00262 5870 301 1 38 301 15 -1.369 0.184 0.763 0.244 -0.00257 5870 301 1 38 301 21 -0.932 0.271 1.013 0.344 -0.00341 5870 301 1 38 301 20 -0.332 0.349 1.529 0.435 -0.00514 5870 301 1 38 301 21 0.158 0.299 1.513 0.317 -0.00509 5870 336 1 38 301 27 -1.224 0.099 0.597 0.346 -0.00201 5870 336 1 38 301 24 -0.835 0.082 0.919 0.270 -0.00309 5870 336 1 38 301 16 -0.376 0.098 1.367 0.156 -0.00460 5870 336 1 38 301 27 -0.012 0.117 1.345 0.307 -0.00452 5870 336 1 38 301 10 0.428 0.076 1.354 0.344 -0.00455 5870 353 1 38 301 27 -1.868 0.118 1.055 0.204 -0.00355 5870 353 1 38 301 16 -1.481 0.097 1.332 0.195 -0.00448 5870 353 1 38 301 24 -1.056 0.125 1.366 0.274 -0.00459 5870 353 1 38 301 10 -0.703 0.131 1.495 0.162 -0.00503 5870 353 1 38 301 15 -0.347 0.107 1.732 0.204 -0.00582 5870 386 1 38 301 16 -1.422 0.142 0.793 0.129 -0.00267 5870 386 1 38 301 22 -1.101 0.191 1.004 0.136 -0.00338 5870 386 1 38 301 12 -0.656 0.125 1.205 0.162 -0.00405 5870 386 1 38 301 22 -0.372 0.234 1.391 0.092 -0.00468 5870 386 1 38 301 15 0.252 0.199 1.844 0.150 -0.00620 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): y QA Procedures Done (y/n): bias only (see log) UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 34855, plate -9999 ------- 00:37 34855 0.0 bias ----- sequence 34856, plate 2285 ------- 6.8 4.5 8.0 7.9 NOT DONE 11:56 34857 900.1 target 12:13 34858 900.1 target 12:16 34859 10.0 flat 12:19 34860 2.0 arc 12:35 34861 0.0 bias 13:03 34862 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 03:38Z imager -59.36 0.0000 38.27 0.0000 43.43 0.0000 1.000000 06:57Z imager -15.10 0.0000 56.21 0.0000 77.16 0.0000 1.000000 09:18Z imager 295.72 0.0000 34.22 0.0000 40.57 0.0000 1.000000 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 03:46Z imager 1.00000 0 -277 -235 -59 38.3 5.7 11 0 -- 1.3 04:51Z imager 1.00000 0 -244 -202 -59 38.3 5.4 15 4 -- 1.5 06:57Z unknown 1.00000 0 -320 -217 -15 56.2 4.7 22 25 -- 1.7 09:18Z imager 1.00000 0 -415 -389 296 34.2 6.2 20 20 -- 2.8 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 00:24Z 41 15 11 5 (N) 234 - PtlyCldy 00:56Z 40 17 19 20 (NNE) 262 - MstlyClr 01:28Z 40 18 17 7 (N) 249 - MstlyClr 01:59Z 41 19 24 17 (NNE) 264 - MstlyClr 02:32Z 41 18 13 19 (NNE) 306 - MstlyClr 03:03Z 42 18 12 21 (NNE) 381 - Clear 03:33Z 42 18 11 352 (NNW) 362 - Clear 04:04Z 42 18 16 356 (NNW) 291 - Clear 04:35Z 41 18 17 19 (NNE) 332 - Clear 05:06Z 41 15 19 5 (N) 331 - Clear 05:37Z 41 18 13 14 (NNE) 357 - Clear 06:08Z 40 19 15 359 (NNW) 363 - MstlyClr 06:38Z 41 17 17 15 (NNE) 341 - PtlyCldy 07:09Z 40 18 22 24 (NNE) 352 - PtlyCldy 07:39Z 41 16 21 22 (NNE) 306 - PtlyCldy 08:09Z 41 13 14 15 (NNE) 281 - MstlyClr 08:40Z 42 9 14 10 (N) 221 - MstlyClr 09:10Z 44 3 12 23 (NNE) 223 - PtlyCldy 09:41Z 42 6 18 40 (NE) 238 - PtlyCldy 10:12Z 41 6 10 47 (NE) 217 - PtlyCldy 10:43Z 43 4 14 21 (NNE) 204 - PtlyCldy 11:13Z 43 0 12 81 (E) 186 - MstlyClr 11:44Z 42 2 18 51 (NE) 204 - MstlyClr 12:15Z 42 5 15 51 (NE) 219 - PtlyCldy 12:46Z 41 7 13 67 (ENE) 300 - PtlyCldy 13:16Z 41 9 15 77 (ENE) 249 - PtlyCldy ---=== TELESCOPE STATUS ===--- 00:30Z Doors open, fans and louvers on 01:00Z Doors closed for dark-dome bias 01:30Z Doors open, enclosure off 12:35Z Enclosure on, doors ad louvers closed, fans off Telescope parked at 121,30 with Cart 6 (Plate 2285) mounted Counterweights at 250 Spectro autofill reconnected No interlocks bypassed Dewar weights at 13:30Z Imager: 224lbs 19psi Spectro: 170lbs 25psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_6_0b Watcher: v2_30_2 MCP: v5_26_0 TPM: tpm_v3_1_0 AstroDa: v15_6 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_06 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.8380 -5.8440 Axial B -5.6820 -5.6770 Axial C 0.9370 0.9420 Trans D -9.9880 -9.9940 Lateral E 10.3759 10.3759 Lateral F 11.9507 11.9380 GALILS Commanded: 4700. -2450. 3000. -7800. 3300. 3000. Actual: 4711. -2457. 2997. -7825. 3296. 3024. SETMIR VALUES PriDesOrient: 0.00 -1.68 7.05 385.25 92.02 PriOrient: 0.00 -1.52 7.02 384.99 93.81 SECONDARY: ---------- Focus: 0.00 Air Temp.: 5.3 Alt.: 29.999997 MIGS TONIGHT NOMINAL Axial A 1.5390 1.5420 Axial B 1.0470 1.0470 Axial C 1.1690 1.1650 Trans D -0.1100 -0.1240 GALILS Commanded: 1593520. 1603214. 1554181. -6800. -5900. Actual: 1593447. 1603642. 1554316. -6820. -5882. SETMIR VALUES SecDesOrient: 1257.00 15.87 0.00 -50.00 136.76 SecOrient: 1256.87 15.82 -0.15 -49.65 136.77 ---=== PROBLEMS IN DETAIL ===--- Death Of A Server, A Play In Three Acts --------------------------------------- Scene: A mountaintop somewhere in southern New Mexico. Act 1 Olena called me at home around 22:00Z to ask for help regarding a problem with getting the murmur server to run. I logged in from home and found that not only was it not running, but all the other servers appeared to have died. The last message in the murmur log carried a timestamp of 21:34Z, so it was sometime after this that the server died. It likely corresponded with the time when Olena issues a cleanupMurmur as part of routine afternoon checkout procedures, but the server did not come back. This, in turn, prevented them from commanding the telescope or instruments and we decided not to proceed until I came into work and could assess the situation. I did try from home to restart the murmur server using all the methods I know of to no avail. Olena reported that the other servers also died instantly as soon as a start_servers command was issued. Act 2 After arriving at APO, we tried rebooting the crates in hopes that on the next cleanupMurmur the server would start. This didn't work. We called, in turn, Eric Neilsen, Margaret Votava and Fritz Stauffer but were only able to locate Eric. He went over the same procedures for bringing back the murmur server as we have outlined in Folklore, including checking the status of the semaphores (and whether there are any which need to be cleaned up). Rebooting host2 was discussed as a last resort, but first we agreed that calling Margaret and/or Kurt was a better idea. Kurt had me look at the list of all semaphores with an ipcs command, and we found all but one owned by observer (the other was owened by products). dscIpcrm, to clean up the semaphores, returned "observer has nothing to cleanup" -- Kurt says this is a bug. He had me try running $murmur_dir/bin/mur_ipcrm.csh which reported removing some of the dozens of semaphores owned by observer (but having a value of zero), although on another ipcs we found all the same semaphores still there. In other words, the cshell script didn't clean them up as it is supposed to. Kurt decided that the best thing to do would be to start cleaning up the semaphores by hand with the command "dscIpcrm -s (semid)"; it would be tedious but might solve the problem, if the root cause were that the murmur server couldn't start because there were no more avilable semaphores. That would, qualitatively, explain the behavior we were seeing. He logged in from home and began that task and promised to report back. Act 3 Moments later, Eric called to ask whether we wanted to try a reboot of host2, and that he was about to leave home for a while. I decided to go for it, in case that was the most straightforward solution, even though it's a little like taking a sledgehammer to the problem. I sent Kurt an email warning him of the reboot, and received an email from almost instantly that he had solved the problem. By that point it was too late -- host2 had gone down, but this wasn't a problem as it turned out (nor would it, on its own, have solved the problem). Kurt found a missling link in /astrolog that impeded the servers from starting, and until the corrupt link was fixed, they would never come up on their own. He corrected the problem and started the murmur server again, this time with success. Around that time host2 rebooted and he called us back. I started up an iop session and did a new cleanupMurmur, giving this output: im> cleanupMurmur Stopping murmur server /astrolog/astroda.v15_6.log already exists, deleteing it /astrolog/astroda.v15_6.log removed Restarting the murmur server murmur server restarted Trying to ensure the murmur log exists MURMUR MESSAGE <00000000> 00:16:29 Sun Nov 13 Arg# 1 (text only): "Trying to provoke murmur" Appname: "IDA 5000" The murmur log now exists Creating link to /astrolog/iop.murmur.log from /astrolog/astroda.v15_6.log Link created Cleaning semaphores observer has nothing to cleanup At that point, we were back in business and proceeded with the rest of checkout. Kurt sent this explanation breaking down the sequence of events: "When I logged into sdsshost2 as observer and ran 'ipcs', I found lots of services used by observer just like you did (screen capture below). "Looking in the /astrolog/murmur_internal.errors file, I found some messages like Sat Nov 12 22:17:01 2005, MURSRVR: fatal error in msgdb constructor Sat Nov 12 22:17:40 2005, Unable to create murmur semaphores, error: No space left on device "Even though the murmur error complained about semaphores, I wanted to clean up all ipc services. I used 'ipcrm -m ' to remove the shared memory services owned by observer (I had already cleaned up a couple of services owned by products). To remove the semaphores, I used 'ipcrm -s '. "At that point, I 'setup iop v4_6_0b' from the observer account. I started iop and tried 'cleanupMurmur'. This did not work and the complaint seemed to be that it couldn't find /astrolog/astroda.v15_6.log. I exited iop, ran 'dscRun start_mursrvr', and verified that this succeeded in starting the murmur server using 'ps -ef | grep mursrvr' (it did succeed). I stopped the murmur server using 'dscRun stop_mursrvr'. I again started iop and tried 'cleanupMurmur'. This time it succeeded in starting the murmur server. I exited and re-entered iop at this point to generate some murmur messages, and verified that the messages showed up in /astrolog/iop.murmur.log. "At that point, I received your email saying that sdsshost2 would be rebooted, so I logged off. "My sense is that some process is not cleaning up ipc services and at some point the available ones are used up. Unfortunately, "dscIpcrm" is broken in astroda v15_7 and earlier, so it is not much help in cleaning things up. As I mentioned, $MURMUR_DIR/bin/mur_ipcrm.csh may be useful until astroda v15_8 fixes dscIpcrm. Kurt" ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 23789624 observer 600 21344 1 dest 0x00000000 8224825 observer 600 21344 0 0x00000000 13795386 observer 600 21344 0 0x00000000 6488123 observer 600 21344 0 0x00000000 7667772 observer 600 21344 0 0x00000000 6979645 observer 600 21344 0 0x00000000 7012414 observer 600 21344 0 0x00000000 8126527 observer 600 21344 0 0x00000000 11206720 observer 600 21344 0 0x00000000 7798849 observer 600 21344 0 0x00000000 8323138 observer 600 21344 0 0x00054087 22315075 observer 666 512 0 0x00000000 8192068 observer 600 21344 0 0x00000000 21004357 observer 600 21344 0 0x000500aa 21069894 observer 666 512 0 0x00000000 21102663 observer 600 21344 0 0x00000000 10092616 observer 600 21344 0 0x00000000 20283467 observer 600 21344 0 0x00000000 11239500 observer 600 21344 0 0x00000000 11272269 observer 600 21344 0 0x00000000 11370574 observer 600 21344 0 0x00000000 11403343 observer 600 21344 0 0x00000000 8847440 observer 600 21344 0 0x00000000 11468881 observer 600 21344 0 0x00000000 11501650 observer 600 21344 0 0x00000000 13533267 observer 600 21344 0 0x00000000 12943444 observer 600 21344 0 0x00000000 11665493 observer 600 21344 0 0x00000000 12517462 observer 600 21344 0 0x00000000 12451927 observer 600 21344 0 0x00000000 12157016 observer 600 21344 0 0x00000000 11796569 observer 600 21344 0 0x00000000 11829338 observer 600 21344 0 0x00000000 12976219 observer 600 21344 0 0x00000000 13008988 observer 600 21344 0 0x00000000 15466589 observer 600 21344 0 0x00000000 12615774 observer 600 21344 0 0x00000000 12648543 observer 600 21344 0 0x00000000 20316256 observer 600 21344 0 0x00000000 12222561 observer 600 21344 0 0x00000000 12714082 observer 600 21344 0 0x00000000 13041763 observer 600 21344 0 0x00000000 13107300 observer 600 21344 0 0x00000000 13140069 observer 600 21344 0 0x00000000 13172838 observer 600 21344 0 0x00000000 13336679 observer 600 21344 0 0x00000000 13303912 observer 600 21344 0 0x00000000 13369449 observer 600 21344 0 0x00000000 13402218 observer 600 21344 0 0x00000000 13434987 observer 600 21344 0 0x00000000 13500524 observer 600 21344 0 0x00000000 13566061 observer 600 21344 0 0x00000000 13598830 observer 600 21344 0 0x00000000 15204463 observer 600 21344 0 0x00000000 13729904 observer 600 21344 0 0x00000000 13828209 observer 600 21344 0 0x00000000 13860978 observer 600 21344 0 0x00000000 13893747 observer 600 21344 0 0x00000000 13926516 observer 600 21344 0 0x00000000 13959285 observer 600 21344 0 0x00000000 13992054 observer 600 21344 0 0x00000000 14024823 observer 600 21344 0 0x00000000 14057592 observer 600 21344 0 0x00000000 14090361 observer 600 21344 0 0x00000000 14123130 observer 600 21344 0 0x00000000 14155899 observer 600 21344 0 0x00000000 14254204 observer 600 21344 0 ------ Semaphore Arrays -------- key semid owner perms nsems ------ Message Queues -------- key msqid owner perms used-bytes messages