APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday March 26, 2005 (MJD 53456) ---=== OBSERVING TEAM ===--- Night: John Barentine Jon Brinkmann (on-site engineering support) The ET Team: Jian Ge (UF) Julian Van Eyken (UF) Bo Zhao (UF) The Phone Support Team: Dan Long Mark Klaene Carlos Gonzales ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- No on-sky operations tonight because of a critical failure of the rolling enclosure electrical system (see Problems section). ---=== OBSERVING LOG ===--- Afternoon --------- No problems during checkout. Night ----- Lost the entire night due to an enclosure failure (see Problems). endNight was run on the cloudy night tapes and rejoiced. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): n/a QA Procedures Done (y/n): n/a UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 31498, plate -9999 ------- 23:29 31498 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:21Z 23 23 18 25 (NNE) 424 - MstlyCldy 23:53Z 27 20 19 4 (N) 89 - MstlyCldy 00:25Z 26 20 1 24 (NNE) 76 - MstlyCldy 00:57Z 25 23 7 15 (NNE) 69 - MstlyCldy 01:30Z 25 18 19 4 (N) 64 - MstlyCldy 02:02Z 24 16 16 6 (N) 53 - PtlyCldy 02:34Z 25 14 11 346 (NNW) 67 - MstlyClr 03:06Z 24 13 14 14 (NNE) 64 - MstlyClr 03:39Z 23 14 15 4 (N) 45 - Clear 04:11Z 23 14 14 4 (N) 42 - Clear 04:43Z 23 13 17 32 (NNE) 46 - Clear 05:15Z 24 11 14 353 (NNW) 34 - Clear 05:47Z 24 10 11 342 (NNW) 48 - Clear 06:19Z 24 10 19 8 (N) 56 - Clear 06:52Z 24 11 23 20 (NNE) 66 - Clear 07:28Z 24 12 24 23 (NNE) 90 - Clear 08:00Z 24 12 20 21 (NNE) 98 - Clear 08:32Z 24 13 16 23 (NNE) 69 - Clear 09:05Z 24 14 22 21 (NNE) 59 - Clear 09:37Z 24 18 19 12 (NNE) 85 - Clear 10:09Z 25 17 19 30 (NNE) 95 - Clear 10:41Z 25 17 30 21 (NNE) 88 - Clear 11:13Z 25 15 18 29 (NNE) 88 - Clear 11:46Z 24 17 16 20 (NNE) 81 - Clear ---=== TELESCOPE STATUS ===--- 02:30Z Fans on, doors open 06:15Z Enclosure on 07:00Z Fans off, doors and louvers closed Telescope parked at 121,30 with Cart 6 (Plate 8006) mounted Spectro autofill reconnected Counterwights at 297 No interlocks bypassed Enclosure is currently inoperable Dewar weights at 12:00Z Imager: 219lbs 22psi Spectro: 305lbs 24psi ---=== SOFTWARE USED ===--- IOP/SOP: v3_143_0a Watcher: v2_30_0 MCP: v5_26_0 TPM: tpm_v2_47_0 AstroDa: v14_47 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v1_98 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5810 0.5770 Axial B 0.5920 0.5880 Axial C 0.9430 0.9460 Trans D -8.1950 -8.1960 Lateral E 9.9314 10.0330 Lateral F 11.5189 11.5951 GALILS Commanded: 4300. -1900. 2850. -13300. -7600. -7650. Actual: 4300. -1900. 2850. -13300. -7600. -7650. SETMIR VALUES PriDesOrient: 0.00 4.57 1.18 64.25 -249.76 PriOrient: 0.00 4.56 0.88 65.00 -249.58 SECONDARY: ---------- Focus: 0.00 Air Temp.: -3.9 Alt.: 29.638648 MIGS TONIGHT NOMINAL Axial A 1.5550 1.5680 Axial B 1.0280 1.0460 Axial C 1.1310 1.1590 Trans D -0.1880 -0.2020 GALILS Commanded: 1643372. 1632191. 1564247. -13800. -9950. Actual: 1619200. 1604950. 1528550. -13800. -9950. SETMIR VALUES SecDesOrient: 1257.00 27.97 -10.00 -90.00 257.89 SecOrient: 1280.04 24.71 -8.41 -88.93 259.79 ---=== PROBLEMS IN DETAIL ===--- Rolling Enclosure Failure At Opening ------------------------------------ On opening toward the beginning of the night, 03:15Z, we had an almost immediate and decidedly critical error involving the rolling enclosure. The enclosure moved about five feet back from its stow position and immediately came to a jarring halt. An audible alarm sounded from the south wall of the enclosure. I tried actuating the enclosure again but it did not move; I tried moving it the other direction (i.e., back to stow) with the same results. The alarm was coming from the surge suppressor on the south wall, which also showed a red LED lit next to the word "FAULT". I checked the rails (again) to look for obstructions but found none. At that point I started making phone calls. Dan suggested an electrical problem stemming from an ice buildup on the outside rails feeding electricity to the rolling enclosure. The symptom to check for is whether the enclosure itself (e.g., overhead lights) had power -- if not, it would suggest the ice failure mode. Indeed, on returning to the enclosure, there was no power to the overhead lights. Also, the outside ODH sensor was buzzing every few seconds, while not blinking, possibly related to the power failure. I called Mark from the telescope to see if he could offer any advice on the electrical aspect of the problem. He suggested checking the status of the rail heating circuit on the east wall of the lower enclosure; the light was on and the switch was set to "AUTO". He also had me check each of the circuit breaker boxes -- upstairs and downstairs -- to look for tripped breakers. None found. He asked me to check the status of the enclosure motor controller cabinet on the north wall. The LED display read a flashing letter "E" and the green LED by the "PANEL CONTROL" label was blinking. Throughout all this, the LED indicating that the enclosure was enabled to move remained on, as it is powered on a separate circuit on the lower enclosure level. I checked the overhead lights again and this time found them powered, but the enclosure didn't move when commanded. Admitting the limit of his knowledge, Mark handed me off to the SDSS staff again. I reached Jon Brinkmann by phone, who in turn called Carlos for advice (French was not reachable). Carlos didn't know the meaning of the panel display but suggested tripped relays inside the cabinet. Jon had me open the cabinet and reset those relays, without luck. It sounded as though the enclosure motors were energizing when I turned the key in the pendant, but the enclosure staunchly refused to move. He also had me check the status lights on the surge suppressor upstairs to make sure there was power to all three phases of the circuit (there was). He had me power cycle the enclosure motor cabinet a couple of times; at first, the cycle returned with an "OH" display indicating overheating motors, but the normal "0.0" display was returned on the next power cycle, although the enclosure still didn't move. As a last resort, he had me check the status of the downstairs relays that switch between utility and generator power; both were firmly in the "COMMERCIAL" position. Jon drove in a short while later to attack the problem from the ground. He checked the enclosure rails for obstructions or ice, finding none. He switched on a utility light next to the enclosure motor cabinet and it glowed only dimly, suggesting the leg of the circuit powering the enclosure motors was underpowered. It occasionally flickered on brighter, and when it flickered I heard a sparking noise off the south end of the enclosure near the position of the sliding contacts on the outside rail, suggesting the contacts were arcing across the ice we presumed was there. Banging the guardrail by hand caused the lights to flicker inside and the sparking noise to be heard. We determined at that point ice was most likely to blame. Jon checked things downstairs and tried cycling the breaker for the rail heating circuit; despite the fact that the light indicated the heaters were on, he heard what sounded like a relay click when he turned the breaker back on. More about that momentarily. When the lights flickered to full power, Jon tried moving the enclosure and it worked; he was able to return the enclosure to the stow position. We decided to try moving the enclosure back again, but at the same distance from the end of the rails as before it stopped. We were able to get it back to fully closed and stopped there, such that we would be able to at least close for the night if we could not solve the problem. Jon remembered that there was a long (20'-plus) boat hook in the shop we might use to either knock the ice off the rails or maybe lash a heat gun to and use to warm the rails manually. The other possibility was that perhaps the rail heat, once it came back on (assuming it had been off earlier) might produce enough power to melt the ice. We decided that these avenues were unlikely to yield success, as ice might just reform later on in the night, possibly preventing us from closing -- even if we did manage to get the enclosure pulled all the way back. The low air temperature (-4 to -5C) meant refreezing of the ice was likely. With the enclosure safely on, and in consulatation with the ET Team, we decided to declare the telescope closed for the night, in hopes that predicted clear weather tomorrow will melt the ice.