APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday September 30, 2003 (MJD 52913) ---=== OBSERVING TEAM ===--- Swing: Mike Harvanek Night: Howard Brewington Pete Newman Support: French Leger (at the start AND the end of the night!) ---=== OBSERVING PLAN ===--- Science. ---=== OBSERVING SUMMARY ===--- Time lost to weird weather and more secondary mirror oscillations, but we still finished 3 plates (1089, 1425, 1472) and got signal on 2 more (1030 and perhaps 1250 - see log). Good news: the M2 oscillation is probably not the piezo actuators. Bad news: we don't know what it is. ---=== OBSERVING LOG ===--- Afternoon: After doing an initNight -tapeLoad there was an error message stating that there was no repsonse from id5 (see Problems section). An initNight -tapeRelabel was issued and this seemed to solve the problem. However, after the bias was started there were error messages stating that both of the id5 drives were being shutdown (see Problems section). The procedure for re-enabling two failed drives following auto-recovery was attempted but it failed as well. Finally, photo2 crate was rebooted and a ptvmeReset was issued. However, the drives for id4, id5 and id6 would not wake up. Another initNight -tapeRelabel was issued. This solved the problem but bias run 4160 was lost in the process. Counterweight 4 made a horrible screaching sound about halfway down. Night: A series of minor problems at the start, culminating in logTool crashing (terminal MONX1 ran out of memory?) so the following tale is from Pete's memory. On checking the azimuth fiducials, the azimuth brake status was "neither". We found the azimuth brake air line valve turned off, we swtiched it on and all was well. However, when we slewed in aizumth, we heard and felt virbration through the telescope. Although we found nothing obviously wrong, a subsequent discussion with French suggests this might be down to the windbaffle drive that was worked on today. However, see later: this MAY have been M2 vibrating! After opening and topping off the spectro dewars, we had all axes abort druing a test slew. Again, no physical cause was found, although we had seen some windbaffle touches but have not yet had time to determine if the timings are related. 2003-10-01 01:05:38Z sdssmcp ampMgt dsc_E_trcErr MGT: bad az state ABORT_EVENT: ID_ERROR_LIMIT 2003-10-01 01:05:39Z sdssmcp ampMgt dsc_E_trcErr MGT: bad alt state ABORT_EVENT: ID_ERROR_LIMIT 2003-10-01 01:05:39Z sdssmcp ampMgt dsc_E_trcErr MGT: bad rot state ABORT_EVENT: ID_ERROR_LIMIT Further slews seemed ok. But we then found M2's peizo actuators dancing their merry oscillatory dance. Slews to different atitudes stopped the oscillation for only a few seconds, so we unplugged the actuators from the piezo controller per folklore, and plan on not imaging tonight. French's crew will have another go at the existing piezo servo board tomorrow. And then logTool crashed (actually I think it was the X server on MONX1 that had a problem). But now I'm all caught up. Moon up after twilight, so we started with spectroscopy in any case: Observing sequence: 01:48Z Plate 1425, cartridge 3. Done in three exposures, 2940 sec. FK5 star found easily, seeing very good. 03:17Z Plate 1030, cartridge 2. Low-z program. One good exposure. FK5 needed to find the field, then a large (0.03) scale change lost the guide stars, so the FK5 was found (easily) again. The large scale change was mystifying. This plate was drilled for T=13.3C, and the previous plate for T=13.0C. We do not that the pattern of guide stars on this plate is somewhat unusual, which the stars mostly along the noth and south edges of the plate. 04:05Z We then closed as Holloman AFB radar showed precipitation approaching rapidly from the NE. As it transpired, the radar was being fooled by a cold front not real precipitation. However, when the cold front arrived (bringing east wind and poor seeing as reported by thte 3.5m observer) the humidity started a quite rapid rise, so we elected to stay closed. The dewpoint depression dropped below the 3K closure limit at 05:50Z. It eventually rose again. 07:12Z Plate 1089, cartridge 1. Done in 3 exposures, 3220 sec. Guide stars on gotoField, but lost them after the initial scale change. Is that related to having the secondary partially disabled? 08:49Z Plate 1472, cartridge 7. Done in 3 exposures, 5100 sec, when combined with data from MJD 52903. FSTAR program. 10:48Z Plate 1250, cartridge 8. Taurus program. During a guided science exposure at about 11:28Z, we lost the guide stars for no apparent reason. Altitude was about 78deg and decreasing, all axes were running and there were no excessive fiducial errors. We go to an FK5 start: Argh! Secondary oscillation! See problem section for details. We read out the one exposure we had made, but SoS did not reduce r1, so we don't know if there is useful signal. Could someone check and let us know so we can update the database, please? That saw the end of darkness. At least endNight rejoiced! ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4160 23:38Z 23:52Z 100 O 73.09 76.65 38 bias data lost - see log 4161 00:20Z 00:30Z 100 O 83.65 86.00 30 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): Yes QA Procedures Done (y/n): Yes UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 22476, plate -9999 ------- 23:28 22476 0.0 bias ----- sequence 22477, plate 1425 ------- 17.5 17.2 20.4 20.7 DONE 23:37 22477 0.0 bias 02:14 22480 720.1 target 02:42 22481 1100.1 target 03:05 22482 1120.1 target 03:12 22483 240.0 smear ----- sequence 22484, plate 1030 ------- 15.1 10.5 12.5 11.1 Not done 03:39 22484 10.0 flat 03:42 22485 2.0 arc 04:10 22486 1500.1 target ----- sequence 22487, plate 1089 ------- 20.2 16.5 24.0 22.8 DONE 07:28 22487 10.0 flat 07:31 22488 2.0 arc 07:48 22489 720.1 target 08:14 22490 1300.1 target 08:37 22491 1200.1 target 08:44 22492 240.0 smear ----- sequence 22493, plate 1472 ------- 32.4 39.8 27.7 54.7 Tonight 08:59 22493 10.0 flat 4.0 7.7 3.7 9.1 MJD 52903 09:01 22494 2.0 arc ---- ---- ---- ---- 09:31 22495 1500.1 target 36.4 47.5 31.4 63.8 DONE 10:04 22496 1800.1 target 10:37 22497 1800.1 target 10:44 22498 240.0 smear ----- sequence 22499, plate 1250 ------- 11:05 22499 10.0 flat 11:08 22500 2.0 arc 11:38 22501 785.3 target Read out early, lost guide stars near end. Sos failed to reduce all of this exposure, we we do not know if there is useful S/N on it. ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 02:46Z Cart 3 -87.26 0.0010 44.64 0.0049 116.10 -0.0016 1.000200 04:02Z Cart 2 3.06 0.0016 57.12 0.0006 182.51 0.0190 0.999860 07:41Z Cart 1 -27.19 0.0037 54.08 0.0025 157.33 -0.0066 1.000230 09:07Z Cart 7 225.72 -0.0031 67.18 0.0028 61.08 -0.0168 1.000100 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL6818 Stays: JL6819 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 02:46Z Cart 3 1.00020 -1974 -436 0 -87 44.7 15.4 9 123 BG38 1.3 04:02Z Cart 2 0.99986 1381 2304 0 3 57.1 15.2 16 129 BG38 1.6 07:41Z Cart 1 1.00023 -2270 -762 -50 -27 54.1 14.0 16 145 BG38 1.7 09:07Z Cart 7 1.00010 -988 269 -50 226 67.2 14.1 15 177 BG38 1.7 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:15Z 68 28 5 274 (W) 334 - Party cloudy 23:49Z 66 23 5 291 (WNW) 299 - " 00:20Z 65 26 10 285 (WNW) 307 - Clear 02:57Z 59 35 8 132 (SE) 390 - " 03:28Z 58 35 17 127 (SE) 431 - " 03:59Z 59 33 13 128 (SE) 404 - " 04:30Z 56 33 18 88 (E) 418 - " 05:02Z 51 39 25 75 (ENE) 658 - " 05:35Z 49 43 19 84 (E) 1149 - " 06:07Z 48 43 17 80 (E) 1115 - " 06:40Z 49 41 14 122 (ESE) 1052 - " 07:11Z 55 37 17 136 (SE) 531 - " 07:41Z 57 36 15 145 (SE) 437 - " 08:12Z 56 37 14 150 (SSE) 487 - " 08:45Z 57 33 14 175 (S) 475 - " 09:17Z 56 30 13 175 (S) 441 - " 09:48Z 54 36 15 184 (S) 516 - " 10:21Z 55 36 12 180 (S) 506 - " 10:54Z 55 36 13 178 (S) 465 - " 11:25Z 53 36 11 174 (S) 453 - " 11:59Z 54 35 9 177 (S) 521 - " 12:31Z 52 36 9 179 (S) 516 - " 13:04Z 52 36 4 149 (SSE) 536 - " ---=== TELESCOPE STATUS ===--- 01:06Z Enclosure doors opened, fans and blowers on, louvers opened. 01:06Z Enclosure off. 04:05Z Enclosure on - precip on radar 05:50Z Doors closed, fans off - humidity over limits. 07:10Z Doors open, fans on, enclosure off. 13:00Z Enclosure on, shutdown. No interlocks bypassed. Status at 11:43Z: Telescope stowed at: 30 deg Instrument mounted: Imager Counterweights at: 270 Autofill systems: On 180L LN2 dewar scales: Spectro 35 lb, 20 psi (day crew are here!) Imager 93 lb, 15 psi ---=== SOFTWARE USED ===--- IOP/SOP: v3_119_0 Watcher: v2_24_0 MCP: v5_20_0 TPM: tpm_v2_32_0 AstroDa: v14_47 TCC: 2.7.1 sdssProcedures: v1_74 tccMon: v2_2 SoS: v4_9_13 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5470 0.5460 Axial B 0.6340 0.6290 Axial C 1.0910 1.0900 Trans D -8.5740 -8.5750 Lateral E 10.8839 10.9093 Lateral F 10.8585 10.8712 GALILS Commanded: 4600. -2850. 850. -7250. 19000. 18450. Actual: 4600. -2866. 866. -7275. 19011. 18425. SETMIR VALUES PriDesOrient: 0.00 0.00 15.20 829.90 1.60 PriOrient: 0.00 0.00 14.91 830.49 3.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 20.3 Alt.: 30.023598 MIGS TONIGHT NOMINAL Axial A 1.5140 1.5100 Axial B 1.1060 1.0990 Axial C 1.1140 1.1160 Trans D -0.8360 -0.8210 GALILS Commanded: 1628650. 1548115. 1574566. -3900. -8750. Actual: 1628641. 1548187. 1574504. -3899. -8726. SETMIR VALUES SecDesOrient: 1257.00 0.00 -25.00 0.00 146.27 SecOrient: 1257.00 -0.04 -25.02 0.25 146.58 ---=== PROBLEMS IN DETAIL ===--- id5 no response --------------- 2003-09-30 23:30:12Z: tapeWaitFor reports success Waiting for command response from id5 (crate 2, ICC 1), archiver server (client 2) 2003-09-30 23:30:49Z: IOP Not ready (blew up on pad): tapeMount: Request sense failure Failed to receive exabyte command completion status invoked from within "iccExec $tapeList archReadSense 1" invoked from within "set sense1 [iccExec $tapeList archReadSense 1]..." (procedure "tapeRewind" line 5) invoked from within "tapeRewind $tapeList " invoked from within "if {$doRewind} { tapeRewind $tapeList }" (procedure "tapeLoad" line 5) invoked from within "tapeLoad $tapeList $doRewind $doUnload" id5 shutdown ------------ dsc E archShutDown2 Archiver shutting down second drive (dsc_E_archShutDown2 Archiver shutting down second drive : 2003-09-30 23:41:43Z) archEnable (archEnable is 0: 2003-09-30 23:41:53Z) dsc E archWrFail Tape write error while archiving frame 4160 11.55 (dsc_E_archWrFail Tape write error while archiving frame 4160 11.55 : 2003-09-30 23:41:43Z ) archDrvSts (drive 2: Error: 2003-09-30 23:41:53Z) dsc E archStsFail Failed to receive exabyte command completion status (dsc_E_archStsFail Failed to receive exabyte command completion status : 2003-09-30 23:44:23Z ) dsc E archShutDown1 Archiver shutting down first drive (dsc_E_archShutDown1 Archiver shutting down first drive : 2003-09-30 23:41:43Z) dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 1 (dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 1 : 2003-09-30 23:41:43Z ) dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 2 (dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 2 : 2003-09-30 23:41:43Z ) archUseDrv (drive 2: archUseDrv is 0: 2003-09-30 23:41:53Z) Secondary oscillation --------------------- During observations of plate 1250, around 05:20Z, we lost guide stars. When we eventually went to an FK5 star, we saw an ellipse. We went to the telescope (left tracking), turned off the fans, and realised there was a vibration so strong that you could hear it (many hertz) and FEEL it with a hand resting on the altitude drive quadrant! Since we thought we had disconnected the piezo actuators, we went and woke French. He checked the piezos were indeed disconnected. We found that slewing the telescope back below 45deg stopped the oscillation. After several tests, tracking the Galil position through TPM plots, we found that slewing from non-oscillation ONLY in altitude up to 65 deg did not start the oscillation, but as soon as we then slewed in azimuth, it started. This was the end of the night, so we have handed the problem directly to French and crew who are working on it now.