APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday May 17, 2004 (MJD 53143) ---=== OBSERVING TEAM ===--- Swing: Mike Harvanek Night: Howard Brewington, Pete Newman Support: Craig Loomis, Connie Rockosi (UW, by email) Wendell Jordan (soon) ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Clouds kept us closed, then one plate finished (1643), then...they're baaack: ~40Hz structural vibration and ~4 Hz hunting in altitude at high altitude. We abandoned observations during instrument change at about 09Z when we detected major vibrations. See below for gory details. We have probably diagnosed the guider image delays. ---=== OBSERVING LOG ===--- Afternoon: ---------- Spectro collimation checked; no adjustment necessary. See Problems section for details. Night: ------ Overcast from sunset until 07:10Z (about 3 hours before morning twilight). Around 05Z, while slewing from zenith to 30 deg altitude after switching to cartridge 9 (inside the enclosure), the telescope aborted and Howard heard what sounded like a plastic-handled tool falling on the southeast side of the telescope. We searched around, under and inside the telescope and found nothing. When we tried moving the telescope again, the were no further problems. Craig Loomis may have found the underlying problem behind the guider delays. There are two causes. First, and generally only a few seconds, is when SoS rsyncs files from sdsshost, causing the local network to reach capacity. We have to live with that. Secondly, and lasting up to 50 seconds, is when the guider Macintosh determines that the modification time of the sdsshost:/data/spectro/guider/ directory differs from that expected. This causes the Macintosh to ask for the details of all the files in the directory. There are two things that can be done to mitigate this delay. First, at the time we tested, there were over 4000 files in the directory. However, more than half were completely redundant left-overs from earlier work. These were removed. Second, the guider code should be changed to put its calibration exposures in a directory other than /data/spectro/guider, which should only be written to by the guider Macintosh. This will avoid the Mac needing to scan the whole directory to determine what has changed. PR6044 filed, and Connie is picking it up. Until this gets fixed, expect the following pattern. The guider runs fine with just a couple of seconds read-out delay until you obtain a guider flat. That causes a file to be written into /data/spectro/guider, and that causes the next guider exposure to be delayed about 20 seconds. After that, all is back to normal. Note that during this testing, we took a LOT of bias exposures to keep SoS busy. We eventually opened at 07:10Z to thin patchy cloud for spectroscopy. Observing sequence: 07:23Z Plate 1643, cartridge 6. Main survey. Done in 3 exposures, 2950 sec. Guide stars on gotoField with prior offsets, but we lost them on the scale change. However, an FK5 was found just off the fibers, so only 3 minutes lost. During instrument change to cartridge 2 (which started at 08:47:39Z), Pete noticed first that the altitude axis started hunting at about 4 Hz as it approached zenith, then a structural vibration of ~40 Hz set it. Aborting the altitude axis by inserting the pin stopped both. However, the 40 Hz vibration started again as soon as the track command to slew to the next field was issued. That was quickly aborted with a stop button. Recognising the symptoms seen last winter (last reported in the 2004-03-13 engineering day log and earlier during 2003 October), we put the telescope away and will shortly be waking the on-call engineer (Wendell). We did not notice any problems during earlier cartridge changes tonight, but these involved slews in only altitude. A short while before this problem, TPM suddenly reported a lot of errors related to mirror temperatures and mirror servos. These are listed in the problems section at the end of this log. We have no idea if these two events are related. endNight is running as this log is being sent. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4638 23:13Z 23:29Z 100 O -66.55 -62.58 40 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): y QA Procedures Done (y/n): n UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 26752, plate -9999 ------- 23:06 26752 0.0 bias ----- sequence 26753, plate 1676 ------- 23:17 26753 10.0 flat collimation 23:19 26754 2.0 arc " 23:22 26755 2.1 arc " 23:25 26756 2.1 arc " ----- sequence 26757, plate -9999 ------- 01:10 26757 0.0 bias ----- sequence 26758, plate 1676 ------- 01:16 26758 0.0 bias Testing guider comms 02:22 26759 0.0 bias 02:28 26760 0.0 bias 02:40 26761 0.0 bias 02:42 26762 0.0 bias ----- sequence 26763, plate -9999 ------- 02:56 26763 0.0 bias ----- sequence 26764, plate 1676 ------- 03:02 26764 0.0 bias 03:04 26765 0.0 bias 03:06 26766 0.0 bias 03:08 26767 0.0 bias 03:10 26768 0.0 bias 03:12 26769 0.0 bias 03:14 26770 0.0 bias 03:16 26771 0.0 bias 03:19 26772 0.0 bias 03:21 26773 0.0 bias 03:23 26774 0.0 bias 03:25 26775 0.0 bias 03:27 26776 0.0 bias 03:29 26777 0.0 bias 03:31 26778 0.0 bias 03:33 26779 0.0 bias 03:35 26780 0.0 bias 03:38 26781 0.0 bias 03:40 26782 0.0 bias 03:42 26783 0.0 bias 03:44 26784 0.0 bias 03:46 26785 0.0 bias 03:48 26786 0.0 bias 03:50 26787 0.0 bias 03:52 26788 0.0 bias 04:38 26789 0.0 bias 04:40 26790 0.0 bias 04:42 26791 0.0 bias 04:44 26792 0.0 bias 04:46 26793 0.0 bias 04:49 26794 0.0 bias 04:51 26795 0.0 bias 04:53 26796 0.0 bias 04:55 26797 0.0 bias 04:57 26798 0.0 bias 04:59 26799 0.0 bias 05:01 26800 0.0 bias 05:04 26801 0.0 bias 05:06 26802 0.0 bias 05:08 26803 0.0 bias 05:10 26804 0.0 bias 05:12 26805 0.0 bias 05:14 26806 0.0 bias 05:17 26807 0.0 bias 05:19 26808 0.0 bias 05:21 26809 0.0 bias 05:23 26810 0.0 bias 05:25 26811 0.0 bias 05:27 26812 0.0 bias 05:29 26813 0.0 bias ----- sequence 26814, plate 1643 ------- 07:37 26814 10.0 flat 07:40 26815 2.0 arc 08:11 26816 1200.1 target 08:31 26817 1000.1 target 08:46 26818 750.1 target ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 07:52Z 6 1643 253.01 0.0006 61.14 0.0029 93.19 0.0242 1.000240 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL7543 Stays: JL7544 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 07:52Z 6 1643 1.00024 -2369 -763 50 253 61.2 14.0 14 277 BG38 1.1 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:54Z 67 20 17 228 (SW) 698 - Overcast 23:27Z 65 19 16 253 (WSW) 689 - " 01:04Z 61 14 13 265 (W) 883 - " 01:36Z 60 13 8 264 (W) 918 - " 02:08Z 59 12 10 265 (W) 964 - Mostly cloudy 02:40Z 59 12 14 271 (W) 1051 - " 03:10Z 59 12 13 268 (W) 1122 - " 03:41Z 59 12 12 268 (W) 1021 - " 04:11Z 58 11 11 270 (W) 995 - " 04:41Z 58 10 10 267 (W) 945 - " 05:11Z 58 10 9 265 (W) 898 - " 05:41Z 58 11 9 263 (W) 756 - " 06:12Z 57 10 13 285 (WNW) 976 - " 06:42Z 57 10 15 290 (WNW) 977 - " 07:13Z 57 10 11 267 (W) 927 - Thin cloud 07:45Z 57 9 9 272 (W) 999 - " 08:17Z 56 9 9 301 (WNW) 1034 - " 08:50Z 56 10 8 272 (W) 1189 - " 09:22Z 56 10 12 267 (W) 1354 - " 09:55Z 56 10 10 272 (W) 1338 - " ---=== TELESCOPE STATUS ===--- 01:28Z Doors open, fans on. 07:15Z Enclosure off. 09:00Z Enclosure on, shut down. No interlocks bypassed. Status at 10:23Z: Telescope stowed at: 30 deg altitude Instrument mounted: Cartridge 2 Counterweights at: 270 Autofill systems: On 180L LN2 dewar scales: Spectro 83 lb, 17 psi (filled about 07Z) Imager 205 lb, 24 psi ---=== SOFTWARE USED ===--- IOP/SOP: v3_137_0 Watcher: v2_29_0 MCP: v5_25_0 TPM: tpm_v2_42_0 AstroDa: v14_47 TCC: TCC 2.7.2 November 11 2003 sdssProcedures: v1_90 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5410 0.5430 Axial B 0.6210 0.6240 Axial C 1.2240 1.2260 Trans D -8.4490 -8.4440 Lateral E 10.8585 10.8712 Lateral F 10.8331 10.8458 GALILS Commanded: 4600. -2850. 850. -9250. 18400. 17850. Actual: 4600. -2866. 866. -9275. 18411. 17825. SETMIR VALUES PriDesOrient: 0.00 0.00 15.20 829.90 1.60 PriOrient: 0.00 0.00 14.91 830.49 3.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 19.1 Alt.: 29.999612 MIGS TONIGHT NOMINAL Axial A 1.5100 1.5080 Axial B 1.0980 1.0960 Axial C 1.1090 1.1070 Trans D -0.0020 0.0040 GALILS Commanded: 1628550. 1548191. 1574320. -3900. -8750. Actual: 1628641. 1548187. 1574504. -3899. -8726. SETMIR VALUES SecDesOrient: 1257.00 0.00 -25.00 0.00 146.27 SecOrient: 1256.93 0.05 -24.98 0.28 146.52 ---=== PROBLEMS IN DETAIL ===--- Spectro Collimation Check: -------------------------- inital sp1 Coll_motor_A 6088 Coll_motor_B 15474 Coll_motor_C 12186 inital sp2 Coll_motor_A 5151 Coll_motor_B 5147 Coll_motor_C 5153 rotator at 45 degrees mount im> caget tpm_TM_PMSSAvg tpm_TM_TelairAvg.VALA tpm_TM_PMSSAvg 13.58 tpm_TM_TelairAvg.VALA 14.0925 All cameras in focus so no adjustment necessary: COLLIMATE: Mean offset = -0.07 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Mean offset = -0.12 pix COLLIMATE: Camera b2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Mean offset = -0.13 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Mean offset = 0.14 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) Problem #1 - A flurry of TPM errors =================================== We got all these is a short space of time, and they did not repeat: m1 temp (Excessive M1 - PMSS Difference:Excessive Telescope - Ambient Difference:Excessive Radial Gradient:Excessive Front-Back Difference (Entire Mirror):Excessive Front-Back Difference (Left Half):Excessive Front-Back Difference (Right Half):Excessive Front-Back Difference (Lower Half):Excessive Front-Back Difference (Upper Half): 2004-05-18 08:30:15Z ) env temp (Excessive Lower - Ambient Difference:Excessive Return - Ambient Difference:Excessive Floor - Ambient Difference: 2004-05-18 08:30:15Z m1 mig (Communications Fault: 2004-05-18 08:30:15Z) tmicro (Disconnected Sensor in M2B7:Disconnected Sensor in M2B6:Disconnected Sensor in M2B5:Disconnected Sensor in M2B4:Disconnected Sensor in M2B3:Disconnected Sensor in M2B2:Disconnected Sensor in M2B1:Disconnected Sensor in M2B0:Disconnected Sensor in M1B7:Disconnected Sensor in M1B6:Disconnected Sensor in M1B5:Disconnected Sensor in M1B4:Disconnected Sensor in M1B3:Disconnected Sensor in M1B2:Disconnected Sensor in M1B1:Disconnected Sensor in M1B0:Disconnected Sensor in M0B7:Disconnected Sensor in M0B6:Disconnected Sensor in M0B5:Disconnected Sensor in M0B4:Disconnected Sensor in M0B3:Disconnected Sensor in M0B2:Disconnected Sensor in M0B1:Disconnected Sensor in M0B0:Communications Fault: 2004-05-18 08:30:15Z ) m2 galil (Communications Fault: 2004-05-18 08:30:15Z) m1 galil (Communications Fault: 2004-05-18 08:30:15Z) ca (Channel Access Failure: TPM Down?: 2004-05-18 08:30:15Z) m2 temp (Excessive Radial Gradient:Excessive Front-Back Differential (Entire Mirror):Excessive Front-Back Difference (Left Half):Excessive Front-Back Difference (Right Half):Excessive Front-Back Difference (Lower Half):Excessive Front-Back Difference (Upper Half): 2004-05-18 08:30:15Z ) m2 mig (Communications Fault: 2004-05-18 07:46:42Z)