APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday August 20, 2004 (MJD 53238) ---=== OBSERVING TEAM ===--- Swing: Jurek Krzesinski Support: Wendel Jordan, Carlos Gonzales, Steve Bastian Night Howard Brewington, Pete Newman ---=== OBSERVING PLAN ===--- Science. ---=== OBSERVING SUMMARY ===--- Important notes: During afternoon checkout, Jurek noted that water vapor condenses on r2 dewar, then drops on sp2 power supplies boards/wires (very close to 15 pin plugs) when the telescope is parked at stow positon with rotator at 45 deg. We still have the imager dewar 1 bias problem (PR 6128, critical high). We seemed to have a primary mirror Galil controller failure. Despite the above and waves of clouds, we finished three plates (1144, 1891, 1892) and got good data on two more. We had some trouble with a dirty tape drive during endNight, so there _may_ be multiple copies of data on the tapes. ---=== OBSERVING LOG ===--- Afternon: --------- This afternoon we (Carlos, Wendel, Jurek) took some HgCd lamp arc exposures monitoring at the same time voltages and the current on one HgCd lamp (#2). Nothing unusual was found. There were only some voltage variations on the first HgCd arc exposure when the lamp was cold (frame: 28169). I looked at raw frames and there is nothing unusual in them. 28169 and following HgCd arc exposures were alike. Carlos and Wendel suggested that they can monitor HgCd lamp votage behaviour in low temperatures, simulating real observations (meaning leaving enough time for lamps to cool down) on a testing banch. At the moment it doesn't look like we have any HgCd power supply problem and what we see (see Dan's e-mail with HgCd lamps intensity change plots) is connected only with internal lamp characteristics. Anyway, I would like to thank Wendell, Carlos and Steve for helping me with those tests. Telescope -ok. Spectrographs in focus - ok., MIGs - M1 E,F axes had to be homed. DA - problem with id3-1 -> "couldn't check tape", I took the tape out and put it back, then did tapeRelabel - it solved the problem, imager -ok. Guider: dark collected inside enclosure t=13.1 C (guiderDark 30 10), however I suggest, that it will be collected again after you guys open the telescope, so the dark will match better the ambient temperature (it takes about 6 minutes to complete). We tried: at 12:20Z, when Tamb = 11.6 C, but guiderDark kept failing, saying "Error: Timed out waiting for arrival of dark gcam from TCC" after the first couple of exposures. We re-set the TCC connection from the SOP session, and guiderDark worked onthe third attempt. However, the file in /data/spectroguiderDarks was still the old one created by Jurek. So, we moved that out of the way and tried a fourth time. This failed as above, but we reset the TCC connection again, and the fifth time worked. Night: ------ Shortly after we arrived at 00:30Z, ID3 became unhappy; the watcher messages are in the Problems Section. Nevertheless, we cleaned the tape drives, exchanged the tapes, rebooted the Photo 01 crate, did an initNight -tapeRelabel, and goDrift -bias, which solved the problem. The last frame of the doghouse bias, run 4784, frame 36, showed strong horizontal banding on the control room monitors across all CCDs in dewar 1, particularly strong in the g chip. We ran another doghouse bias, run 4785, for an extended period and did not notice a recurrence until frame 485 (though there may have been some others). We copied the frames to phoenix.apo.nmsu.edu:/data/photo/53238/ and updated the critical/high PR 6128 for this problem. Note that 4785/485/g is particularly nasty. Shortly before sunset, we had a spike in the 1u dust counts over 14000, which seemed to subside within half an hour. Given the primary is going to be realuminized nexxt week, we decided to press on anyway and ventilate the enclosure. However, local cloud also kept use closed until 03:43Z when it became clear enough to try spectroscopy under a partly cloudy and moonlit sky. We had to stop and restart the WinVNC server on the DIMM computer. Observing sequence: Guide stars acquired on gotoField unless noted otherwise. 04:13Z Plate 1684, cartridge 1. Main SDSS survey. Not done in one exposure before this plate's design time expired. 04:52Z Plate 1891, cartridge 2. Done in 4 exposures, 4300 s (plus one bad exposure) - see below. SEGUE faint plate requiring (S/N)^2 >= 30. Despite the guider dark frames taken this afternoon, we had to disable three guide fibres(2, 6, 11) which were initially tracking hot pixels. Increasing the guiding exposure time from 10s to 15s did not improve matters. We will take another set of guiderDarks later, but note that the current ambient temperature (Tamb=13.4C) is actually slightly above the enclosure temperature this afternoon. 06:38Z Plate 1892, cartridge 7. Done in 3 exposures, 2340s. SEGUE bright plate requiring (S/N)^2_blue >= 15. During this plate, we got the following error in the watcher: Modu="mir o MoveOneGalil" (2004-08-21 06:53:14Z) When we checked the TPM M1/M2 Galil Axes display, all the M1 actual an commanded numbers were zero! The TCC reported: mir stat 0 14 I PrimDesOrient=-2862.62, -3.60, 26.90, 1470.40, 198.00; PrimDesOrientAge=54.8 0 0 W Expected="XQ #STATUS" 0 0 W Received="E=40374.; F=39338.; MG "OK"" 0 0 F Modu="prt_ReadMatch"; Text="bad reply" 0 14 I SecDesOrient=-1287.08, 0.00, -25.00, 0.00, -57.67; SecDesOrientAge=54.8 0 14 I SecCmdMount=-1573050., -1653500., -1627200., 4550., -350. 0 14 I SecActMount=-1572556., -1653287., -1626339., 4550., -350. 0 14 I SecOrient=-1283.83, -0.19, -25.05, -0.13, -45.56 0 14 I SecStatus=1, 1, 1, 1, 1 0 14 I Received=" 0 piezo status word" 0 14 I Received="-000001676, -000001676, -000001676 piezo corrections (microsteps)" 0 14 W Text="failed for one or more mirrors: Prim" 0 14 : Cmd="mir stat" We halted and read out the on-going exposure (which was in any case bad because of clouds), then homed and relaxed the primary mirror (we happened to be observing near the meridian at alt=69, so left the telescope tracking the field). That seemed to restore normal mirror positions, but we then had no guide star. However, an FK5 was found just off the fibre, and we took a midCalibs (postCalibs) as a precaution. We lost a total of about 40 minutes to this problem. We think this is the third time this has happened this run - what is going on? We opened a PR against the Galils. SoS reported scattered light warnings for the r cameras for this plate. 08:36Z Plate 1144, cartridge 9. Done in 1 exposure, 1800s, when combined with exposures from other nights. Rockosi program plate requiring standard S/N. We had clouds roll through and seeing degrade during this plate. We declared this plate done despite the S/N on r1 being a bit low. 10:02Z Plate 1887, cartridge 6. Not done in 3 exposures, 3600s. SEGUE faint plate (see above). Waves of clouds passed through. endNight failed at the first attempt, reporting a problem with /astrolog/53238/ioptape.archlog. That file noted I/O errors on a tape drive, which was found to be calling for a cleaning tape. We cleaned the drive and re-ran endNight. It failed again in the same way. We cleaned the second drive again and replaced both tapes and tried again, remembering after endNight had started that the errors already in the ioptape.archlog file would cause problems...so we edited it in the fly to remove evidence of the earlier failures. endNight seemed to complete, but with a not on seeing the errors in the ioptape.archlog file, so maybe the edit was too late. In any case, the archlog file does indicate everything got written to tape. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4782 22:38Z 22:44Z 100 O 18.30 19.74 20 ignore Doghouse 4783 22:54Z 23:10Z 100 O 22.27 26.45 42 ignore Doghouse 4784 01:33Z 01:51Z 100 O 62.30 66.83 44 ignore Dewar 1 problems 4785 01:58Z 08:11Z 100 O 68.42 162.01 639 ignore Dewar 1 problsms ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): y QA Procedures Done (y/n): y UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 28165, plate -9999 ------- 20:44 28165 0.0 bias ----- sequence 28166, plate 1684 ------- Tests - ignore 20:50 28166 0.0 bias 20:53 28167 10.0 flat 20:56 28168 2.0 arc 21:08 28169 2.0 arc 21:11 28170 2.0 arc 21:14 28171 2.0 arc 21:17 28172 2.0 arc 21:20 28173 2.0 arc 22:12 28174 0.0 bias 22:15 28175 10.0 flat 22:17 28176 2.0 arc 22:23 28177 2.1 arc 22:26 28178 2.1 arc ----- sequence 28179, plate -9999 ------- 00:52 28179 0.0 bias 00:56 28180 0.0 bias ----- sequence 28181, plate 1684 ------- 8.9 6.1 8.0 7.6 Not done 04:26 28181 10.0 flat 04:28 28182 2.0 arc 04:47 28183 900.1 target ----- sequence 28184, plate 1891 ------- 44.3 31.0 58.8 47.4 DONE 05:01 28184 10.0 flat 05:04 28185 2.0 arc 05:23 28186 900.1 target 05:46 28187 1200.1 target 06:09 28188 1200.1 target 06:29 28189 1000.1 target 06:32 28190 10.0 flat 06:34 28191 2.0 arc ----- sequence 28192, plate 1892 ------- 15.8 17.8 24.3 24.2 DONE 06:47 28192 10.0 flat 06:49 28193 2.0 arc 07:10 28194 806.4 target 07:39 28195 10.0 flat 07:41 28196 2.0 arc 08:01 28197 900.1 target 08:14 28198 600.1 target 08:32 28199 840.1 target ----- sequence 28200, plate 1144 ------- 13.8 2.7 18.8 12.9 DONE 08:48 28200 10.0 flat 08:50 28201 2.0 arc 09:25 28202 1800.1 target 09:58 28203 1800.1 target ----- sequence 28204, plate 1887 ------- 18.4 14.8 19.0 17.9 Not done. 10:10 28204 10.0 flat 10:13 28205 2.0 arc 10:37 28206 1200.1 target 11:00 28207 1200.1 target 11:23 28208 1200.1 target 11:26 28209 10.0 flat 11:29 28210 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 04:38Z 1 1684 -103.25 0.0025 50.05 0.0013 103.59 -0.0255 1.000360 05:24Z 2 1891 33.12 0.0031 64.31 0.0022 207.78 0.0225 1.000360 08:02Z 7 1892 -12.98 -0.0029 70.46 0.0021 168.70 0.0043 1.000320 10:59Z 6 1887 232.89 -0.0017 60.63 0.0013 75.13 -0.0254 1.000210 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL7715 Stays: JL7716 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 04:37Z 1 1684 1.00036 -3554 -1837 -75 -103 50.1 13.9 9 226 BG38 1.2 05:24Z 2 1891 1.00036 -3554 -1850 -50 33 64.3 13.7 10 237 BG38 1.2 08:02Z 7 1892 1.00032 -3159 -1535 -50 -13 70.5 12.4 10 204 BG38 1.2 10:59Z 6 1887 1.00021 -2073 -616 -50 233 60.6 12.4 10 203 BG38 1.4 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 20:41Z 62 31 10 254 (WSW) 118729 - Patchy cloud 21:12Z 62 33 14 275 (W) 116552 - " 21:44Z 63 34 17 254 (WSW) 116776 - " 22:17Z 65 34 12 247 (WSW) 117918 - " 22:50Z 66 35 9 238 (WSW) 116608 - " 23:22Z 65 34 10 266 (W) 117006 - " 23:54Z 65 36 14 212 (SSW) 117623 - " 00:26Z 62 34 9 234 (SW) 117526 - " 00:59Z 61 32 8 271 (W) 116862 - " 01:42Z 58 33 5 256 (WSW) 118559 - " 02:34Z 58 35 6 202 (SSW) 121126 - " 03:04Z 57 36 6 224 (SW) 120987 - " 03:34Z 57 35 10 222 (SW) 120686 - " 04:06Z 57 34 9 235 (SW) 122080 - " 04:37Z 56 33 7 220 (SW) 121621 - " 05:09Z 57 34 6 259 (W) 120779 - " 05:41Z 57 34 4 260 (W) 120450 - " 06:14Z 57 34 8 222 (SW) 121853 - " 06:46Z 56 34 8 207 (SSW) 119568 - " 07:19Z 55 34 10 248 (WSW) 120576 - " 07:50Z 54 34 9 215 (SW) 119425 - " 08:20Z 54 36 6 191 (S) 117924 - " 08:51Z 54 37 7 192 (SSW) 119631 - " 09:23Z 54 36 8 189 (S) 118729 - " 09:54Z 54 36 9 216 (SW) 118539 - " 10:26Z 54 36 11 209 (SSW) 119777 - 10:58Z 54 37 13 210 (SSW) 120275 - 11:29Z 53 37 10 216 (SW) 120040 - 12:00Z 53 38 10 223 (SW) 118940 - 12:33Z 52 39 15 215 (SW) 120014 - ---=== TELESCOPE STATUS ===--- 02:20Z Doors open, fans on 04:05Z Enclosure off 11:45Z Enclosure on, shut down. No interlocks bypassed. Status at 11:55Z: Telescope stowed at: 30 deg altitude Instrument mounted: Cartridge 6 Counterweights at: 247 Autofill systems: On 180L LN2 dewar scales: Spectro 231 lb, 25 psi Imager 218 lb, 19 psi ---=== SOFTWARE USED ===--- IOP/SOP: v3_140_0 Watcher: v2_29_0 MCP: v5_25_0 TPM: tpm_v2_42_0 AstroDa: v14_47 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v1_92 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5270 0.5260 Axial B 0.6950 0.7130 Axial C 0.9980 0.9990 Trans D -8.6840 -8.6780 Lateral E 11.5316 11.5062 Lateral F 11.5062 11.4808 GALILS Commanded: 4850. -3650. 4050. -6150. 40200. 39150. Actual: 4837. -3652. 4065. -6127. 40183. 39144. SETMIR VALUES PriDesOrient: 0.00 -3.60 26.90 1470.40 198.00 PriOrient: 0.00 -3.80 26.76 1470.93 196.30 SECONDARY: ---------- Focus: 0.00 Air Temp.: 17.9 Alt.: 30.198085 MIGS TONIGHT NOMINAL Axial A 1.5010 1.7810 Axial B 1.1100 1.3910 Axial C 1.0870 1.3710 Trans D 0.0730 0.0630 GALILS Commanded: 1628631. 1548153. 1574452. -3900. -8750. Actual: 1628641. 1548187. 1574504. -3899. -8726. SETMIR VALUES SecDesOrient: 1257.00 0.00 -25.00 0.00 146.27 SecOrient: 1256.97 0.01 -25.01 0.26 146.55 ---=== PROBLEMS IN DETAIL ===--- Spectrographs were collimated (13.1C) - no change was done: ---------------------------------------------------- IDL 5.5a> collimate, 28177 Reading /data/spectro/53238/sdR-b1-00028177.fit Reading /data/spectro/53238/sdR-b1-00028178.fit COLLIMATE: Min offset = -0.77 pix COLLIMATE: Max offset = 0.56 pix COLLIMATE: RMS across CCD = 0.32 pix COLLIMATE: Mean offset = -0.07 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of 2.7 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-53238-b1-00028177.log PostScript file = Collimate-53238-b1-00028177.ps Reading /data/spectro/53238/sdR-b2-00028177.fit Reading /data/spectro/53238/sdR-b2-00028178.fit COLLIMATE: Min offset = -0.73 pix COLLIMATE: Max offset = 0.18 pix COLLIMATE: RMS across CCD = 0.23 pix COLLIMATE: Mean offset = -0.17 pix COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 6.1 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-53238-b2-00028177.log PostScript file = Collimate-53238-b2-00028177.ps Reading /data/spectro/53238/sdR-r1-00028177.fit Reading /data/spectro/53238/sdR-r1-00028178.fit COLLIMATE: Min offset = -0.54 pix COLLIMATE: Max offset = 0.72 pix COLLIMATE: RMS across CCD = 0.34 pix COLLIMATE: Mean offset = -0.02 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 156 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 156" Log file = Collimate-53238-r1-00028177.log PostScript file = Collimate-53238-r1-00028177.ps Reading /data/spectro/53238/sdR-r2-00028177.fit Reading /data/spectro/53238/sdR-r2-00028178.fit COLLIMATE: Min offset = -0.44 pix COLLIMATE: Max offset = 0.44 pix COLLIMATE: RMS across CCD = 0.22 pix COLLIMATE: Mean offset = 0.12 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -1054 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston -1054" Log file = Collimate-53238-r2-00028177.log PostScript file = Collimate-53238-r2-00028177.ps IDL 5.5a> --== Misc. Watcher Messages ==-- The following three messages caused us to clean the tape drives on ID3, replace the tapes, and reboot the crate. TEXTONLY *** WARNING *** dscArch: Couldn't check tape tape ID 2 (TEXTONLY *** WARNING *** dscArch: Couldn't check_tape tape ID 2 : 2004-08-21 00:50:41Z) dsc E archDirBad Bad tape directory drive ID 1 (dsc_E_archDirBad Bad tape directory drive ID 1 : 2004-08-21 00:50:41Z ) TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 2 (TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 2 : 2004-08-21 00:50:41Z ) ======================================================= TPM error reports ----------------- The following errors recurred: dewar (Imager Dewar Scale Communications Fault:: 2004-08-21 08:33:54Z) m2 mig (Communications Fault: 2004-08-21 11:02:25Z) Note that at one point tonight, we had a repeat of the massive network overload issue whereby TPM reported many failres to read data.