APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday June 22, 2008 (MJD 54640) ---=== OBSERVING TEAM ===--- Howard Brewington Audrey Simmons Dan Oravetz (swing) Jon Brinkmann (support) Kaike Pan (support) Steph Snedden (support) Fritz Stauffer (support) ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Dan spent four hours (with the help of Fritz, Jon, and Steph) trying to fix a mismatched NTP time server problem within the new galileo computer, along with a configuration problem with the guider mac computer. The combination of these problems prevented the guider to update. Please see the afternoon section of the log for a copy of Fritz's email where he described what happened in more detail. Dan discovered that when the spectrographs were focused for the current temperature inside the enclosure, according to collimate, a postcal returned a yellow xsigma value for b1 (afternoon section). Triplet problem? Sporatic rain and clouds kept us in all night. We completed cloudy night tasks: spectro biases & darks and dithered flats. ---=== OBSERVING LOG ===--- Afternoon: ---------- Restarted the servers. A `cameraDown im' was issued in IOP by Connie to hopefully stop the critical alert emails regarding the imager while it's being worked on. sdsshost2.apo.nmsu.edu % df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 142G 127G 7.6G 95% / /dev/sda1 99M 30M 64M 33% /boot /dev/sdb1 917G 690G 181G 80% /data sdssfiles1:/data1/data 1.5T 599G 794G 44% /files Swapped out the spectro dewar. Cartridge 3 is mounted. Spectro focus is set for ~14C (sp1) and ~13C (sp2). **During the spectro collimation procedures I encountered a problem with the b1 xsigma and wsigma values in SoS. As can be seen in the last entry of collimate in the Problems section, all cameras were in focus: b1 requested a move of -0.8 degrees, b2 requested a move of only 1.0 degrees, r1 requested 124 steps, and r2 180 steps. These numbers are also in accordance with the table in the Afternoon Checkout procedure that lists the offsets collimate requests when perfectly in focus for different cartridges. So, according to the output from collimate, the spectrographs were right on the money. I took a post-cal after checking the collimate output and SoS showed the b1 xsigma and wsigma values to be noticeably higher than the values for the other cameras, in fact b1 wsigma was yellow. I called Kaike and he suggested it might be a problem with the triplet since the blue focusing information is relative to the red focus. **The frame number was not incrementing normally in the SAOImage display when I tried `gcam 5' in SOP and we received the following error: "GCamReply="could not open file: guider:gimg0001.fits; error: No Error" Images were not showing up in /data/spectro/54640/guider/ either. We tried rebooting the guiderMac, but continued to get the same error. We also got the following error on the guiderMac desktop repeatedly: "Network time could not set the time on the Macintosh. The name of the time server does not exist in the domain name server database." **Email from Fritz explaining the details of what happened: "There were two problems: a poorly configured guider camera computer and a crippled galileo that runs an NTP time server and a domain name service, DNS, server. The first problem is the guider camera computer configuration. The guider camera depended on one NTP server and one site DNS server, and both of these servers are on galileo. Generally, I setup the computers to use redundant servers, so two NTP servers and two DNS caches which depend on two site DNS servers. I never have touched the guider mac's configuration, so it is the original setup. The second problem is galileo. galileo, which is an old Sun machine, now fails to stay up, so I replaced it during the day with a new spare galileo built with Linux. When I turned on the new galileo, the DNS server failed to start due to a misconfiguration, and the NTP server failed to start because the server code was broken. Because of the NTP server monitoring, I knew about the NTP server problem, and I had just fixed that problem when you called. The DNS problem was tipped off because the guider camera could not resolve the IP address from the utc-time.apo.nmsu.edu address and because it could not create files on the NFS mounted file system, since it could not resolve the computer which owns the NFS export. The correct fix is to fix the guider mac's configuration. I need to see if I can add the second NTP server, and I need to change the DNS from galileo to one of the DNS caches and preferably both DNS caches. One more thing that should be done is to put in a monitoring script that verifies the DNS caches and DNS servers are working and will email site critical alerts if they aren't." Night: ------ Note: Last night's observations of plate 2812 never showed up on SOS. Even when sos_redo was issued, the computer hung. 02:50Z - Started ventilating with the doors and louvers open, fans on. It was too cloudy to open so we started spectro monthly checkout engineering tasks. 04:35Z - Spectro biases, exposures 53738-53763 in SOS. apo_plotbias, 53738 (BIAS) Filename 02% 05% 10% 50% 90% 95% 98% ------------------- ------ ------ ------ ------ ------ ------ ------ sdR-b1-00053738.fit -5.9 -3.9 -2.8 2.4 7.3 8.7 10.6 sdR-b2-00053738.fit -8.0 -6.1 -4.9 1.4 6.8 9.2 10.5 sdR-r1-00053738.fit -7.4 -5.6 -3.4 1.6 7.5 8.6 10.8 sdR-r2-00053738.fit -7.3 -6.3 -4.2 1.1 6.3 8.4 10.5 05:50Z - Stopped ventilating since it started raining. 05:52Z - Began spectro darks, exposures 53764-53765. apo_plotbias, 53763 (DARK) Filename 02% 05% 10% 50% 90% 95% 98% ------------------- ------ ------ ------ ------ ------ ------ ------ sdR-b1-00053763.fit -3.4 -1.2 0.6 6.5 12.2 14.2 16.4 sdR-b2-00053763.fit -5.4 -3.1 -0.6 5.7 12.0 14.3 16.8 sdR-r1-00053763.fit -4.8 -2.8 -1.1 4.4 10.9 13.1 15.3 sdR-r2-00053763.fit -5.9 -3.8 -1.8 4.5 10.8 12.9 15.1 07:02Z - Started dithered flats, exposures 53766-53780 (flats) and 53781-53782 (arcs). Collimators returned to their original positions. 09:54Z - The clouds persisted, and brought lightning and scattered showers with them. There is no hope in opening. We called it a night. 09:56Z - endNight rejoiced. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): yes QA Procedures Done (y/n): yes UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 23:21 53728 0.0 bias 00:25 53729 0.0 bias ----- sequence 53730, plate 2901 ------- 00:57 53730 10.0 flat 00:59 53731 2.0 arc 01:06 53732 3.0 arc 01:09 53733 3.0 arc 01:24 53734 3.0 arc 01:27 53735 3.0 arc 01:35 53736 10.0 flat 01:37 53737 2.0 arc 04:37 53738 0.0 bias 04:39 53739 0.0 bias 04:41 53740 0.0 bias 04:43 53741 0.0 bias 04:46 53742 0.0 bias 04:48 53743 0.0 bias 04:51 53744 0.0 bias 04:53 53745 0.0 bias 04:55 53746 0.0 bias 04:57 53747 0.0 bias 05:00 53748 0.0 bias 05:02 53749 0.0 bias 05:05 53750 0.0 bias 05:07 53751 0.0 bias 05:09 53752 0.0 bias 05:12 53753 0.0 bias 05:14 53754 0.0 bias 05:16 53755 0.0 bias 05:18 53756 0.0 bias 05:21 53757 0.0 bias 05:24 53758 0.0 bias 05:27 53759 0.0 bias 05:29 53760 0.0 bias 05:31 53761 0.0 bias 05:34 53762 0.0 bias 06:10 53763 0.0 dark 06:28 53764 0.0 dark 06:47 53765 0.0 dark 07:07 53766 10.0 flat 07:11 53767 10.0 flat 07:16 53768 10.0 flat 07:20 53769 10.0 flat 07:24 53770 10.0 flat 07:28 53771 10.0 flat 07:32 53772 10.0 flat 07:37 53773 10.0 flat 07:41 53774 10.0 flat 07:45 53775 10.0 flat 07:49 53776 10.0 flat 07:53 53777 10.0 flat 07:58 53778 10.0 flat 08:02 53779 10.0 flat 08:06 53780 10.0 flat 08:10 53781 2.0 arc 08:13 53782 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:24Z 70 39 13 215 (SW) 819 - clouds 23:56Z 68 36 9 215 (SW) 759 - clouds 00:27Z 60 38 9 174 (S) 487 - clouds 00:59Z 59 37 5 145 (SE) 590 - clouds 01:29Z 58 36 13 352 (NNW) 864 - clouds 01:59Z 51 42 16 33 (NNE) 1007 - clouds 02:30Z 54 39 6 81 (E) 760 - clouds 03:02Z 59 34 6 358 (NNW) 875 - clouds 03:34Z 58 38 7 254 (WSW) 1478 - clouds 04:07Z 58 37 8 211 (SSW) 1761 - clouds 04:39Z 59 38 4 180 (S) 2104 - " 04:58Z 59 37 9 184 (S) 1943 - " 05:32Z 59 37 11 189 (S) 1726 - " 06:04Z 60 36 11 181 (S) 1446 - " 06:36Z 60 37 9 190 (S) 1831 - " 07:08Z 59 37 11 177 (S) 1502 - " 07:41Z 58 35 7 155 (SSE) 993 - " 08:13Z 56 37 12 63 (ENE) 1061 - " 08:45Z 54 38 11 139 (SE) 728 - " 09:17Z 54 38 7 177 (S) 770 - " 09:49Z 51 39 7 106 (ESE) 794 - " ---=== TELESCOPE STATUS ===--- 02:50Z - Began ventilation. Doors and louvers opened, fans on. 05:50Z - Stopped ventilating. No interlocks bypassed. Status at: 09:55Z Telescope stowed at: 30,121,00 mount Instrument mounted: Cartridge 3 Counterweights at: 250 lbs LN2 autofill systems: Connected and turned on. 180L LN2 dewar scales: Spectro 279.20 lbs, 13.23 psi (read 18psi when visually checked) Imager 159.80 lbs, 0.09 psi (imager being worked on) ---=== SOFTWARE USED ===--- IOP/SOP: v4_38_0 Watcher: v2_36_0 MCP: v6_4_0 TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_42 SoS: v5_1_8 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0210 0.0200 Axial B -0.0300 0.0000 Axial C -0.0220 -0.0230 Trans D -0.0160 0.0120 Lateral E 10.0711 10.0584 Lateral F 11.6332 11.6205 GALILS Commanded: 4600. -2000. 2650. -12450. -6500. -6550. Actual: 4600. -2000. 2650. -12439. -6477. -6546. SETMIR VALUES PriDesOrient: 0.00 0.00 0.00 100.00 -200.00 PriOrient: 0.00 0.00 0.00 99.59 -200.69 SECONDARY: ---------- Focus: 0.00 Air Temp.: 11.8 Alt.: 89.999998 The telescope was actually at 30-degrees altitude when these were taken, I'm not sure why the altitude reported above under the Secondary says it was at zenith. MIGS TONIGHT NOMINAL Axial A -0.0410 -0.0440 Axial B -0.0570 -0.0570 Axial C -0.0600 -0.0670 Trans D -0.0230 -0.0110 GALILS Commanded: 1627510. 1653135. 1470887. -5900. -7800. Actual: 1627496. 1653168. 1470885. -5887. -7780. SETMIR VALUES SecDesOrient: 1257.00 60.00 -4.00 -20.00 118.90 SecOrient: 1256.99 59.99 -4.01 -19.92 119.29 ---=== PROBLEMS IN DETAIL ===--- Spectro collimation with cart. 3 -------------------------------- IDL 5.5a> collimate, 53732 Reading /data/spectro/54640/sdR-b1-00053732.fit Reading /data/spectro/54640/sdR-b1-00053733.fit COLLIMATE: Min offset = -0.38 pix COLLIMATE: Max offset = 0.62 pix COLLIMATE: RMS across CCD = 0.22 pix COLLIMATE: Mean offset = 0.07 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of -2.6 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-54640-b1-00053732.log PostScript file = Collimate-54640-b1-00053732.ps Reading /data/spectro/54640/sdR-b2-00053732.fit Reading /data/spectro/54640/sdR-b2-00053733.fit COLLIMATE: Min offset = -0.60 pix COLLIMATE: Max offset = 0.10 pix COLLIMATE: RMS across CCD = 0.18 pix COLLIMATE: Mean offset = -0.15 pix COLLIMATE: Camera b2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of 5.2 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-54640-b2-00053732.log PostScript file = Collimate-54640-b2-00053732.ps Reading /data/spectro/54640/sdR-r1-00053732.fit Reading /data/spectro/54640/sdR-r1-00053733.fit COLLIMATE: Min offset = -0.58 pix COLLIMATE: Max offset = 0.56 pix COLLIMATE: RMS across CCD = 0.31 pix COLLIMATE: Mean offset = -0.07 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 634 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 634" Log file = Collimate-54640-r1-00053732.log PostScript file = Collimate-54640-r1-00053732.ps Reading /data/spectro/54640/sdR-r2-00053732.fit Reading /data/spectro/54640/sdR-r2-00053733.fit COLLIMATE: Min offset = -0.58 pix COLLIMATE: Max offset = 0.50 pix COLLIMATE: RMS across CCD = 0.29 pix COLLIMATE: Mean offset = 0.12 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -1110 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston -1110" Log file = Collimate-54640-r2-00053732.log PostScript file = Collimate-54640-r2-00053732.ps % Program caused arithmetic error: Floating illegal operand According to the afternoon checkout collimation table, I accounted for the moves collimate would ask for if cartridge 3 were perfectly in focus: 634 + 100 = 734 -1110 - 300 = -1410 sp1> mechPiston 734 sp1> mechSend s sp1: -1025 -1025 -1025; 15.2C sp2> mechPiston -1410 sp2> mechSend s sp2: 8073 7407 7608; 12.9C IDL 5.5a> collimate, 53734 Reading /data/spectro/54640/sdR-b1-00053734.fit Reading /data/spectro/54640/sdR-b1-00053735.fit COLLIMATE: Min offset = -0.44 pix COLLIMATE: Max offset = 0.54 pix COLLIMATE: RMS across CCD = 0.21 pix COLLIMATE: Mean offset = 0.02 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of -0.8 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-54640-b1-00053734.log PostScript file = Collimate-54640-b1-00053734.ps Reading /data/spectro/54640/sdR-b2-00053734.fit Reading /data/spectro/54640/sdR-b2-00053735.fit COLLIMATE: Min offset = -0.42 pix COLLIMATE: Max offset = 0.20 pix COLLIMATE: RMS across CCD = 0.16 pix COLLIMATE: Mean offset = -0.03 pix COLLIMATE: Camera b2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of 1.0 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-54640-b2-00053734.log PostScript file = Collimate-54640-b2-00053734.ps Reading /data/spectro/54640/sdR-r1-00053734.fit Reading /data/spectro/54640/sdR-r1-00053735.fit COLLIMATE: Min offset = -0.52 pix COLLIMATE: Max offset = 0.64 pix COLLIMATE: RMS across CCD = 0.32 pix COLLIMATE: Mean offset = -0.01 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 124 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 124" Log file = Collimate-54640-r1-00053734.log PostScript file = Collimate-54640-r1-00053734.ps Reading /data/spectro/54640/sdR-r2-00053734.fit Reading /data/spectro/54640/sdR-r2-00053735.fit COLLIMATE: Min offset = -0.73 pix COLLIMATE: Max offset = 0.33 pix COLLIMATE: RMS across CCD = 0.29 pix COLLIMATE: Mean offset = -0.02 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 180 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston 180" Log file = Collimate-54640-r2-00053734.log PostScript file = Collimate-54640-r2-00053734.ps % Program caused arithmetic error: Floating illegal operand So, sp1 is in focus for ~15C and sp2 is in focus for about 13C. sp1> mechPiston -1000 sp1: -2026 -2026 -2025; 15.1C Now, sp1 is in focus for ~14C and sp2 is in focus for ~13C.