APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday August 2, 2004 (MJD 53220) ---=== OBSERVING TEAM ===--- Night: Dan Long, Pete Newman Support: Craig Loomis, Jon Brinkmann (by phone), French Leger ---=== OBSERVING PLAN ===--- Post-summer shake-down tasks, night 1 of 4. ---=== OBSERVING SUMMARY ===--- Post-shutdown shake-down tests in the enclosure. We were prevented from opening by nearby thunderstorms and rain on site. Tasks completed: - Landru made useable. - Spectrographs focused and monthly check-out completed. - MCP fiducial tables updated. - Imager proven alive. We had several problems, but nothing very unexpected (so far) considering the complexity of the work done over the summer. ---=== OBSERVING LOG ===--- This is the first night of shake-down following the 2004 summer maintenance shutdown. We are aware that there has been substantive work down on almost all systems, including the telescope, imager and spectrographs. Dan built, installed and declared test IOP v3_140_0, which merges the main survey and SEGUE cmponents of IOP. This will require testing on sky, but no immediate problems were seen. The operating system on landru (the observers' console) has had a big "up"grade to a version supported by Fermilab. This has several knock-on consequences: - Several standard software tools are presently missing (e.g. netscape, pico). - The version of ssh on landru has changed, resulting in a new host key for landru, so first attempts to log in to each machine _from_ landru will result in a dialog about known hosts. This was traced by Craig Loomis to the fact that the host key for landru had not been retained from the previous operating system, which confused the other machines. - There seems to be some difficulty passing on ssh keys from landru through sdsshost to the CVS repository at FNAL. A workaround is to start a new ssh agent on sdsshost, add a key to it, then use cvs: sdsshost> ssh-agent tcsh -l sdsshost> ssh-add (then enter your passphrase) sdsshost> cvs ci ... (should work) hoggpt is down waiting on a disk replacement, so no infrared sky camera. Craig loomis The coolant loop pump for the spectrographs is off, waiting for engineering check-out. Spectro camera r2, which was working during the afternoon, started delivering all pixels reading 65535. We power-cycled all components of sp2, then issued an iack. The next bias showed b2 messed up by the power cycle and r2 now reading all zeros. Another bias showed b2 looking close to normal and r2 still zeros. We issued another iack and tried again, recalling the the day crew recently required several iack's on the imager to get its cameras back to normal. Note that during all this, the watcher reported all temperatures and voltages to be withing tolerances. Recalling a previous episode of anomalous biases from the spectrographs being due to a bad connector on the output of the camera electronics saddlebag, we powered down the cameras, and removed and checked the main SCSI-type connectors on both ends of the cable. All looked normal (though we do not claim this was a detailed examination). We powered up again, and lo! r2 was returning senssible data again. We ran a series of 10 biases which returned the bias level to normal. We focussed the spectrograph cameras, principally b2 which required a movement of over 200 degrees following its dismantlement. The locking ring now moves very smoothly! The other cameras were quite close to focus. We built new fiducial tables for all axes, checked them in as v1_21 and declared them current. We encountered some error messages during the process (see problem section ! below for the gory details), but they seem to be harmless warnings, so we read the new tables into the MCP. TPM is reporting a disconnected temperature sensor all night and an M2 mig comms fault later: tmicro (Disconnected Sensor in M2B4:: 2004-08-03 01:18:50Z) m2 mig (Communications Fault: 2004-08-03 07:32:51Z) sdss-commish seems to not be synchronised to the GPS clock, running about 4 minutes fast as of 00:50 MDT. The was discovered when a watcher running on 'commish reported the more recent packets as coming from the future... Also, the watcher "RealTime" display running on sdss-commish did not show the correct Tape Drive states, while one running on sdsshost did. The imager coughed and spluttered before starting. The first goDrift of the night complained that amplifier voltages were not at their nominal operating values, despite watcher and cc nn vcheck's showing they were. That goDrift was quit without resetting anything. However, on issuing an iack, the imager immediately began spewing forth data. That bias run was endDrifted and another started with the same result. Eventually, we decided the imager's amplifiers hadn't been "turned on" following the most recent PSU failure, so we issued a setImagerAmpsOn command, and the next goDrift started correctly (and with clearly different gains judged from the noise seen on the monitors). We completed spectro monthly checkout (details at the end of this section). Near the start, we discovered the fuse for the flat-field lamps had blown. Dan replaced it and all was well. During the dithered flats stage, the spectro display monitors behaved strangely, showing for sp1 a solid blue screen for the top three-fourths and a prior flat on the rest. The sp2 screen was also "frozen". However, the DA continued to deliver good data to sdsshost. We suspect the VxWorks board in the spectro DA crate that feeds the monitors was hung. This is a replacement for the board destroyed by a lightning on the main control building this summer. We left the monitors in the hung state, so that Craig Loomis and/or Jon Brinkmann can look for a cause tomorrow. The monthly check completed successfully otherwise. Dan spotted that the current version of the TCC is still 2.7.2 dated November 11, 2003, although Russell Owen had said a new version with the azimuth axis tilt swindle removed had been installed. Russel has been emailed to clarify the position. endNight was run onto the cloudy-night tapes. - - - - - - - - - - - - Details of the spectro monthly checkout: sp1> mechSend s sp1> mechSend s spMechVersion v2_5_0 Bootup 719771169 SpectroID 1 SlitID 37 Air On Shutter_open_sensor Off Shutter_closed_sensor On Left_open_sensor Off Left_closed_sensor On Right_open_sensor Off Right_closed_sensor On Coll_motor_A 1 Coll_motor_B 2 Coll_motor_C 0 Requested_exp.time 0.0 Exp_time_left 0.0 Last_exp.time 2.1 Exp_state None Shutter_open_transit .73 Shutter_close_transit .53 Coll_motor_A_status 0x85 Coll_motor_B_status 0x85 Coll_motor_C_status 0x85 Coll_motor_B_status 0x85 Coll_motor_C_status 0x85 sp2> mechSend s sp2> mechSend s spMechVersion v2_5_0 Bootup 719771169 SpectroID 2 SlitID 5 Air On Shutter_open_sensor Off Shutter_closed_sensor On Left_open_sensor Off Left_closed_sensor On Right_open_sensor Off Right_closed_sensor On Coll_motor_A 0 Coll_motor_B 0 Coll_motor_C 0 Requested_exp.time 0.0 Exp_time_left 0.0 Last_exp.time 2.1 Exp_state None Shutter_open_transit .74 Shutter_close_transit .65 Coll_motor_A_status 0x85 Coll_motor_B_status 0x85 Coll_motor_C_status 0x85 12 8 11 18 Flat before 2038 2052 2057 2065 sp2> mechMoveColl 500 500 -500 12 8 9 16 2038 2052 2031 2063 sp1> mechMoveColl 500 500 -500 sp2> mechMovecoll 1000 1000 -1000 7 3 0 7 2033 2047 2023 2055 wavelength 3801 5786 3783 5770 6171 9240 6153 9215 sp2 mechMoveColl 1100 -1100 0 3801 5786 3804 5802 6171 9239 6177 9248 sp1> mechMoveColl -1000 1000 0 3782 5757 3804 5802 6149 9207 6177 9247 too far sp1> mechMoveColl 500 -500 0 7 3 0 7 2033 2047 2023 2055 3793 5772 3804 5802 6161 9223 6177 9247 sp1> mechPiston 750 sp2> mechPiston -1000 b1 ring +21 b2 ring -30 IDL 5.5a> collimate, 27551 Reading /data/spectro/53220/sdR-b1-00027551.fit Reading /data/spectro/53220/sdR-b1-00027552.fit COLLIMATE: Min offset = -0.60 pix COLLIMATE: Max offset = 0.30 pix COLLIMATE: RMS across CCD = 0.25 pix COLLIMATE: Mean offset = -0.08 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of 2.7 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-53220-b1-00027551.log PostScript file = Collimate-53220-b1-00027551.ps Reading /data/spectro/53220/sdR-b2-00027551.fit Reading /data/spectro/53220/sdR-b2-00027552.fit COLLIMATE: Min offset = -0.36 pix COLLIMATE: Max offset = 0.57 pix COLLIMATE: RMS across CCD = 0.26 pix COLLIMATE: Mean offset = 0.09 pix COLLIMATE: Camera b2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of -3.3 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-53220-b2-00027551.log PostScript file = Collimate-53220-b2-00027551.ps Reading /data/spectro/53220/sdR-r1-00027551.fit Reading /data/spectro/53220/sdR-r1-00027552.fit COLLIMATE: Min offset = -0.47 pix COLLIMATE: Max offset = 0.67 pix COLLIMATE: RMS across CCD = 0.30 pix COLLIMATE: Mean offset = -0.01 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 85 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 85" Log file = Collimate-53220-r1-00027551.log PostScript file = Collimate-53220-r1-00027551.ps Reading /data/spectro/53220/sdR-r2-00027551.fit Reading /data/spectro/53220/sdR-r2-00027552.fit COLLIMATE: Min offset = -0.49 pix COLLIMATE: Max offset = 0.38 pix COLLIMATE: RMS across CCD = 0.19 pix COLLIMATE: Mean offset = 0.06 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -549 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston -549" Log file = Collimate-53220-r2-00027551.log PostScript file = Collimate-53220-r2-00027551.ps Final collimator positions: sp1> mechSend s --> 751 1750 249 sp2> mechSend s --> 1600 -601 -2500 27553 - 27559 Dithered Flats 27552, 27560 Associated Arcs Sometime during the dithered flats, the scrolling displays stopped updating. The top 2/3 of the sp1 display was blue. The rest of sp1 and all of sp2 showed a flat. 27562 - 27586 25 biases Filename 02% 05% 10% 50% 90% 95% 98% ------------------- ------ ------ ------ ------ ------ ------ ------ sdR-b1-00027580.fit -6.2 -5.1 -2.9 2.8 7.3 9.6 10.7 sdR-b2-00027580.fit -8.3 -6.9 -5.6 1.0 7.6 9.0 11.5 sdR-r1-00027580.fit -7.0 -5.8 -4.0 1.1 6.9 8.0 10.3 sdR-r2-00027580.fit -7.8 -6.4 -4.8 0.9 6.2 7.2 9.3 27587 - 27588 3 900s darks Filename 02% 05% 10% 50% 90% 95% 98% ------------------- ------ ------ ------ ------ ------ ------ ------ sdR-b1-00027587.fit -3.3 -1.3 0.9 6.6 12.5 14.8 17.1 sdR-b2-00027587.fit -5.2 -3.0 -1.2 6.2 12.8 15.5 18.1 sdR-r1-00027587.fit -4.9 -2.9 -1.7 5.1 11.0 13.1 15.6 sdR-r2-00027587.fit -6.5 -5.4 -4.4 3.0 10.6 13.5 20.6 ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4737 07:05Z 07:15Z 100 O 127.79 130.19 20 ignore 4738 07:19Z 07:24Z 100 O 131.13 132.53 18 ignore 4739 07:29Z 08:15Z 100 O 133.67 145.38 81 bias doghouse ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): y QA Procedures Done (y/n): y (per monthly check-out procedures) UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 27504, plate 1573 ------- 01:30 27504 0.0 bias Testing and calibrations 01:34 27505 0.0 bias 01:40 27506 0.0 bias 01:43 27507 0.0 bias 01:46 27508 0.0 bias 01:56 27509 0.0 bias 01:58 27510 0.0 bias 02:03 27511 0.0 bias 02:07 27512 0.0 bias 02:20 27513 0.0 bias 02:25 27514 0.0 bias 02:28 27515 0.0 bias 02:34 27516 0.0 bias 02:39 27517 0.0 bias 02:55 27518 0.0 bias 03:12 27519 0.0 bias 03:16 27520 0.0 bias 03:19 27521 0.0 bias 03:21 27522 0.0 bias 03:23 27523 0.0 bias 03:25 27524 0.0 bias 03:27 27525 0.0 bias 03:29 27526 0.0 bias 03:31 27527 0.0 bias 03:33 27528 0.0 bias 03:35 27529 0.0 bias 03:38 27530 0.0 bias 04:08 27531 10.0 flat Focussing 04:11 27532 2.0 arc 04:27 27533 2.1 arc 04:30 27534 2.1 arc 04:44 27535 2.1 arc 04:47 27536 2.1 arc 05:07 27537 10.0 flat 05:09 27538 2.0 arc 05:21 27539 10.0 flat 05:33 27540 10.0 flat 05:47 27541 2.0 arc 06:05 27542 2.0 arc 06:15 27543 2.0 arc 06:20 27544 2.0 arc 06:39 27545 10.0 flat 06:42 27546 2.0 arc 06:54 27547 2.1 arc 06:57 27548 2.1 arc 07:13 27549 2.1 arc 07:16 27550 2.1 arc 07:41 27551 2.1 arc 07:43 27552 2.1 arc 07:59 27553 10.0 flat Dithered flats 08:03 27554 10.0 flat 08:07 27555 10.0 flat 08:11 27556 10.0 flat 08:14 27557 10.0 flat 08:18 27558 10.0 flat 08:22 27559 10.0 flat 08:26 27560 2.0 arc Focus checks 08:29 27561 2.0 arc 08:33 27562 0.0 bias Bias library 08:35 27563 0.0 bias 08:38 27564 0.0 bias 08:40 27565 0.0 bias 08:42 27566 0.0 bias 08:44 27567 0.0 bias 08:46 27568 0.0 bias 08:48 27569 0.0 bias 08:50 27570 0.0 bias 08:52 27571 0.0 bias 08:55 27572 0.0 bias 08:57 27573 0.0 bias 08:59 27574 0.0 bias 09:01 27575 0.0 bias 09:03 27576 0.0 bias 09:05 27577 0.0 bias 09:07 27578 0.0 bias 09:09 27579 0.0 bias 09:11 27580 0.0 bias 09:13 27581 0.0 bias 09:15 27582 0.0 bias 09:18 27583 0.0 bias 09:20 27584 0.0 bias 09:22 27585 0.0 bias 09:24 27586 0.0 bias 10:02 27587 0.0 dark Darks 10:20 27588 0.0 dark 10:38 27589 0.0 dark ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- No tapes sent or stored. ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 00:45Z 60 52 7 125 (SE) -9999 - Rain & lightning 01:20Z 59 51 7 176 (S) -9999 - 01:51Z 58 52 7 183 (S) -9999 - 02:24Z 59 52 6 204 (SSW) 1301 - 02:56Z 58 53 7 226 (SW) 1198 - 03:29Z 57 51 10 181 (S) 1229 - 04:01Z 55 52 8 187 (S) 1932 - 04:33Z 57 52 10 208 (SSW) 1679 - 05:03Z 58 52 10 179 (S) 1665 - 05:33Z 57 51 10 177 (S) 1569 - 06:06Z 57 51 9 196 (SSW) 1365 - Overcast 06:38Z 56 51 7 190 (S) 1152 - 07:09Z 56 51 5 182 (S) 1349 - 07:46Z 54 51 6 186 (S) 1497 - 08:19Z 54 51 7 182 (S) 1660 - 08:51Z 55 51 5 199 (SSW) 1768 - 09:24Z 55 51 5 182 (S) 1833 - 09:56Z 54 51 4 186 (S) 1497 - 10:28Z 55 51 6 193 (SSW) 1563 - 10:59Z 55 52 7 213 (SSW) 2363 - ---=== TELESCOPE STATUS ===--- Enclosure on and unventilated all night. No interlocks bypassed. Status at 11:23Z: Telescope stowed at: 30 deg altitude Instrument mounted: Cartridge 5 Counterweights at: 260 Autofill systems: On 180L LN2 dewar scales: Spectro 68 lb, 24 psi (Day crew will change) Imager 210 lb, 21 psi ---=== SOFTWARE USED ===--- IOP/SOP: v3_140_0 (test) Watcher: v2_29_0 MCP: v5_25_0 TPM: tpm_v2_42_0 AstroDa: v14_47 TCC: 2.7.2 sdssProcedures: v1_92 SoS: v4_10_7 hoggPT: down plate-mapper: not obtained due to ssh problems ---=== MIRROR NUMBERS ===--- Not obtained - mechanical collimation due 2004 August 3. ---=== PROBLEMS IN DETAIL ===--- Problem 1 - switchMCP error messages ==================================== During the build of the new fiducial tables, we were unable to switch to the new tables as switchMCP returned an error: switchMCP *** Can't find server name for address 192.41.211.10: No response from server *** Can't find server name for address 192.41.211.10: No response from server *** Can't find server name for address 192.41.211.10: No response from server v5_25_0 This seems to be related to the calls to nslookup from the swtichMCP script, as nslookup is returning the same error as the first line of its reponse: prn@sdsshost:/p/mcpbase/etc/ > nslookup sdsshost *** Can't find server name for address 192.41.211.10: No response from server Server: dnscache2.apo.nmsu.edu Address: 192.41.211.11 Non-authoritative answer: Name: sdsshost.apo.nmsu.edu Address: 192.41.211.171 which in turn seems related to dnscache1 (not dnscache2) being the default DNS server, but being unavailable at the time (updating?). In any case, this MAY be a red herring, as IOP fiducialsVersion is returning the new version number, and the code producing the error in switchMCP seems to be merely checks the correspondence between several machine IP names and addresses, which are, indeed, correct. On that basis, we decided to reboot the MCP in any case.