APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday August 23, 2008 (MJD 54702) ---=== OBSERVING TEAM ===--- Paul Harding David Schlegel Elena Malanushenko (day) Steph Snedden Dmitry Bizyaev ---=== OBSERVING PLAN ===--- Super-ShakeDown ! ---=== OBSERVING SUMMARY ===--- Progress in testing the new guider was achieved: centered and focused images of all fibers have been obtained. Due to high humidity we were unable to go further and test the guider on the sky. In addition to problems written by Dan Long in day log, we have the following: 1. telescope is significantly out of balance, even with pretty unusual positions for the adjustable counterweights. 2. The northwest kinematic latch for the spectro corrector does not retract fully to the unlatched position. 3. Ssh keys authorization does not work, apparently, between many servers, including sdsshost2, sos, and DA servers. We think the problem began on July 22, see "problems" for detail. 4. Sometimes 3 FF lamps are on rather than 4, see "problems" and log. --- We were unable to complete any procedure from Spectro monthly checkout because of problems with ssh keys (-> sos) and warm sp1(red). ---=== OBSERVING LOG ===--- Afternoon: ----------- Dan Long's information: 1) sp1 is still 5-8 deg warmer then it should be. 2) secondary mirror does not move properly. 3) imager server is not alive. We operate with old versions of all software. All systems were checked. Some problem are discovered (surprised?). Corrector's SW latch is not getting unlatched completely because it cannoyt reach the limit switch. We decided it is safe to mount corrector anyway but we need engeneering support to remove the corrector. We switched instrument lift to manual position as interlock does not allow us to operate lift with this problem. Cart 6 is mounted. Counterweights are at 300 instead of expected 250 and the telescope is still out of balance very much. It goesup with currents 2.5K, and down with 4.5K. After AL brake gets applied, the telescope jumps up by ~ 1.5 degrees in AL. Fiducials are checked and look ok. We were worried about AZ movement as it was not operated for a month, but it is seen that it is doing good. Night: ------ 00:10Z Ventilation started. It works OK. 00:38Z sdsshost2 was rebooted. Before the reboot, the "im" server widnow didn't show up and watcher didn't work. The reboot of the watcher servers fixed the problem. 00:40Z astro crates were rebooted because there was no connection to ad2. Fixed. 01:10Z Many files are left and have not been archived in data pool. Teamster restarted. It didn't help, as we'll see below. 01:55Z Humidity climbs up, so we switched the ventilation back to internal. 02:00Z - ~08:00Z New guider was tested. The fisrt images of FF lamp on telescope were far out of focus and mostly out of frames, just two fibers were seen. Paul detached the camera and checked the centering and assembling. Then David corrected the binning for the camera. After that the camera with lenses was put on a tripod and roughly focused on real sky. Nice pictures! First light! Then the camera went back to the telescope and well focused; well centered images of fibers were obtained. 06:19Z after ~1 hour break we turned on FF lamps again. Only 3 of them came on. We turned it off an on again. Now all 4 lamps are on. These 3 FF lamps instead of 4 were observed several times during the night. 06:40Z photo and astro crates were cold-rebooted (including the back-plates). Test imager bias sequence were started. After that we get in watcher: dsc E archFTPHung Archiver FTP file upload is hung (idRR-007644-z1-0007.fit) (dsc_E_archFTPHung Archiver FTP file upload is hung (idRR-007644-z1-0007.fit): 2008-08-24 06:57:12Z ) 09:45Z endNight is failed. It fails at the point when it tries to obtain IR camera images but cannot do that because of the ssh keys problem, see the "problems" section. We went manually through the endNight tasks. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 7644 02:28Z 02:45Z 100 O 78.81 83.32 44 ignore 7645 06:56Z 07:00Z 100 O 146.18 147.12 18 ignore 54702 07:56Z 07:58Z 100 O 161.16 161.76 54324 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 01:50 54322 0.0 bias ----- sequence 54323, plate 3106 ------- 02:02 54323 10.0 flat 07:59 54324 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 01:15Z 56 46 19 75 (ENE) 1110 - 01:47Z 54 49 23 73 (ENE) 1143 - some clouds 02:19Z 53 50 21 66 (ENE) 1314 - '' 02:51Z 53 48 21 86 (E) 1083 - '' 03:23Z 52 49 24 73 (ENE) 1028 - '' 03:56Z 52 48 27 72 (ENE) 940 - '' 04:28Z 51 49 24 73 (ENE) 973 - '' 05:01Z 51 47 24 73 (ENE) 1012 - '' 05:34Z 51 48 13 87 (E) 1021 - '' 06:07Z 50 47 26 74 (ENE) 979 - '' 06:40Z 50 46 21 75 (ENE) 944 - almost clear 07:12Z 50 46 12 90 (E) 960 - '' 07:44Z 49 47 7 104 (ESE) 1025 - '' 08:16Z 49 47 6 104 (ESE) 963 - '' 08:48Z 49 47 11 99 (E) 977 - clouds 09:20Z 50 48 6 93 (E) 849 - '' 09:52Z 50 48 11 123 (ESE) 781 - '' ---=== TELESCOPE STATUS ===--- 00:10Z - Began ventilation. Doors and louvers opened, fans on. 01:55Z - Ventilation stopped (high humidity). Status at: 09:45Z Telescope stowed at: 30,121,00 mount Instrument mounted: Cartridge 6 Counterweights at: 300 lbs LN2 autofill systems: Connected and turned on. 180L LN2 dewar scales: Spectro 252 lbs, 14.7 psi Imager 107 lbs, 17.4 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_38_0 Watcher: v2_36_0 MCP: v6_4_0 TPM: --- AstroDa: v15_13_1 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_42 SoS: plate-mapper: ---=== MIRROR NUMBERS ===--- We didn't take mirror numbers because of the secondary's actuators problem. ---=== PROBLEMS IN DETAIL ===--- Major problems with sp1(red) temperature and actuators for the secondary mirror, see in Dan Long's day log. tmicro (Disconnected Sensor in M2B2:Disconnected Sensor in M0B4:: 2008-08-24 02:23:51Z) --- 1. Strange, humidity for 2.5m is 37% at the weather page, whereas 3.5 has 83%, and in-enclosure 2.5 humidity is 75%. Inconsistency. Dew point temperature is -1.4C against 9.4 at 3.5m. It looks incorrect for 2.5m as well. 2. There is no sos:/data/spectro/MJD directory (at sos). Log file indicates that "rsync" does not want to work because ssh requires password (ssh key is not set up). Because of this, sos database does not to update with new MJDs. 3. A lot of errors of this type: dsc E archFTPHung Archiver FTP file upload is hung (idRR-007644-i1-0035.fit) (dsc_E_archFTPHung Archiver FTP file upload is hung (idRR-007644-i1-0035.fit): 2008-08-24 04:18:13Z ) There were old data in the data pool. We cleaned the pool, but once the new data started caming into the pool, the error again returned. Actually, the gang transfer works. The message about inability to upload file via FTP may suggest that servers cannot establish connection because of the same ssh authorization problem. The "current" page on sos is from July 22. Also, ther were frames in the pool from July 22. These indicate that the problem started on July 22. **** We are unable to transfer data from the DA to disk, and haven't been able to since about July 22. It seems to be related to the SoS ssh key / rsync problem. Something on sdsshost2 changed on July 22. The relevant murmur messaged for the DA transfers follow: 2008-08-24 06:58:03Z sdssid1 dArch dsc_W_archWrAttempt Error archiving frame 7644 7.41: Failed to close file 13 2008-08-24 06:58:03Z sdssid3 dArch dsc_W_archWrAttempt Error archiving frame 7644 7.43: Failed to close file 13 2008-08-24 06:58:04Z sdssid1 dArch dsc_W_archWrRetry Retrying write of file idRR-007644-z1-0007.fit 2008-08-24 06:58:04Z sdssid1 dArch dsc_W_archWrAttempt Error archiving frame 7644 7.41: Failed to get header (status = 0x8000c434) 2008-08-24 06:58:04Z sdssid1 dArch dsc_E_archWrFail Fatal error or retries exhausted. File idRR-007644-z1-0007.fit not archived! 2008-08-24 06:58:04Z sdssid3 dArch dsc_W_archWrRetry Retrying write of file idRR-007644-z3-0007.fit 2008-08-24 06:58:04Z sdssid3 dArch dsc_W_archWrAttempt Error archiving frame 7644 7.43: Failed to get header (status = 0x8000c434) 2008-08-24 06:58:04Z sdssid3 dArch dsc_E_archWrFail Fatal error or retries exhausted. File idRR-007644-z3-0007.fit not archived! ********** 2008-08-24 07:14:47Z sdssid4 dArch dsc_E_archFTPHung Archiver FTP file upload is hung (idRR-007645-i4-0009.fit) 2008-08-24 07:14:47Z sdssid5 dArch dsc_E_archFTPHung Archiver FTP file upload is hung (idRR-007645-i5-0009.fit) 2008-08-24 07:14:48Z sdssid6 dArch dsc_E_archFTPHung Archiver FTP file upload is hung (idRR-007645-i6-0009.fit) ***** 4. The northwest kinematic latch for the spectro corrector does not retract fully to the unlatched position. It gets close, but not quite enough to hit the unlatched sensor. The latched position is fine. We felt it was safe to mount the corrector, but were very careful to watch the force readout when raising it past the latch. We don't want to remove the corrector, though, because we are afraid it might hang upon the latch. 5. Sometimes 3, sometimes 4 FF lamps come on. The right bottom lamp (if to look from the back) behaves bad all the time. 6. The telescope is out of balance. traditionally, 255-270 for the position of counterweights was used. Now we tried even 300, and any way got -4800 over 2500 Volts at AlT=60 bottom heavy balance. Missed technician is sleeping on the primary mirror? Too heavy new coating?