APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday August 20, 2007 (MJD 54333) ---=== OBSERVING TEAM ===--- DayShift: Howard Brewington Observers: Viktor Malanushenko Audrey Simmons Jon Brinkmann (support) ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Mostly photometric conditions tonight, but unfortunately we were flying blind. SOS was extremely slow at updating the reductions of our observations, so we had difficulty monitoring our progress during the night (see log and problem section for details). We obtatined good signal on cart 2 (plate 2474), cart 6 (2624), and cart 8 (2627). We finished cart 7 (plate 2566). Misc problems in log (TCC, disk space, hand paddle for instChange). Misc watcher errors in problem section. ---=== OBSERVING LOG ===--- Afternoon: ---------- We did not have enough space on sdsshost2:/data/* , and I had already RM-ed all the MJDs that were posted as "Safe to Remove." So, Jon B. had to compress some data to allow us to operate tonight. Jon also has plans to order larger drives for sdsshost2. TCC got rebooted earlier today; it restarted with local time instead TAI, which would not allow track commands to work. Jon B. made a software adjustment to fix the problem. When I did the checkout at the cartridge bay, the elevator would not work. I discovered that the pull-chain for the south door was hanging in front of the corner mirror at the south-east part of the bay. I used the auto hand-paddle on the north-side of the telescope today. I noticed that the upper number on the paddle, which should be 1000+ when mounting or dismounting a cart, was only reading about 120. However, I was able to mount and dismount a cart without any problems. Night: ------ Note: When we started observations, SOS had problems updating. Jon believes that SOS was hung up on trying to rereduce compressed files from previous nights, which were setup to relieve disk space. We flew blindly throughout the night, but believe that all of our exposures are ok. 02:19Z - Opened enclosure to clear, photometric conditions. 02:50Z - Cartridge 2, seguefaint140, plate 2474 mounted. fk5 used. Error message: using centroider for guider since dark does not exist (dark files were compressed, so we just ran with the old guider (Gun-guider off) for this plate instead). 03:20Z - Precals. 03:31Z - Started first 20 min exposure (disabled fibers 5 and 7 due to hot pixels and bad seeing). Seeing 1.6" 03:54Z - 15 min exposure. 04:13Z - 20 min exposure. 04:36Z - Postcals. Did not finish this plate. 04:52Z - Mounted cart 7, seguefaint149, plate 2566. fk5 star used. 05:06Z - Precals. 05:17Z - 20 min exposure. Seeing 1.0" 05:41Z - 20 min exposure. 06:06Z - 20 min exposure. **Note: We noticed that SOS was not processing any files at this time. We killed four "frozen" processes and restarted them (see problem section for notes). SOS started working again soon afterwards, but very very slowly. 06:34Z - 20 min exposure. 06:56Z - 20 min exposure. 07:20Z - 20 min exposure. 07:42Z - Postcals. Finished this plate. 07:56Z - Mounted cart 6, segue154, plate 2624. fk5 star used. 08:00Z - Precals. 08:06Z - 20 min exposure. Seeing 2.0" 08:30Z - 25 min exposure. 08:58Z - 25 min exposure. 09:29Z - 25 min exposure. 09:57Z - Postcals. Did not finish this plate. 10:11Z - Mounted cart 8, seguefaint155, plate 2627. GSFOGTF. 10:12Z - Precals. 10:19Z - 25 min exposure thought to be taken, but camera check prevented the button to be clicked. Did not realize that exposure was not underway. 10:45Z - 25 min exposure. Seeing 2.0" 11:14Z - Postcals. Did not finish this plate. 12:39Z - endNight rejoiced. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 6900 02:06Z 03:07Z 100 O 69.81 85.15 60 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): yes QA Procedures Done (y/n): yes UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 46037, plate 2627 ------- 00:46 46037 10.0 flat 00:49 46038 2.0 arc 00:53 46039 3.0 arc 00:56 46040 3.0 arc ----- sequence 46041, plate 2627 ------- 02:09 46041 0.0 bias 02:19 46042 0.0 bias ----- sequence 46043, plate -9999 ------- 02:49 46043 0.0 bias ----- sequence 46044, plate 2474 ------- 03:22 46044 10.0 flat 03:24 46045 2.0 arc 03:53 46046 1201.0 target 04:11 46047 900.0 target 04:35 46048 1200.0 target 04:38 46049 10.0 flat 04:41 46050 2.0 arc 04:44 46051 10.0 flat 04:46 46052 2.0 arc ----- sequence 46053, plate 2566 ------- 05:09 46053 10.0 flat 05:12 46054 2.0 arc 05:39 46055 1201.0 target 06:03 46056 1200.0 target 06:28 46057 1201.0 target 06:55 46058 1201.0 target 07:18 46059 1200.0 target 07:42 46060 1200.0 target 07:45 46061 10.0 flat 07:48 46062 2.0 arc ----- sequence 46063, plate 2624 ------- 08:02 46063 10.0 flat 08:05 46064 2.0 arc 08:29 46065 1201.0 target 08:57 46066 1502.0 target 09:24 46067 1500.0 target 09:56 46068 1501.0 target 09:59 46069 10.0 flat 10:02 46070 2.0 arc ----- sequence 46071, plate 2627 ------- 10:14 46071 10.0 flat 10:16 46072 2.0 arc 11:12 46073 1500.0 target 11:16 46074 10.0 flat 11:18 46075 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 03:37Z 2 2474 -89.88 -0.0030 55.74 0.0011 109.98 0.0415 1.000100 05:59Z 7 2566 138.07 0.0029 71.97 0.0057 -52.78 0.0212 1.000370 08:12Z 6 2624 23.01 0.0012 49.96 0.0033 199.24 0.0262 1.000430 10:19Z 8 2627 263.53 0.0007 59.33 0.0026 103.08 0.0239 1.000430 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 01:49Z 2 0 1.00000 0 1247 0 121 30.0 21.1 2 329 - N/A 03:37Z 2 2474 1.00010 -988 -4 -350 -90 55.7 20.5 5 356 - 1.6 05:59Z 7 2566 1.00037 -3654 -2296 -400 138 72.0 18.7 5 202 - 1.0 08:12Z 6 2624 1.00043 -4246 -2584 -250 23 50.0 16.7 8 131 - 2.0 10:19Z 8 2627 1.00043 -4246 -2661 -300 264 59.3 19.4 11 24 - 2.0 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 00:18Z 75 46 10 296 (WNW) 732 - Clear, photometric 00:49Z 74 46 3 271 (W) 621 - " 01:21Z 71 45 4 272 (W) 583 - " 01:54Z 69 44 0 324 (NW) 586 - " 02:26Z 69 43 5 355 (NNW) 679 - " 02:58Z 69 43 3 340 (NNW) 613 - " 03:29Z 69 43 4 353 (NNW) 645 - " 04:02Z 68 43 2 342 (NNW) 615 - " 04:32Z 67 43 4 11 (N) 570 - " 05:03Z 69 42 3 297 (WNW) 610 - " 05:34Z 67 43 3 218 (SW) 585 - " 06:04Z 65 45 2 247 (WSW) 856 - " 06:34Z 66 46 5 189 (S) 847 - " 07:05Z 65 47 7 204 (SSW) 1101 - " 07:36Z 64 48 5 182 (S) 1086 - " 08:06Z 61 48 3 138 (SE) 1025 - " 08:36Z 62 48 1 115 (ESE) 1169 - " 09:06Z 63 45 6 77 (ENE) 1138 - " 09:36Z 65 41 6 20 (NNE) 1168 - " 10:08Z 66 39 10 25 (NNE) 1224 - " 10:39Z 67 40 10 32 (NNE) 1119 - " 11:10Z 66 40 15 19 (NNE) 1137 - " 11:41Z 66 36 14 22 (NNE) 951 - 12:13Z 66 35 12 27 (NNE) 963 - ---=== TELESCOPE STATUS ===--- 01:00Z - Doors and louvers opened, fans on. 02:19Z - Enclosure off. 11:32Z - Enclosure is on, doors and louvers are closed, and fans are off. No interlocks bypassed. Status at: 11:38Z Telescope stowed at: 30,121,00 mount Instrument mounted: Cartridge 8 Counterweights at: 250 lbs LN2 autofill systems: Connected and turned on. 180L LN2 dewar scales: Spectro 181.30 lbs, 24.25 psi Imager 212.10 lbs, 22.74 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_37_0 Watcher: v2_36_0 MCP: v6_4_0 TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.9.1 2007-06-21 sdssProcedures: v2_35 SoS: v5_1_8 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.2110 -5.2130 Axial B -0.2135 -0.2135 Axial C -0.0015 -0.0016 Trans D 11.6390 11.6400 Lateral E 9.9441 9.9568 Lateral F 11.5189 11.5316 GALILS Commanded: 4600. -2000. 2650. -9300. -9800. -9800. Actual: 4600. -2000. 2650. -9300. -9800. -9800. SETMIR VALUES PriDesOrient: 0.00 0.00 0.00 0.00 0.00 PriOrient: 0.00 0.00 0.00 0.00 0.00 SECONDARY: ---------- Focus: 0.00 Air Temp.: 22.8 Alt.: 30.030315 MIGS TONIGHT NOMINAL Axial A 0.7974 0.7973 Axial B 0.6040 0.6041 Axial C 0.5425 0.5425 Trans D 0.6001 0.5999 GALILS Commanded: 1798659. 1309546. 1643428. -9450. -4400. Actual: 1798650. 1309550. 1643400. -9450. -4400. SETMIR VALUES SecDesOrient: 1257.00 -60.00 -140.00 -190.00 200.12 SecOrient: 1257.02 -60.00 -140.01 -189.47 200.13 ---=== PROBLEMS IN DETAIL ===--- See log for misc problems. SOS problems: ------------- approx. at 05:45Z was: I try command 'sos_status' some times , no effect - i.e. we have situation "The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nmsu.edu." next step: I run sos_stop and killed old jobs : [observer@sos 54333]$ kill -9 16420 [observer@sos 54333]$ kill -9 17314 [observer@sos 54333]$ kill -9 16423 [observer@sos 54333]$ kill -9 17319 , check sos_status and run sos_start . --------------------------------------------- [observer@sos 54333]$ sos_status The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nmsu.edu. The following aporsync jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 16420 1 0 69 0 - 487 11f338 21:24 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ aporsync_red.sh 000 S observer 17314 1 0 69 0 - 488 11f338 21:28 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ aporsync_blue.sh You may need to kill old jobs if there is a problem. The following startapo jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 16423 16420 0 68 0 - 495 11f338 21:24 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ startapo.sh 000 S observer 17319 17314 0 68 0 - 493 11f338 21:28 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ startapo.sh You may need to kill old jobs if there is a problem. No lock files exist. [observer@sos 54333]$ sos_status The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nmsu.edu. The following aporsync jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 16420 1 0 69 0 - 487 11f338 21:24 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ aporsync_red.sh 000 S observer 17314 1 0 69 0 - 488 11f338 21:28 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ aporsync_blue.sh You may need to kill old jobs if there is a problem. The following startapo jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 16423 16420 0 66 0 - 495 11f338 21:24 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ startapo.sh 000 S observer 17319 17314 0 68 0 - 493 11f338 21:28 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ startapo.sh You may need to kill old jobs if there is a problem. No lock files exist. [observer@sos 54333]$ kill -9 16420 [observer@sos 54333]$ kill -9 17314 [observer@sos 54333]$ kill -9 16423 [observer@sos 54333]$ kill -9 17319 [observer@sos 54333]$ sos_status The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nmsu.edu. No aporsync jobs are currently running. No startapo jobs are currently running. No lock files exist. [observer@sos 54333]$ sos_start Setting $RAWDATA_DIR=/data/spectro Setting $RAWDATA_HOST=sdsshost.apo.nmsu.edu Setting $ASTROLOG_DIR=/data/spectro/astrolog Setting $SPECLOG_DIR=/data/spectro/astrolog Setting $SPECTROLOG_DIR=/data/spectro/spectrologs Now running Son-of-Spectro version v5_1_8. [observer@sos 54333]$ sos_status Son-of-Spectro cron IS running for USER=observer MACHINE=sos.apo.nmsu.edu. Loaded version=v5_1_8. No aporsync jobs are currently running. No startapo jobs are currently running. No lock files exist. [observer@sos 54333]$ sos_redo plate=2474 Executing: echo "remove2redo,plate=2474" | idl IDL Version 5.5a (linux x86). (c) 2001, Research Systems, Inc. Installation number: 96636-4. Licensed for use by: Apache Point Observatory, NMSU % Compiled module: REMOVE2REDO. REMOVE2REDO: Selecting MJD=54333 REMOVE2REDO: Reading headers for all 4 files on MJD=54333 . FITS_PURGE_NANS: Warning: Disposing of invalid FITS header keyword IPARATE . FITS_PURGE_NANS: Warning: Disposing of invalid FITS header keyword IPARATE . FITS_PURGE_NANS: Warning: Disposing of invalid FITS header keyword IPARATE . FITS_PURGE_NANS: Warning: Disposing of invalid FITS header keyword IPARATE REMOVE2REDO: No sdR files for MJD=54333 PLATE= 2474 % Program caused arithmetic error: Floating illegal operand [observer@sos 54333]$ ............ [observer@sos 54333]$ sos_status Son-of-Spectro cron IS running for USER=observer MACHINE=sos.apo.nmsu.edu. Loaded version=v5_1_8. The following aporsync jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 22472 1 0 68 0 - 486 11f338 Aug20 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ aporsync_blue.sh 000 S observer 22476 1 0 68 0 - 487 11f338 Aug20 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ aporsync_red.sh You may need to kill old jobs if there is a problem. The following startapo jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 22479 22472 0 68 0 - 490 11f338 Aug20 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ startapo.sh 000 S observer 22490 22476 0 68 0 - 490 11f338 Aug20 ? 00:00:00 /bin/sh /local/ups/prd/idlspec2d/v5_1_8/bin/ startapo.sh You may need to kill old jobs if there is a problem. No lock files exist. [observer@sos 54333]$ --== Misc Watcher Messages ==-- MCP: comm timeout (Last packet: 2007-08-20 23:39:47Z : 2007-08-20 23:44:00Z) m1 temp (Excessive M1 - PMSS Difference:Excessive Telescope - Ambient Difference:Excessive Radial Gradient:Excessive Front-Back Difference (Entire Mirror):Excessive Front-Back Difference (Left Half):Excessive Front-Back Difference (Right Half):Excessive Front-Back Difference (Lower Half):Excessive Front-Back Difference (Upper Half): 2007-08-21 00:27:02Z ) m1 mig (Communications Fault: 2007-08-21 00:36:10Z) tmicro (Disconnected Sensor in M2B7:Disconnected Sensor in M2B6:Disconnected Sensor in M2B5:Disconnected Sensor in M2B4:Disconnected Sensor in M2B3:Disconnected Sensor in M2B2:Disconnected Sensor in M2B1:Disconnected Sensor in M2B0:Disconnected Sensor in M1B7:Disconnected Sensor in M1B6:Disconnected Sensor in M1B5:Disconnected Sensor in M1B4:Disconnected Sensor in M1B3:Disconnected Sensor in M1B2:Disconnected Sensor in M1B1:Disconnected Sensor in M1B0:Disconnected Sensor in M0B7:Disconnected Sensor in M0B6:Disconnected Sensor in M0B5:Disconnected Sensor in M0B4:Disconnected Sensor in M0B3:Disconnected Sensor in M0B2:Disconnected Sensor in M0B1:Disconnected Sensor in M0B0:Communications Fault: 2007-08-21 00:36:10Z ) m2 galil (Communications Fault: 2007-08-21 00:36:10Z) tracking (Excessive Rot Tracking Error:Excessive Az Tracking Error:Excessive Alt Tracking Error: 2007-08-21 00:24:29Z ) m1 galil (Communications Fault: 2007-08-21 00:36:10Z) currents (High Rot Drive Current:High Az #2 Drive Current:High Az #1 Drive Current:High Alt #2 Drive Current:High Alt #1 Drive Current: 2007-08-21 00:24:29Z )currents (High Rot Drive Current:High Az #2 Drive Current:High Az #1 Drive Current:High Alt #2 Drive Current:High Alt #1 Drive Current: 2007-08-21 00:24:29Z m2 temp (Excessive Front-Back Difference (Lower Half):Excessive Front-Back Difference (Upper Half): 2007-08-21 00:27:02Z ) current time=84597.003057 (2007-08-20 23:29:24Z sdssmcp) DID48 interrupt with no set time (2007-08-20 23:37:54Z sdssmcp) (DID48 interrupt with no set time (2007-08-20 23:37:54Z sdssmcp): 2007-08-20 23:38:00Z ) DID48 interrupt with no set time (2007-08-20 23:37:21Z sdssmcp) failed to get time from tai-time.apo.nmsu.edu: S sntpcLib TI (2007-08-20 23:36:19Z sdssmcp) ) reboot (The MCP has rebooted (2007-08-21 01:03:59Z sdssmcp): 2007-08-21 01:04:06Z) dewar (Imager Dewar Low Weight:Spectro Dewar Low Weight:Imager Dewar Scale Communications Fault:Spectro Dewar Scale Communications Fault: 2007-08-21 03:02:29Z ) m1 temp (Excessive M1 - PMSS Difference:Excessive Telescope - Ambient Difference:Excessive Radial Gradient:Excessive Front-Back Difference (Entire Mirror):Excessive Front-Back Difference (Left Half):Excessive Front-Back Difference (Right Half):Excessive Front-Back Difference (Lower Half):Excessive Front-Back Difference (Upper Half): 2007-08-21 03:02:29Z ) m1 mig (Communications Fault: 2007-08-21 03:02:29Z) tmicro (Disconnected Sensor in M2B7:Disconnected Sensor in M2B6:Disconnected Sensor in M2B5:Disconnected Sensor in M2B4:Disconnected Sensor in M2B3:Disconnected Sensor in M2B2:Disconnected Sensor in M2B1:Disconnected Sensor in M2B0:Disconnected Sensor in M1B7:Disconnected Sensor in M1B6:Disconnected Sensor in M1B5:Disconnected Sensor in M1B4:Disconnected Sensor in M1B3:Disconnected Sensor in M1B2:Disconnected Sensor in M1B1:Disconnected Sensor in M1B0:Disconnected Sensor in M0B7:Disconnected Sensor in M0B6:Disconnected Sensor in M0B5:Disconnected Sensor in M0B4:Disconnected Sensor in M0B3:Disconnected Sensor in M0B2:Disconnected Sensor in M0B1:Disconnected Sensor in M0B0:Communications Fault: 2007-08-21 03:02:29Z ) m2 galil (Communications Fault: 2007-08-21 03:02:29Z) m1 galil (Communications Fault: 2007-08-21 03:02:29Z) currents (High Rot Drive Current:: 2007-08-21 02:43:17Z) m2 mig (Communications Fault: 2007-08-21 03:02:29Z)