APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday March 11, 2003 (MJD 52710) ---=== OBSERVING TEAM ===--- Night: Steph Snedden, Pete Newman Development: Eric Neilsen (FNAL, by phone) Support: Jon Brinkmann (by phone this morning) Visitors: A bunch of folk from UTEP and U. Colorado. ---=== OBSERVING PLAN ===--- Shakeup before moonset (02:14MST, 09:14Z), then science if possible. ---=== OBSERVING SUMMARY ===--- Shakedown testing until moonset, during which one plate was completed (993). We got two hours of imaging: Strip 29N, -1 <= lambda <= 63 BUT we had a serious archiver problem from which we have not managed to recover. As a consequence, endNight has not been run. See the problem section for details of what happened, what we tried for recovery, and the state we are in as we give up at 7am MST and hand off to the day staff. ---=== OBSERVING LOG ===--- Shakedown ========= Counterweight 1 is disabled and parked at ~700 (near upper limit) until the cause of its high current draw is determined. The counterweight control box has been repaired from last night. We are therefore able to balance the telescope with imager or spectrograph corrector/cartridge mounted. After opening, while slewing up to 30 deg alt, watcher reported: altitude (Windscreen touched in altitude: UP (2003-03-12 01:32:20Z sdssmcp)) After filling the spectro dewars we made a slew to check fiducials. The rotator error was too large to correct, ~1060 counts, but accepting the last crossing via mcpMenu solved the problem. Items tested are described below, followed by science. ++++++++++++++++++++++++++++++++++++++++++ PR4903 - instChange hangs. Feedback -> needstest. We handed control to Eric at FNAL to debug the instChange hangs we've experienced recently. We were running IOP v3_110_0 as the test version. After his tests, suspicion for the hangs falls on the watchTCCSlew process, so a new test version, v3_111_0 was cut with that removed, which was tested without failure. The PR is now in needstest for further testing on the sky during shakedown. ++++++++++++++++++++++++++++++++++++++++++ PR 4425 - New guider image processing and centroiding. Needstest -> assigned. We tested this under version v3_111_0, with some good success but some work left to do. First, the multiGuiderDark command that establishes a library of dark frames used for the guider seems to work, but silently, so we asked Eric to include some feedback messages in the next version. After some debugging by Eric, we tested the new guider on the sky. We let it guide, and made it recover from deliberate offsets in three axes, and giving it scale errors to compute. That much works apparently well. However, we have some known problems and caveats: - We have only tested this in excellent seeing and clear sky. A real test will require poor seeing and poor signal as well. - The FWHMs reported by the new code is systematically smaller than in the old code, e.g. a median of 1.2" in the new code and 1.6" in the old when we switched code in mid plate. This is of serious concern, but it may be that the new code is more accurate! - The efficiency meter is wrong under the new code, reading ~8% in good conditions. - At the moment, we have to explicitly issue a loadGuiderDark command loads a specific exposure-length dark frame that is supposed to match the guiding exposure time. This conflicts with common practice of changing guider exposure times to match conditions. We suggest that the startGuider command automatically loads the dark frame that matches the specified expoure time. - We need to add a task to the swing-shift procedures to get new guider dark frame sets at a range of exposure times. We do not yet know how often this will be necessary, and we do not have a disposal procedure for clearing out old darks (they accumulate in /data/spectro/guiderDarks/). - Switching from between the old and new guiding codes requires a new guider flat to be taken, as the code re-uses variables that are calcuated when the flat is taken. - The SOP session died and dumped its core (what a quaint phrase!), which Eric belives may point to a problem in the underlying new C code. Because of these problems, we changed the PR back from needstest to assigned. However, we tested the _old_ guiding code under this new version of SOP, and decided to declare it current in order to at least adopt the solution to the instChange problem described earlier. ++++++++++++++++++++++++++++++++++++++++++ PR1206: endDrift questions. We are confused! The PR suggests the questions asked by endDrift should have changed, but they look the same to us! im> endDrift 2003-03-12 12:06:35Z: stopping drift with: iccExec id1 id2 id3 id4 id5 id6 ad1 ad2 ad3 ad4 readoutEnd 2003-03-12 12:07:12Z: closed shutters 2003-03-12 12:07:12Z: endData completed Flavor of this run (science, bias, calibration, engineering, ignore, help) [science]: Stripe, Strip (e.g, 22,N or SS,N) [29,N]: Mountain estimate of the overall data quality (unknown, bad, acceptable, good) [unknown]: good Please provide a one line comment [] subarcsec seeing leap seconds = 32 Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3721 04:03Z 04:52Z 100 O -60.57 -48.37 94 engineering 3722 09:25Z 09:36Z 29 N 26.95 29.65 32 ignore 3723 09:37Z 12:07Z 29 N 28.33 65.91 264 science Doing a cameraDetach spawning makeRunLog ++++++++++++++++++++++++++++++++++++++++++ Imager Shutter Tsst We ran this test in parallel with on-sky shake-up tests with the enclosure removed, the imager in the doghouse, LED illuminator mounted and lit. We started goDrift -bias, run 3721. See Imaging Details below for frame-by-frame information. ++++++++++++++++++++++++++++++++++++++++++ Hartmann focusing tests We started to test this under v3_111_0 with the new centroiding code, but it bombed (as above). At this point we were within sight of moonset, so gave up on these tests and switched to the imager to try science and at the same time test the IOP PRs. ++++++++++++++++++++++++++++++++++++++++++ Science ======= At the end of testing the guider code, we were observing on plate 993, cartridge 4, which we finished in 3600 sec of exposures. 08:40Z Change instruments to the imager. See imaging sections below for details. 11:21Z Problem with id5 tapes, although full run is in on the pool disks. See problem section #1. endNight NOT run - see problem section. There is also an unarchived bias run in the pool that needs to be written to tape. The archiver is currently paused on all nodes! ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3721 04:03Z 04:52Z 100 O -60.57 -48.37 94 engineering 3722 09:25Z 09:36Z 29 N 26.95 29.65 32 ignore 3723 09:37Z 12:07Z 29 N 28.33 65.91 264 science 3724 13:45Z 14:04Z 100 O 85.23 90.05 46 bias ---=== IMAGING RUN DETAILS ===--- Run 3721 - Jim's shutter tests ------------------------------ Imager in doghouse, T-bars unlatched, enclosure removed, LED illuminator mounted and lit. Shutter controlled from IOP using "cc openall" and "cc closeall" commands. A 'cc imstat' was issued after each change of state to confirm the shutters had indeed moved. All the photo frames marked ** transferred using getAllPhotoFrames and scp to: sdss-commish.apo.nmsu.edu:/data/scratch/jeg_shutter_tests/52710/ and compressed them there. Jim please note: as that is a shared data area, we attach an expiry date of six months to the files, after which the directory may be removed! Please arrange to transfer the files elsewhere before 2003 September 10. Frame Notes ----- ------------------------------------------------- 8 Start a goDrift -bias. cc imstat confirms all shutters are NOT_OPEN 20 cc openall just after start of frame. Pink! 22 ** Frame before close #1 23 cc closeall 25 ** Frame before open #1 26 cc openall failed - did not get control of the camera line 27 cc openall succeeded. 29 ** Frame before close #2 30 cc closeall 32 ** Frame before open #2 33 cc openall 35 ** Frame before close #3 36 cc closeall 38 ** Frame before open #3 39 cc openall 41 ** Frame before close #4 42 cc closeall 44 ** Frame before open #4 45 cc openall 47 ** Frame before close #5 48 cc closeall (late in the frame) 50 ** Frame before open #5 51 cc openall 53 ** Frame before close #6 54 cc closeall (late in the frame) 56 ** Frame before open #6 57 cc openall 59 ** Frame before close #7 60 cc closeall 62 ** Frame before open #7 63 cc openall 65 ** Frame before close #8 66 cc closeall - failed to get control of the camera line 67 cc closeall 69 ** Frame before open #8 70 cc openall 72 ** Frame before close #9 73 cc closeall 75 ** Frame before open #9 76 cc openall 78 ** Frame before close #10 79 cc closeall - failed to get control of the camera line 82 cc closeall 84 ** Frame before open #10 85 cc openall 86 cc closeall 86 endDrift ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Run 3722: 29N, IGNORE We did not get in focus before the start of the ramp up, although skippy confirmed the rotator position was good. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Run 3723: 29N, 31 <= lambda <= 63 Frame Lambda Notes ----- ------ ------------------------------------------------------------- 9 +28.5 skippy reports rotator and nuErr are ok. 13 +29.1 seeingRun reports good seeing at start of ramp. 18 +29.9 and not the focus loop has conferged, too. 25 +31.0 Start science. Seeing excellent throughout (~ 1 arcsec) 239 +63.0 End of science, start ramp down 245 +63.9 18deg twilight starts 252 +65.0 End of ramp down. ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 3722 10 4 53 5.800 0.081 2.800 0.114 -0.00473 54 74 3723 9 4 38 6.600 0.142 3.500 0.154 -0.00421 54 76 3723 19 4 51 6.400 0.083 3.400 0.100 -0.00327 57 76 3723 70 4 51 6.700 0.668 2.500 0.142 -0.00099 73 76 3723 124 4 75 7.400 0.102 0.800 0.110 0.00198 93 76 3723 177 4 74 6.100 0.797 -1.700 0.560 0.00458 114 75 3723 229 4 99 5.200 0.415 -3.300 0.121 0.00428 132 73 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 3723 29 1 76 119 20 1.408 0.196 0.207 0.184 -0.00069 3723 29 1 76 119 21 1.255 0.184 0.512 0.181 -0.00172 3723 29 1 76 119 26 1.365 0.178 0.573 0.154 -0.00193 3723 29 1 76 119 16 1.364 0.161 0.688 0.162 -0.00231 3723 29 1 76 119 15 1.350 0.139 0.602 0.115 -0.00202 3723 46 1 76 113 17 1.146 0.112 -0.378 0.287 0.00127 3723 46 1 76 113 19 1.120 0.114 -0.323 0.207 0.00108 3723 46 1 76 113 17 1.272 0.149 -0.283 0.267 0.00095 3723 46 1 76 113 18 1.141 0.156 -0.132 0.140 0.00044 3723 46 1 76 113 20 1.195 0.169 -0.093 0.131 0.00031 3723 81 1 76 101 23 0.917 0.174 -1.005 0.197 0.00338 3723 81 1 76 101 25 0.816 0.161 -0.905 0.231 0.00304 3723 81 1 76 101 20 0.851 0.153 -0.696 0.206 0.00234 3723 81 1 76 101 22 0.816 0.075 -0.834 0.238 0.00280 3723 81 1 76 101 19 0.798 0.115 -0.789 0.142 0.00265 3723 99 1 76 95 23 0.731 0.092 -1.366 0.115 0.00459 3723 99 1 76 95 23 0.624 0.134 -1.244 0.122 0.00418 3723 99 1 76 95 17 0.689 0.141 -1.134 0.201 0.00381 3723 99 1 76 95 25 0.726 0.081 -1.301 0.155 0.00437 3723 99 1 76 95 31 0.692 0.100 -1.240 0.163 0.00417 3723 134 1 76 81 22 0.236 0.163 -2.536 0.287 0.00852 3723 134 1 76 81 39 0.113 0.161 -2.544 0.249 0.00855 3723 134 1 76 81 40 0.226 0.130 -2.575 0.157 0.00866 3723 134 1 76 81 22 0.235 0.137 -2.579 0.135 0.00867 3723 134 1 76 81 23 0.157 0.144 -2.698 0.157 0.00907 3723 151 1 76 75 27 0.111 0.146 -1.986 0.107 0.00668 3723 151 1 76 75 31 0.071 0.158 -2.005 0.144 0.00674 3723 151 1 76 75 27 0.160 0.131 -2.021 0.093 0.00679 3723 151 1 76 75 19 0.027 0.136 -1.895 0.172 0.00637 3723 151 1 76 75 34 0.115 0.185 -1.956 0.232 0.00658 3723 187 1 75 61 38 0.116 0.306 -2.090 0.182 0.00703 3723 187 1 75 61 43 0.061 0.236 -1.977 0.119 0.00665 3723 187 1 75 61 50 0.151 0.304 -2.023 0.162 0.00680 3723 187 1 75 61 43 0.193 0.315 -2.097 0.186 0.00705 3723 187 1 75 61 32 0.236 0.262 -2.089 0.183 0.00702 3723 204 1 74 55 39 0.430 0.118 -2.271 0.112 0.00764 3723 204 1 74 55 47 0.463 0.097 -2.120 0.113 0.00713 3723 204 1 74 55 44 0.622 0.127 -2.228 0.148 0.00749 3723 204 1 74 55 41 0.657 0.128 -2.173 0.131 0.00730 3723 204 1 74 55 42 0.647 0.097 -2.205 0.115 0.00741 3723 241 1 72 43 49 0.127 0.154 -2.029 0.122 0.00682 3723 241 1 72 43 69 0.141 0.150 -1.913 0.096 0.00643 3723 241 1 72 43 61 0.283 0.167 -1.936 0.134 0.00651 3723 241 1 72 43 41 0.321 0.133 -1.839 0.106 0.00618 3723 241 1 72 43 56 0.289 0.175 -1.862 0.101 0.00626 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): Yes QA Procedures Done (y/n): No UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 19865, plate -9999 ------- 01:45 19865 0.0 bias ----- sequence 19866, plate 1266 ------- 03:03 19866 240.0 smear ----- sequence 19867, plate -9999 ------- 03:21 19867 0.0 bias ----- sequence 19868, plate 993 ------- 15.8 22.4 13.4 17.0 DONE 06:13 19868 10.0 flat 06:16 19869 2.0 arc ----- sequence 19870, plate -9999 ------- 06:24 19870 0.0 bias ----- sequence 19871, plate -9999 ------- 06:36 19871 0.0 bias ----- sequence 19872, plate 993 ------- 07:01 19872 900.1 target 07:24 19873 1200.1 target 07:58 19874 1500.1 target 08:05 19875 240.0 smear 08:09 19876 10.0 flat 08:11 19877 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 02:38Z 1 1266 1.00000 0 650 -467 121 90.0 6.2 13 221 03:23Z 1 0 1.00000 0 964 -150 121 90.0 6.3 16 262 06:25Z 4 0 1.00011 -1085 103 -150 -30 59.9 7.0 13 232 06:37Z 4 0 1.00000 0 1006 -150 -35 58.6 6.7 15 226 08:24Z 4 993 1.00016 -1579 -275 -180 -67 40.9 6.1 12 242 BG38 1.2 10:26Z imager 1.00000 0 -592 -449 100 76.3 5.4 9 270 i23 0.93 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:30Z 51 6 7 281 (W) 140 - Clear above 15deg 00:07Z 49 3 8 264 (W) 139 - " 00:39Z 47 6 8 272 (W) 166 - " 01:11Z 45 7 8 258 (WSW) 248 - " 01:43Z 44 9 11 247 (WSW) 291 - " 02:16Z 43 10 11 231 (SW) 313 - " 02:47Z 43 9 12 243 (WSW) 279 - " 03:17Z 43 8 14 257 (WSW) 204 - " 03:47Z 43 7 10 249 (WSW) 216 - " 04:17Z 44 3 15 253 (WSW) 204 - " 04:49Z 44 4 14 248 (WSW) 225 - " 05:22Z 44 5 14 244 (WSW) 222 - " 05:54Z 44 5 11 246 (WSW) 237 - " 06:25Z 44 5 12 231 (SW) 322 - " 06:59Z 43 4 16 230 (SW) 245 - " 07:31Z 43 1 14 234 (SW) 261 - " 08:03Z 43 1 14 241 (WSW) 340 - " 08:36Z 42 3 8 250 (WSW) 340 - " 09:08Z 42 5 9 263 (W) 448 - " 09:40Z 42 7 9 267 (W) 471 - " 10:13Z 42 7 8 268 (W) 541 - " 10:45Z 41 8 8 278 (W) 552 - " 11:17Z 41 9 9 274 (W) 514 - " 11:50Z 41 8 10 276 (W) 632 - " 12:26Z 42 7 13 304 (NW) 569 - " 12:59Z 42 7 8 312 (NW) 548 - " 13:31Z 41 7 7 315 (NW) 625 - " 14:03Z 42 7 10 326 (NW) 542 - ---=== TELESCOPE STATUS ===--- Opened at sunset, closed after sunrise. Exact times NOT recorded - sorry! Status at 13:54Z: Telescope stowed at: 30 deg Instrument mounted: Imager Counterweights at: Upper limits (CW1 in-operable) Autofill systems: On 180L LN2 dewar weights: Spectro 270 lb Imager 118 lb ---=== SOFTWARE USED ===--- IOP/SOP: v3_110_0, testing v3_111_0. Watcher: v2_21_0 MCP: v5_18_0 TPM: tpm_v2_27_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_65 SoS: v4_9_13 hoggPT: v1_6_7 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0920 0.0790 Axial B 0.7950 0.7970 Axial C 0.8220 0.8240 Trans D -9.0710 -9.1130 Lateral E 1.8840 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5377. -3699. 923. -208. 31535. 30647. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 5.9 Alt.: 29.999277 MIGS TONIGHT NOMINAL Axial A 1.5170 1.5170 Axial B 1.1560 1.1560 Axial C 1.1010 1.0990 Trans D -0.6920 -0.7070 GALILS Commanded: 1018435. 954032. 975153. -3450. -8000. Actual: 1619636. 1555273. 1576326. -3383. -7925. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1256.99 -0.02 -20.04 -0.51 130.90 ---=== PROBLEMS IN DETAIL ===--- Problem #1 - id5 problems during imaging ---------------------------------------- At 11:21:57Z, watcher started reporting errors on id5's tapes: TEXTONLY *** WARNING *** dscArch: Couldn't check tape tape ID 1 (TEXTONLY *** WARNING *** dscArch: Couldn't check_tape tape ID 1 : 2003-03-12 11:21:57Z ) archDrvSts (drive 1: Disabled: 2003-03-12 11:22:33Z) dsc E ivFrameCtx Invalid Frame context (dsc_E_ivFrameCtx Invalid Frame context : 2003-03-12 11:22:08Z) archUseDrv (drive 1: archUseDrv is 0: 2003-03-12 11:22:33Z) TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 1 (TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 1 : 2003-03-12 11:21:57Z ) archUseDrv (drive 1: archUseDrv is 0: 2003-03-12 11:26:39Z) Checking back inthe murmur log found these messages sdeconds earlier: 2003-03-12 11:21:45Z sdssid5 dArch TEXTONLY *** WARNING *** dscArch: Couldn't check_tape tape ID 1 2003-03-12 11:21:45Z sdssid1 dArch dsc_I_frameArchived Frame 3723 178.41 was archived to tapes JL6082 (1424) and JL6089 (1424) 2003-03-12 11:21:45Z sdssid5 dArch TEXTONLY *** WARNING *** dscArch: Shutting down tape ID 1 2003-03-12 11:21:46Z sdssid5 dArch dsc_E_ivFrameCtx Invalid Frame context We followed the procedure for "Handling a non-responsive archiver" in the "Archiver Checklist" section of the Troubleshooting procedures. That did not work. We repeated those steps as a check, then again after removing all the tapes and reboothing that crate (photo 2). In the end, we realised that "iccExec id5 poolDir 3723" showed frames from frame#172 to not be archived, yet "statusDump archQDepth" on id5 showed the queue length to be zero. So the archiver did have frames yet to be archived, yet it thought the depth was zero. So, we are stuck. Summary: - Frames on id1/2/3/4/6 are all archived. - Frames before 172 on id5 are all archived. - Frames from 172 on id5 are mostly not archived. - The archiver thinks it is finished, the queue depth is zero. - endNight has not been run to write the gangs, spectro data and tpm data to tape. We are hereby giving up and handing control to the day staff to attempt recover of the data obtained tonight. NB NB NB!!! In the mess that ensuedd, we did not get a bias run on the telescope. So, we paused the archiver with iccExec [vmeList] archPause 1 then started a bias run. So that should be in the queue, ready to go to tape.