APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday October 31, 2005 (MJD 53675) ---=== OBSERVING TEAM ===--- Howard Brewington (swing) Mike Harvanek (night) Jurek Krzesinski (night) Josh Frieman (by phone) Eric Neilsen (by phone) Margaret Votava (by phone) Jim Gunn (by phone) ---=== OBSERVING PLAN ===--- superNova science and whatever else we can get. ---=== OBSERVING SUMMARY ===--- 82 N from lambda +115, RA 300 to lambda -126, RA 59 (including ramps); 799 frames in photometric and moonless skies in mostly good seeing. We got usable data on one plate (2266) but did not finish it. We had a port conflict involving murmur which resulted in phone calls to both Eric Neilsen and Margaret Votava in the wee hours of the morning. We had to reboot host2 to solve the problem. Many thanks to Eric and Margaret for the late night support. The partial signal for plate 2266 did not show up on the "plan a night's observing" page. It did show up in the database however. Dewar 4 on the imager is not filling. We forgot to check the spectro flow meter when we had the telescope at zenith when we changed instruments in the middle of the night. We checked when we had the telescope at stow this morning (30 degrees) but the reading was obviously bad (10+) due to the orientation of the telescope. ---=== OBSERVING LOG ===--- Afternoon: ---------- The afternoon checkout was uneventful. The specto focus was checked but not adjusted since the numbers were in spec; see Problem Section for collimation results. There was no time to do the specFlats today since the time change back to standard. Sorry. Night: ------ The dust counter is working again. Thanks day staff. Run 5799: On-telescope bias in darkened enclosure. Everything looked okay. Run 5800: 82 N, start lambda +115, RA 300. Photometric, no moon. Ramp started at 01:26Z, frame 24, lambda +118, RA 303. Everything good from the beginning. Seeing between 1.3" and 1.6". Science begins at 01:34Z, frame 37, lambda +120, RA 305. Seeing went up to 2" starting around frame 333, lambda 164.2, RA 349. Seeing went back down to around 1.4" about frame 409, lambda 175.7, RA 0.7 and remained good for the rest of the run. Ramp started at 09:01Z, frame 786, lambda -128, RA 57. Run ended at 09:06Z, frame 799, lambda -126, RA 59.. Around 07:20Z we had a problem with the PT where the image did not read out and the GUI timed out (see the PT log). At that time we started seeing /linuxp/prd/astroda/v15_6/bin/dscMurPipe: Got unexpected retval=0 on hdr read. Expected 6 bytes and only that in the murmur log. Shortly thereafter we had more problems with the murmur server. Clicking on the focus button in logTool output the following error to the iop session running the logTool: im> -- MURMUR CLIENT STATUS : "Can't SEND to MURMUR server - IO error" -- Errno message : -- MURMUR CLIENT STATUS : "Can't SEND to MURMUR server - IO error" -- Occurred when trying to report message : <800100fb> Errno message : mur_send failure, error code 0x800001fa Josh Frieman then called wanting to know if we had stopped the supernova scan. We had not but apparently gang transfer had stopped. We forced the teamster and this appeared to restart gang transfer but this did not help our murmur problems. We tried to stop and then start the murmur server. First we used dscRun stop/start_mursrvr (07:53Z) and then just stop/start_mursrvr (no dscRun; 07:56Z) but this didn't help. We then tried to restart the servers using start_servers -K. The existing servers wouldn't die though. We issued an exit in each of the 3 server windows and then brought them back with the start_servers -K command. They all gave the same "Can't SEND to MURMUR server" shown above. At this point we gave up and called Eric Neilsen (sorry Eric). He asked us to try a cleanupMurmur (08:11Z) but it didn't work. He then tried a cleanupMurmur himself (08:14Z?) but it didn't work for him either. After investigating for quite a while, Eric recommended that we call Margaret Votava which we did (sorry Margaret). With Eric on one speaker phone and Margaret on the other, they investigated the problem for quite some time. We tried to restart the watcher servers again because they were owned by Jon Brinkmann and not observer. (Jon, why did you log into the previously set up server window as yourself? Please leave the server window user as observer.) Unfortunately, this did not solve the murmur server problem. After further investigation, it was decided that rebooting host2 was the next thing to try. We had just ended the supernova drift and so host2 was rebooted at that time (09:15Z). This solved the problem. Apparently, there was a port conflict involving murmur that messed up a lot of stuff. The cause was never figured out but rebooting host2 solved the problem. Thank you very much to Eric and Margaret for their late night support. We lost about 35 minutes to the host2 reboot. Run 5801: Attempting Segue 1356 S. Failed due to murmur problems: when murmur wasn't working we tried to get to the next stripe, but we saw the following errors: scanStripe 1356 S -mu 50 im> goDrift -quick Flavor of this run (science, bias, calibration, engineering, ignore, help) []: science Stripe, Strip (e.g, 22,N or SS,N) [82,N]: 1356,S Please provide a one line comment [] SEGUE 2005-11-01 09:16:04Z: Beginning data taking 2005-11-01 09:16:07Z: idReport: problem in calcStripeParams, setting many params to NaN. Error was: domain error: argument not in valid range continuing with idReport anyway we saw the sky scrolling, but since we expected a lot of problems with fits file headers and restoring information we ended the drift. Run 5802: ??? Sorry. Things were hectic and nobody remembers what this drift was for. We lost logTool and were keeping the log on sticky notes. This run did not make it to the sticky notes. Nothing useful anyway. Run 5803: Trying Legacy 18 S. We ran for 56 frames before terminating this run due to poor seeing. We are switching to spectroscopy. We billed about 45 minutes to "other weather problem" (seeing). Plate 2266 (cartridge 6): Survey. Guide stars on the gotoField. We got 3 useful exposures (3900 sec total) and a 4th exposure that was not useful (s/n^2 on b1 and r2 were too low). SoS warned of whopping fibers and bad sky residuals. Watcher sent the following warning: primary ln2 fill time (FAULT!!!: 2005-11-01 11:27:20Z) We did a cc stopfillseq/startfillseq. However, the error came back: primary ln2 fill time (FAULT!!!: 2005-11-01 11:36:35Z) Jurek went down to the telescope (where he could hear what was going on) and reissued the stopfillseq/startfillseq commands. He heard the imager fill start and then terminate. ltMatch did not run on run 5800 for some reason. endNight had problems with missing files. We did not continue and ran copyUnstoredFilesInIdReport as suggested but got the same message: Continue? y Run 5800 has 1 frames missing from disk. Note that binned scans are not supposed to have gangs, but I'm not yot smart enough to distinguish binned and unbinned scans. If files that really should be there are missing, perhaps you should stop here, run copyUnstoredFilesInIdReport, and rerun endNight. Continue with endNight? n This time we continued. endNight finished. Eric was aware of the potential for endNight failure and will hopefully check on this later. Spooling failed. Josh Frieman was called. The partial signal for plate 2266 did not show up on the "plan a night's observing" page. It did show up in the database however. The watcher primary ln2 fill time fault occurred again: primary ln2 fill time (FAULT!!!: 2005-11-01 12:56:47Z) This time it was followed by a critical error: ln2 empty (DEWAR_2 DEWAR_4: 2005-11-01 12:59:21Z) Jim Gunn was called. Dewar 4 is not filling and is warming up. We attempted to call both Wendell Jordan and Steve Bastian but could reach neither. We will relay the message either verbally or by note. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 5798 22:27Z 22:48Z 100 O 86.35 91.64 48 ignore 5799 00:30Z 00:47Z 100 O 117.05 121.35 43 bias on-telescope 5800 01:16Z 09:12Z 82 N 115.51 -125.02 812 science good 5801 09:16Z 09:17Z 1356 S -10184.00 -134.42 16 ignore murmur problems 5802 09:33Z 09:37Z 100 O -106.72 -105.78 20 ignore ??? 5803 09:44Z 10:14Z 18 S -68.50 -61.00 64 ignore bad seeing ---=== IMAGING RUN DETAILS ===--- See the main log. ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 5800 23 4 103 0.200 0.160 -2.000 0.135 -0.00026 203 55 5800 75 4 111 0.900 0.134 -1.700 0.134 -0.00044 203 55 5800 126 4 88 1.200 0.115 -1.300 0.131 -0.00011 203 55 5800 176 4 80 2.000 0.114 -0.300 0.118 0.00058 203 55 5800 228 4 44 2.100 0.112 -0.300 0.136 -0.00036 203 55 5800 279 4 46 2.500 0.096 -0.300 0.145 0.00007 203 55 5800 331 4 49 2.700 0.566 -0.300 0.742 0.00189 203 55 5800 382 4 44 2.900 0.139 -0.500 0.136 -0.00011 203 55 5800 432 4 65 2.600 0.101 -0.900 0.120 0.00028 203 55 5800 484 4 47 3.200 0.108 -1.200 0.109 0.00014 203 55 5800 535 4 47 3.000 0.096 -2.200 0.161 0.00088 203 55 5800 587 4 41 3.100 0.109 -2.900 0.121 0.00053 203 55 5803 25 4 121 -2.000 0.218 4.600 0.149 0.00437 121 50 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 5803 35 1 50 60 58 -1.397 0.188 0.367 0.248 -0.00123 5803 35 1 50 60 45 -0.879 0.230 0.518 0.207 -0.00174 5803 35 1 50 60 53 -0.476 0.246 0.625 0.223 -0.00210 5803 35 1 50 60 46 -0.243 0.224 0.804 0.215 -0.00270 5803 35 1 50 60 53 0.110 0.264 0.860 0.185 -0.00289 5803 45 1 51 61 46 -1.307 0.298 0.560 0.197 -0.00188 5803 45 1 51 61 58 -1.029 0.279 0.541 0.216 -0.00182 5803 45 1 51 61 50 -0.412 0.320 0.401 0.262 -0.00135 5803 45 1 51 61 55 -0.142 0.379 0.575 0.287 -0.00193 5803 45 1 51 61 48 0.402 0.358 0.354 0.415 -0.00119 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): YES QA Procedures Done (y/n): YES UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 34444, plate -9999 ------- 22:29 34444 0.0 bias ----- sequence 34445, plate 2266 ------- 22:48 34445 10.0 flat 22:50 34446 2.0 arc 22:58 34447 2.1 arc 23:01 34448 2.1 arc ----- sequence 34449, plate -9999 ------- 00:02 34449 0.0 bias 00:10 34450 0.0 bias 09:34 34451 0.0 bias 10:02 34452 0.0 bias ----- sequence 34453, plate 2266 ------- 7.7 14.5 24.2 11.7 not done 10:38 34453 10.0 flat 10:40 34454 2.0 arc 10:59 34455 900.1 target 11:27 34456 1500.1 target 11:54 34457 1500.1 target 12:17 34458 1200.1 target ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 01:29Z imager -23.28 0.0000 55.10 0.0000 70.35 0.0000 1.000000 11:01Z 6 2266 49.61 0.0000 64.42 0.0000 221.38 0.0000 1.000180 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 01:29Z imager 1.00000 0 -179 -80 -23 55.1 4.0 10 323 i'23 1.3" 03:11Z imager 1.00000 0 -235 -136 -23 55.1 3.3 10 315 i'23 1.2" 05:20Z imager 1.00000 0 -302 -203 -23 55.1 3.8 8 22 i'23 1.2" 06:17Z imager 1.00000 0 -434 -334 -23 55.1 4.9 9 20 i'23 1.2" 08:07Z imager 1.00000 0 -288 -189 -23 55.1 4.3 18 355 i'23 1.1" 11:01Z 6 2266 1.00018 -1778 -259 70 49 64.5 5.4 20 27 ---- 1.7" ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:13Z 46 22 6 261 (W) 1396 - 22:44Z 43 21 8 294 (WNW) 1378 - 23:15Z 42 20 5 290 (WNW) 1392 - 23:46Z 39 19 9 296 (WNW) 1387 - 00:17Z 37 18 1 334 (NNW) 1237 - clear 01:06Z 37 18 6 352 (NNW) 1242 - clear 01:38Z 39 14 1 352 (NNW) 1172 - clear 02:09Z 38 15 6 309 (NW) 1148 - clear 02:39Z 38 16 4 340 (NNW) 1227 - clear 03:10Z 38 15 7 317 (NW) 1229 - clear 03:41Z 37 17 8 307 (NW) 1300 - clear 04:11Z 36 17 9 330 (NNW) 1431 - clear 04:42Z 37 17 1 0 (N) 1156 - clear 05:13Z 39 7 11 0 (N) 997 - clear 05:43Z 39 5 10 44 (NE) 891 - clear 06:14Z 41 5 8 7 (N) 1260 - clear 06:46Z 41 6 15 25 (NNE) 1470 - clear Missing info due to murmur problem. 10:00Z 41 1 15 3 (N) 1414 - clear 10:31Z 41 4 9 48 (NE) 1410 - clear 11:01Z 41 7 23 15 (NNE) 2228 - clear 11:31Z 41 7 19 24 (NNE) 2646 - clear 12:01Z 42 7 11 41 (NE) 2736 - clear 12:33Z 42 7 16 34 (NE) 2251 - clear 13:05Z 42 7 13 32 (NNE) 1829 - clear ---=== TELESCOPE STATUS ===--- 23:33Z Doors cracked, louvers opened, and fans on. 00:40Z Enclosure off. 12:30Z Enclosure on, doors closed, fans and blowers off, louvers closed. Telescope stowed at 30 degrees with cartridge 6 (plate 2266) mounted. Counterweights at 245. Spectro auto-fill connected. Spectro dewar 13:16 72 lb, 22 psi Imager dewar 13:16 199 lb, 20 psi No interlocks bypassed. ---=== SOFTWARE USED ===--- IOP/SOP: v4_6_0b Watcher: v2_30_2 MCP: v5_26_0 TPM: tpm_v3_1_0 AstroDa: v15_6 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_06 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.8400 -5.8440 Axial B -5.6830 -5.6770 Axial C 0.9400 0.9420 Trans D -9.9810 -9.9940 Lateral E 10.3759 10.3759 Lateral F 11.9507 11.9380 GALILS Commanded: 4700. -2450. 3000. -7800. 3300. 3000. Actual: 4700. -2450. 3000. -7800. 3300. 3000. SETMIR VALUES PriDesOrient: 0.00 -1.68 7.05 385.25 92.02 PriOrient: 0.00 -1.52 7.02 384.99 93.81 SECONDARY: ---------- Focus: 0.00 Air Temp.: 7.6 Alt.: 30.001630 MIGS TONIGHT NOMINAL Axial A 1.5420 1.5420 Axial B 1.0500 1.0470 Axial C 1.1700 1.1650 Trans D -0.1080 -0.1240 GALILS Commanded: 1593413. 1603951. 1554333. -6800. -5900. Actual: 1593450. 1603650. 1554300. -6800. -5900. SETMIR VALUES SecDesOrient: 1257.00 15.87 0.00 -50.00 136.76 SecOrient: 1257.08 15.93 0.09 -49.49 136.70 ---=== PROBLEMS IN DETAIL ===--- ---=== Misc Watcher Messages ===--- tmicro (Disconnected Sensor in M2B4:: 2005-10-31 22:26:37Z) dsc E archTaskErr Archive task error: FTP connection is hung. (dsc_E_archTaskErr Archive task error: FTP connection is hung. : 2005-10-31 22:35:52Z) --== Spectro Focus Numbers ==-- IDL 5.5a> collimate, 34447 Reading /data/spectro/53675/sdR-b1-00034447.fit Reading /data/spectro/53675/sdR-b1-00034448.fit COLLIMATE: Min offset = -0.80 pix COLLIMATE: Max offset = 0.76 pix COLLIMATE: RMS across CCD = 0.39 pix COLLIMATE: Mean offset = 0.03 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict blue camera ring movement of -1.1 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-53675-b1-00034447.log PostScript file = Collimate-53675-b1-00034447.ps Reading /data/spectro/53675/sdR-b2-00034447.fit Reading /data/spectro/53675/sdR-b2-00034448.fit COLLIMATE: Min offset = -0.53 pix COLLIMATE: Max offset = 0.07 pix COLLIMATE: RMS across CCD = 0.16 pix COLLIMATE: Mean offset = -0.21 pix COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 7.4 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-53675-b2-00034447.log PostScript file = Collimate-53675-b2-00034447.ps Reading /data/spectro/53675/sdR-r1-00034447.fit Reading /data/spectro/53675/sdR-r1-00034448.fit COLLIMATE: Min offset = -0.25 pix COLLIMATE: Max offset = 0.66 pix COLLIMATE: RMS across CCD = 0.25 pix COLLIMATE: Mean offset = 0.13 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -1164 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston -1164" Log file = Collimate-53675-r1-00034447.log PostScript file = Collimate-53675-r1-00034447.ps Reading /data/spectro/53675/sdR-r2-00034447.fit Reading /data/spectro/53675/sdR-r2-00034448.fit COLLIMATE: Min offset = -0.31 pix COLLIMATE: Max offset = 0.33 pix COLLIMATE: RMS across CCD = 0.17 pix COLLIMATE: Mean offset = 0.11 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -1037 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston -1037" Log file = Collimate-53675-r2-00034447.log PostScript file = Collimate-53675-r2-00034447.ps IDL 5.5a> See the main log for problems with murmur. ------------------------------------------ Dewar 4 is not filling on the imager. -------------------------------------