APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday October 3, 2005 (MJD 53647) ---=== OBSERVING TEAM ===--- Swing: John Barentine Night: Dan Long Mike Harvanek "Stellar" Phone Support: Connie Rockosi ---=== OBSERVING PLAN ===--- Supernova science. ---=== OBSERVING SUMMARY ===--- Rain, clouds, high humidity, lightning. Failed imager calibrations (again). The mcp is reporting that the sp2 door is closed when it is open. ---=== OBSERVING LOG ===--- Afternoon --------- No problems during checkout other than a recurrence of the "can't grabInst" problem seen recently, so I did not check spectro focus numbers. Also, on Connie's request, I power-cycled the power supply to photo5 in hopes of making i5 feel better. No luck there. See Problems. Night ----- The mcp is reporting that the sp2 door is closed when it is open. Clouds, rain, lightning and humidity kept us closed. We tried imager calibrations. On the first bias we ran into the same problem that John had this afternoon with the amplifier voltages not being set correctly: Amplifier voltages are not at their nominal operating values!!!! Choose one of the following options: 1. Continue with the amplifier voltages as they are 2. Set the amplifier voltages properly and continue (I don't know if this will work properly; I recommended that you use option 4, and try again unless you know better than I do.) 3. Reset the DA and quit 4. Reset the DA, set the amplifier voltages properly, and quit 5. Quit without resetting anything We mistakenly thought that John had corrected this problem this afternoon and we were worried that the voltages had again gone awry. We called Connie and left a message. Meanwhile we chose option 4 above and then took another bias. Everything seemed okay. Connie called back shortly and after some checking, it was determined that everything was fine. We were still puzzled why the voltages needed resetting again. We talked to John later and found that he had selected option 3 when he encountered this problem and option 3 only resets the DA, not the voltages. That's why the voltages needed to be reset. We proceeded with the imager calibrations but they failed in the same manner as last night. See the Problems section for details. Note that run number 5687 is missing. Connie asked us to do a test bias after we had already started the calibrations. Since this altered the sequence of the calibrations, we stopped this sequence, incremented the run number by 1 and then started over again. Somehow, this increment in the run number caused the run number to go from 5686 to 5688 and run 5687 is missing. We're not sure how this happened. endNight finished. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 5682 20:12Z 21:28Z 100 O 24.87 43.98 142 ignore 5684 23:06Z 23:41Z 100 O 68.39 77.17 69 ignore Test after ph5 power cycle 5691 05:31Z 06:00Z 100 O 165.06 172.15 24 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): YES QA Procedures Done (y/n): NO UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 34109, plate -9999 ------- 22:00 34109 0.0 bias 22:04 34110 0.0 bias 00:31 34111 0.0 bias 00:58 34112 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 21:32Z 60 51 9 218 (SW) -9999 - Overcast 22:03Z 57 52 6 104 (ESE) -9999 - Overcast 22:34Z 55 50 11 104 (ESE) -9999 - Overcast 23:05Z 52 50 14 113 (ESE) -9999 - Overcast 23:37Z 53 48 15 112 (ESE) -9999 - Overcast 00:09Z 53 49 25 118 (ESE) -9999 - Overcast 00:40Z 50 48 21 130 (SE) -9999 - Overcast 00:56Z 51 48 14 113 (ESE) -9999 - rain 01:27Z 49 48 19 81 (E) -9999 - 01:57Z 51 46 14 112 (ESE) -9999 - 02:27Z 49 48 21 114 (ESE) -9999 - 02:57Z 52 46 15 130 (SE) -9999 - 03:27Z 52 46 17 108 (ESE) -9999 - 03:57Z 53 47 19 123 (ESE) -9999 - 04:28Z 54 47 17 127 (SE) -9999 - 04:58Z 52 48 10 88 (E) -9999 - 05:28Z 51 49 0 355 (NNW) -9999 - 05:58Z 50 49 7 85 (E) -9999 - 06:28Z 51 49 14 112 (ESE) -9999 - 07:01Z 53 49 16 138 (SE) -9999 - 07:33Z 49 50 13 206 (SSW) -9999 - 08:05Z 49 50 7 42 (NE) -9999 - 08:39Z 51 50 10 110 (ESE) -9999 - 09:09Z 51 49 3 87 (E) -9999 - 09:41Z 52 49 8 150 (SSE) -9999 - 10:14Z 51 49 15 180 (S) -9999 - 10:46Z 50 49 1 188 (S) -9999 - ---=== TELESCOPE STATUS ===--- Never opened (clouds, rain, humidity, lightning). Telescope stowed at 30 degrees with cartridge 4 mounted. Counterweights at 260. Spectro auto-fill connected. Spectro dewar 10:55 250 lb, 21 psi Imager dewar 10:55 178 lb, 20 psi No interlocks bypassed. ---=== SOFTWARE USED ===--- IOP/SOP: v4_5_0 Watcher: v2_30_2 MCP: v5_26_0 TPM: tpm_v3_1_0 AstroDa: v15_4 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_05 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.8420 -5.8440 Axial B -5.6730 -5.6770 Axial C 0.9410 0.9420 Trans D -9.9910 -9.9940 Lateral E 10.3759 10.3759 Lateral F 11.9380 11.9380 GALILS Commanded: 4700. -2450. 3000. -7800. 3300. 3000. Actual: 4711. -2457. 2997. -7825. 3296. 3024. SETMIR VALUES PriDesOrient: 0.00 -1.68 7.05 385.25 92.02 PriOrient: 0.00 -1.52 7.02 384.99 93.81 SECONDARY: ---------- Focus: 0.00 Air Temp.: 11.4 Alt.: 29.999997 MIGS TONIGHT NOMINAL Axial A 1.5540 1.5420 Axial B 1.0590 1.0470 Axial C 1.1850 1.1650 Trans D -0.0990 -0.1240 GALILS Commanded: 1593406. 1603674. 1554282. -6800. -5900. Actual: 1593447. 1603642. 1554316. -6820. -5882. SETMIR VALUES SecDesOrient: 1257.00 15.87 0.00 -50.00 136.76 SecOrient: 1256.99 15.88 0.02 -49.54 136.73 ---=== PROBLEMS IN DETAIL ===--- The Saga Of i5's Bias Level --------------------------- Connie asked me to power cycle the supply to photo5, then take a bias to assure things had reset themselves properly. After a "cc cold" and an "iack", I attemped to start a bias drift. The control room screens went all blue but nothing appeared. I got this message in my iop session: The old teamster, PID 12108, appears to have died; restarting Amplifier voltages are not at their nominal operating values!!!! Choose one of the following options: 1. Continue with the amplifier voltages as they are 2. Set the amplifier voltages properly and continue (I don't know if this will work properly; I recommended that you use option 4, and try again unless you know better than I do.) 3. Reset the DA and quit 4. Reset the DA, set the amplifier voltages properly, and quit 5. Quit without resetting anything Since to my knowledge the voltages were okay, I picked number 3, then got: Resetting DA (using iccExec $active readoutAbort)... done. 2005-10-03 23:02:31Z: beginData has failed: im: Aborting cameraStart due to improperly set amplifier voltages Error: goDrift ends with error in beginData Tried another goDrift with the same result. I got information up to id5: im> goDrift -bias 2005-10-03 23:05:40Z: No heartbeat found from ad1 in last 500 of murmur log 2005-10-03 23:05:40Z: 2005-10-03 23:05:40Z: Murmur not working for all ad1 node - Losing information!! 2005-10-03 23:05:40Z: No heartbeat found from ad2 in last 500 of murmur log 2005-10-03 23:05:40Z: 2005-10-03 23:05:40Z: Murmur not working for all ad2 node - Losing information!! 2005-10-03 23:05:40Z: No heartbeat found from ad3 in last 500 of murmur log 2005-10-03 23:05:40Z: 2005-10-03 23:05:40Z: Murmur not working for all ad3 node - Losing information!! 2005-10-03 23:05:40Z: No heartbeat found from ad4 in last 500 of murmur log 2005-10-03 23:05:40Z: 2005-10-03 23:05:40Z: Murmur not working for all ad4 node - Losing information!! 2005-10-03 23:06:05Z: Beginning data taking leap seconds = 32 Amplifier voltages are not at their nominal operating values!!!! before it quit. I did another iack on the basis of what it said in folklore, and all of a sudden I had something running on every photo screen. After adjusting scd to display "All chips", I started getting reasonable biases running on every monitor (Run 5684). The bias banding on i5 was still looking bad, but I let it run for a bit. At the end of this drift I got another error: im> filling out runlog /astrolog/53647/runlog-5684.txt for run 5684. makeRunLog failed for run 5684. I have the following diagnostics plus check the murmur log and your screen history: Not enough elements to do sigma (Not sure if this is a problem). Connie called back just after 00:00Z, I reported the test results, and she decided that no new knowledge came about as a result of the test. Spectro Focus ------------- Motor numbers before focusing: sp1 Coll_motor_A 1259 Coll_motor_B 1258 Coll_motor_C 1756 sp2 Coll_motor_A 8514 Coll_motor_B 5447 Coll_motor_C 6008 Temperatures at time of focus check: tpm_TM_PMSSAvg 12.8707 tpm_TM_TelairAvg.VALA 3.05 Tried to do a postCalib but could not grabInst: sp2> grabInst -handlers 900 IOP running as Backup Relay Server ************************************************** ====>>>>>>RELAY SERVER FAILED TO START!!!!<<<===== ************************************************** requestListen spectro: Neither 30310 nor 30307 appears to exist ************************************************** requestListen spectro: Neither 30310 nor 30307 appears to exist requestListen spectro: Neither 30310 nor 30307 appears to exist while executing "error "requestListen $type: Neither $port1 nor $port2 appears to exist"" invoked from within Same error as before. It's looking highly unlikely that we'll open tonight anyway. Dan found the same error when he tried to grabInst from his sop session. He restarted the servers and tried again, without any luck. But he did notice that a grabInst issued without any arguments did *something* (looks to be equivalent to "grabInst activeInstruments -handlers 0 -deadtime 60"). Then he was able to take a bias without an init. Failed imager calibrations: --------------------------- Imager calibrations (u5 leak) were attempted again tonight and failed in nearly the exact same manner and place as last night. I am 100% certain that I did not type anything in the iop window where the script was running. Here is the last successful frame: --------------------------------------------------------------------- stageX 387100 lambda 710.0 time 30.000 5688 19 u5 im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 03:51:46Z: opening camera doors 2005-10-04 03:53:03Z: closing camera doors 2005-10-04 03:53:03Z: issuing stop to im im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 03:53:05Z: Creating/Updating report files and headers leap seconds = 32 2005-10-04 03:53:05Z: scheduling gangs2Unix transfers 2005-10-04 03:53:07Z: finished with iccExec -riskyOn data -async id1 id2 id3 id4 id5 id6 ad1 ad2 a d3 ad4 readoutBegin im cameraStart {} t-g-sdss-2.apo.nmsu.edu 3800 Waiting for read to finish Run 5688 Frame 19 getFrameMSP 5688 19 35 /data/photo/doi/calibdata/idR-5688-u5-19.fit completed --------------------------------------------------------------------- Here is where the problems begin. Note the missing stageX, lambda, and time info for this frame. 5688 20 u5 im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 03:54:54Z: opening camera doors 2005-10-04 03:55:12Z: closing camera doors 2005-10-04 03:55:12Z: issuing stop to im im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 03:55:13Z: Creating/Updating report files and headers leap seconds = 32 2005-10-04 03:55:13Z: scheduling gangs2Unix transfers 2005-10-04 03:55:15Z: finished with iccExec -riskyOn data -async id1 id2 id3 id4 id5 id6 ad1 ad2 a d3 ad4 readoutBegin im cameraStart {} t-g-sdss-2.apo.nmsu.edu 3800 Waiting for read to finish Run 5688 Frame 20 IOP running as Backup Relay Server requestListen: listening 4 2005-10-04 03:56:17Z 11 im camcheck 2005-10-04 03:56:19Z 11 im ln2stat 2005-10-04 03:56:20Z: registered (addr11) listener sdsshost2.apo.nmsu.edu:observer:28098 getFrameMSP 5688 20 35 /data/photo/doi/calibdata/idR-5688-u5-20.fit completed --------------------------------------------------------------------- Here is an "unsolicited response" and a "ping error 4" before the next frame begins: MSP: Unsolicited response "flux -2211.0 aflux 32216699.8 volts 0.0001 avolts -0.9398 vo ltsRms 0.0012 avoltsRms 0.0021 ping stageX 387100 lambda 720.0 time 30.000 stageX 387100 lambda 720.0 time 30.000 flux -2930.3 aflux 32724809.0 volts 0.0001 avolts -0.9709 voltsRms 0.0011 avoltsRms 0.0021 " ping error 4 stageX 387100 lambda 730.0 time 30.000 5688 21 u5 im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 03:57:09Z: opening camera doors flux 26472.1 aflux 33188200.2 volts -0.0007 avolts -0.9989 voltsRms 0.0012 avoltsRms 0.0022 2005-10-04 03:58:25Z: closing camera doors 2005-10-04 03:58:25Z: issuing stop to im im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 03:58:26Z: Creating/Updating report files and headers leap seconds = 32 2005-10-04 03:58:26Z: scheduling gangs2Unix transfers 2005-10-04 03:58:28Z: finished with iccExec -riskyOn data -async id1 id2 id3 id4 id5 id6 ad1 ad2 a d3 ad4 readoutBegin im cameraStart {} t-g-sdss-2.apo.nmsu.edu 3800 Waiting for read to finish Run 5688 Frame 21 getFrameMSP 5688 21 35 /data/photo/doi/calibdata/idR-5688-u5-21.fit completed --------------------------------------------------------------------- At this point, we noticed that dewar 5 frames were not being read out on the scrolling display monitor: stageX 387100 lambda 740.0 time 30.000 5688 22 u5 im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 04:00:05Z: opening camera doors flux 24086.5 aflux 33276031.9 volts -0.0006 avolts -1.0159 voltsRms 0.0012 avoltsRms 0.0021 2005-10-04 04:01:21Z: closing camera doors 2005-10-04 04:01:21Z: issuing stop to im im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3800 2005-10-04 04:01:22Z: Creating/Updating report files and headers leap seconds = 32 2005-10-04 04:01:23Z: scheduling gangs2Unix transfers 2005-10-04 04:01:24Z: finished with iccExec -riskyOn data -async id1 id2 id3 id4 id5 id6 ad1 ad2 a d3 ad4 readoutBegin im cameraStart {} t-g-sdss-2.apo.nmsu.edu 3800 Waiting for read to finish I did a bunch of ctrl c's here to end the script and got the following: Access to shared data area at 0x18000000 lost (errno=0x4 Interrupted system call) Read failed from TCP icc30 (sdssad1) rmt=0x18000000 cnt=0x0009ABE8 ******************************************************************************** * * * Communication between the ICC/OAC is no longer possible. * * * * Perform the following steps until operations are possible. * * * * 1. Check network connectivity between the machines. * * 2. Reboot (reset) the ICC. * * 3. Restart this (OAC) program. * * 4. Power-cycle the VMEbus crates. * * 5. Reboot the local machine (host). * * * ******************************************************************************** 22:01 sdsshost2.apo.nmsu.edu % We attempted to continue where the script died. However, the edited script would not run until we took another initializing bias (note that we had been kicked out of iop by the failure). We did this but then the dewar 5 frames were still not reading out. The initializing bias also caused the run number to be incremented and to continue we would have had to run the original script from the beginning. Since this original script has failed twice in a row in the same manner in nearly the same place, we decided against starting it again. We did another staring bias and this time all the frames read out. We followed this with a regular scrolling bias and it looked okay. Note that frames 20 and 21 were deleted from /data/photo/doi/calibdata in preparation for continuing with the edited script.