APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday May 11, 2008 (MJD 54598) ---=== OBSERVING TEAM ===--- Night: Dan Oravetz, Dmitry Bizyaev Swing: Kaike Pan ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Night was dusty, partly cloudy, with sometimes awful seeing. We collected more signal with plate 2922 (cart. 1) The following MSP Imager calibration completed: QE z 2; QE z 3 See the "problems" section about new teamster problem. Let we rise the following question: In 1 or 2 cloudy nights we'll finish QE and Lin tests for "r", "i", and "z" filters. Of them, QE test will be done for z 4,5, and 6, this is some 6 hours to go. Do we have a clear picture if we can go ahead and simply continue with "u" and "g"? Or we are waiting for an answer? We collected ~ 18K dust when opened. Please check the mirror. During the night total accumulated dust from 20h to 5 h was ~ 50K (both opened and closed). Please check both imager and spectro LN2 dewars, bot are under 100 lbs. ---=== OBSERVING LOG ===--- Afternoon: ----------- All systems were checked, and working properly. The spectro focus was set for about 9 C. Data for mjd 54586-89 were removed. Night: ------ 02:10Z enclosure opened Cart 1 plate 2922 (seguefaint) is on Dust count is pretty high (~4000), we keep an eye on it. No pollen is noticed. 02:39Z GSOGTF 02:51Z precal Precal looks good 03:11Z science 1500 03:39Z science 1500 04:17Z midcal 04:24Z science 1800 postcal 05:05Z dust is continuing to rise, it's already 7000. We have clouds in the field, efficiency is low. We decide to close the dome because [low] scientific output is not worth collecting this dust. 05:15Z dome closed --- 05:23Z imager prepared for calibrations Run 7492 05:30Z fr.1 - bias **05:31Z "QE z 2" fr. 2 - 39 (07:17Z) --- Dust finally goes down, not it's at 5500. Sky is clear. Moon will set soon. ... Oops, after writing this line we got 7500 dust again, seeing jumps up to 5.1" (DIMM) We manage to continue with the imager calibrations --- **07:26Z "QE z 3" 40 - 51 and it failed after that, during fr. 52 The following is the error message: ------------------------------------- getFrameMSP 7492 51 43 /home/observer/calibdata/idR-7492-z3-51.fit completed --------------------------------------------------------------------- WARNING! We receive flux too low even after a retry! stageX 205100 lambda 910.0 time 4.000 7492 52 z3 im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3500 2008-05-12 07:53:34Z: opening camera doors flux -20.7 aflux 6331719.8 volts 0.0000 avolts -1.7231 voltsRms 0.0010 avoltsRms 0.2404 2008-05-12 07:53:58Z: closing camera doors 2008-05-12 07:53:58Z: issuing stop to im im cameraNag {} t-g-sdss-2.apo.nmsu.edu 3500 2008-05-12 07:53:59Z: Creating/Updating report files and headers leap seconds = 32 2008-05-12 07:53:59Z: scheduling gangs2Unix transfers There already is a teamster, PID 7636, owned by dmbiz 2008-05-12 07:54:00Z: finished with iccExec -riskyOn data -async id1 id2 id3 id 4 id5 id6 ad1 ad2 ad3 ad4 readoutBegin beginData has failed: waited too long for the daqBusy Error im> Exposure is Paused, this is camHandler running 2008-05-12 07:54:14Z 26 im ln2stat Exposure is Paused, this is daHandler running IOP running as Backup Relay Server requestListen: listening 4 Exposure is Paused, this is camHandler running 2008-05-12 07:54:20Z 27 im camcheck Exposure is Paused, this is daHandler running 2008-05-12 07:54:22Z: registered (addr14) listener sdsshost2.apo.nmsu.edu:observ er:32386 im> im> mspSend -send ping Error: Invalid MSP command: -SEND PING (reply: error) im> mspSend -echo ping ping im> mspSend -echo status stageX 205100 lambda 910.0 time 4.000 im> mspSend -echo expose flux -248.6 aflux 6295709.7 volts 0.0001 avolts -1.7133 voltsRms 0.0010 avoltsRms 0.2376 im> getQeCCD z 3 QE z 3 stageX 205100 lambda 910.0 time 4.000 7492 53 z3 2008-05-12 07:59:50Z: statusEntry daqBusy on node id1 is not 0 but 1 Error: Aborting: system is already busy ------------------------------------------------------ Frame 51 corresponds to the line: "910. 4.0" in the file "qe_z.list" Probably the orogon of the problem is in dead teamster, cause we restarted it and got the message: im> teamster -force 60 The old teamster, PID 7636, appears to have died; restarting 19619 The problem did not disappear after the teamster was restasted. Attempting to restart the session, we get an error that daqBusy. There are some data left in the pool (poolDir confirms it), apparently from broken exposure #52. Attempt to initialize pool according to procedure failes: it sais we can't do it while daq is busy. 08:18Z we rebooted photo crates. mspSend -echo ping | status,expose,etc do not return anything now taking regular bias as a sanity check... no, ad1 is busy astro crate rebooted... now we are able to take normal bias: run 7493 --> ignore mspSend commands do not return anything again All hard/soft configuration for MSP calibration initialized from scratch again in enclosure and in the control room. 08:59Z MSP bias started, run 7494, frame 1 **Continue QE z 3 from the line "910. 4.0": frame 2-27 Teamster got stuck again during frame 16. We restarted the teamster and calibration continued without interruption. 11:00Z endNight rejoiced --- ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 7491 23:38Z 23:57Z 100 O -66.24 -61.34 46 ignore 7493 08:33Z 08:37Z 100 O 67.85 68.83 21 ignore 7495 10:47Z 10:50Z 100 O 101.59 102.20 18 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 52731, plate -9999 ------- 23:44 52731 0.0 bias ----- sequence 52732, plate 2922 ------- 23:56 52732 10.0 flat 23:59 52733 2.0 arc 00:03 52734 3.0 arc 00:05 52735 3.0 arc 00:20 52736 3.0 arc 00:23 52737 3.0 arc 00:28 52738 10.0 flat 00:31 52739 2.0 arc 00:35 52740 3.0 arc 00:37 52741 3.0 arc ----- sequence 52742, plate -9999 ------- 01:09 52742 0.0 bias ----- sequence 52743, plate -9999 ------- 01:42 52743 0.0 bias ----- sequence 52744, plate 2922 ------- 02:53 52744 10.0 flat 02:56 52745 2.0 arc 03:38 52746 1500.0 target 04:06 52747 1500.0 target 04:19 52748 10.0 flat 04:21 52749 2.0 arc 04:56 52750 1801.0 target 04:59 52751 10.0 flat 05:02 52752 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 03:11Z 1 2922 30.37 0.0010 50.47 0.0030 205.15 0.0189 1.000170 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 03:11Z 1 2922 1.00017 -1679 -285 -75 30 50.5 11.6 11 190 -- 1.3" ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:22Z 62 17 16 214 (SW) 3978 - 22:55Z 61 16 16 248 (WSW) 4067 - clouds 23:26Z 61 16 22 224 (SW) 3294 - '' 23:58Z 59 15 12 204 (SSW) 3265 - '' 00:28Z 56 14 9 187 (S) 3311 - '' 00:58Z 56 13 18 217 (SW) 3461 - '' 01:51Z 52 12 11 189 (S) 3601 - '' 02:21Z 52 14 11 180 (S) 4551 - '' 02:51Z 52 14 11 181 (S) 5232 - clear 03:22Z 53 14 12 198 (SSW) 5640 - clouds are coming 03:52Z 52 14 12 181 (S) 6260 - clouds 04:22Z 53 15 17 200 (SSW) 6960 - '' 04:52Z 53 16 18 196 (SSW) 7063 - '' 05:22Z 53 17 17 181 (S) 6843 - '' 05:55Z 53 17 19 186 (S) 6159 - '' 06:27Z 53 17 21 197 (SSW) 6272 - '' 07:00Z 52 20 16 188 (S) 5986 - '' 07:32Z 52 21 14 213 (SSW) 5169 - clear 08:04Z 51 20 23 213 (SSW) 5298 - '' 08:36Z 51 18 16 223 (SW) 4872 - clouds 09:08Z 50 17 18 218 (SW) 4671 - '' 09:40Z 50 13 16 238 (WSW) 4980 - '' 10:13Z 50 10 13 242 (WSW) 4918 - '' 10:45Z 50 8 12 256 (WSW) 4920 - '' ---=== TELESCOPE STATUS ===--- 01:05Z Opened louvers, cracked doors, turned on fans. 02:10Z Enclosure off. 05:15Z Enclosure on, doors closed, fans and blowers off, louvers closed. No interlocks bypass Status at: 11:00Z Telescope stowed at 121, 30, 00 mount Instrument mounted: cartridge 1 Spectro Dewar: 92 lb, 22.1 psi Imager Dewar : 86 lb, 16.4 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_38_0 Watcher: v2_36_0 MCP: v6_4_0 TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_40 SoS: v5_1_8 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0200 0.0200 Axial B -0.0170 0.0000 Axial C -0.0240 -0.0230 Trans D 0.0010 0.0120 Lateral E 10.0838 10.0584 Lateral F 11.6459 11.6205 GALILS Commanded: 4600. -2000. 2650. -12450. -6500. -6550. Actual: 4600. -2000. 2650. -12439. -6477. -6546. SETMIR VALUES PriDesOrient: 0.00 0.00 0.00 100.00 -200.00 PriOrient: 0.00 0.00 0.00 99.59 -200.69 SECONDARY: ---------- Focus: 0.00 Air Temp.: 13.4 Alt.: 29.999993 MIGS TONIGHT NOMINAL Axial A -0.0420 -0.0440 Axial B -0.0490 -0.0570 Axial C -0.0550 -0.0670 Trans D -0.0200 -0.0110 GALILS Commanded: 1629180. 1651139. 1469426. -5900. -7800. Actual: 1627496. 1653168. 1470885. -5887. -7780. SETMIR VALUES SecDesOrient: 1257.00 60.00 -4.00 -20.00 118.90 SecOrient: 1256.52 60.22 -5.20 -20.70 119.14 ---=== PROBLEMS IN DETAIL ===--- Spectro Focus Info: ------------------- Focus checked by used cartridge 1. It currently set for sp1> caget tpm_TM_PMSSAvg tpm_TM_TelairAvg.VALA tpm_TM_PMSSAvg 10.93 tpm_TM_TelairAvg.VALA 11.62 sp1> mechSend s Coll_motor_A 333 01P0000001834 Coll_motor_B 506 02P0000002007 Coll_motor_C 314 01P0000001813 Coll_motor_A_status 133 01S0133 Coll_motor_B_status 133 02S0133 Coll_motor_C_status 132 01S0132 Requested_exp.time 0 Exp_time_left 0 Last_exp.time 3 Last_open_transit.time 700 Last_close_transit.time 525 Humid_Hartmann 11.2 0xFE27 Humid_Cen_Optics 9.8 0xFE3A Temp_Median 12.9 Sp2> mechSend s Coll_motor_A 2829 01P0000002829 Coll_motor_B 2286 02P0000002286 Coll_motor_C 2633 01P0000002633 Coll_motor_A_status 132 01S0132 Coll_motor_B_status 133 02S0133 Coll_motor_C_status 133 01S0133 Requested_exp.time 0 Exp_time_left 0 Last_exp.time 3 Last_open_transit.time 705 Last_close_transit.time 630 Humid_Hartmann -25.0 0x0 Humid_Cen_Optics 9.9 0xFE38 Temp_Median 10.7 IDL 5.5a> collimate, 52740 Reading /data/spectro/54598/sdR-r1-00052740.fit Reading /data/spectro/54598/sdR-r1-00052741.fit COLLIMATE: Min offset = -0.89 pix COLLIMATE: Max offset = 0.32 pix COLLIMATE: RMS across CCD = 0.34 pix COLLIMATE: Mean offset = -0.33 pix COLLIMATE: Camera r1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict (red) piston movement of 3040 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 3040" Log file = Collimate-54598-r1-00052740.log PostScript file = Collimate-54598-r1-00052740.ps Reading /data/spectro/54598/sdR-r2-00052740.fit Reading /data/spectro/54598/sdR-r2-00052741.fit COLLIMATE: Min offset = -0.79 pix COLLIMATE: Max offset = 0.25 pix COLLIMATE: RMS across CCD = 0.30 pix COLLIMATE: Mean offset = -0.05 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 447 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston 447" Log file = Collimate-54598-r2-00052740.log PostScript file = Collimate-54598-r2-00052740.ps -------- Problem: -------- If somebody owns a teamster, only that person can restart the teamster if it got dead. Other observers has no more permission to do it. It used to be done by any observer. The problem was noticed last nigh with the mager teamster.