APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday September 18, 2009 (MJD 55093) ---=== OBSERVING TEAM ===--- Elena Malanushenko Steph Snedden Paul Harding (Case) Craig Loomis (PU - by STUI messennger) ---=== OBSERVING PLAN ===--- Imaging if conditions permit, otherwise spectro and/or guider commissioning tests ---=== OBSERVING SUMMARY ===--- Clouds, humidity and rain. Finished the out-of-focus flat and arc sequences for Kyle. More boss errors and a collimator actuator problem are noted in the problems section. They took a fair bit of our time tonight. ---=== OBSERVING LOG ===--- Afternoon ----------- restarted server cleared watcher Disk space - ok [elenam@sdsshost2 elenam]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 142G 123G 12G 92% / /dev/sda1 99M 30M 64M 33% /boot /dev/sdb1 917G 800G 71G 92% /data hub25mp:/home 92G 51G 42G 56% /home_hub25m sdssfiles1p:/data1/data 1.8T 50G 1.8T 3% /files iccSoundOff - ok imager bias, run 8044, frames 48, flavour - ignore watcher archiver does not show realtime information about frames Jon said that files1 disk is configured and it is ok to use All data have beed transfered to files1 computer 45 degree alt fiducial read going down, but not up. Night: ------ More clouds, high humidity, rain. 03:40Z - Continued taking flats and arcs at various out of focus values (Kyle's tests). We had more bossActor and ICC related errors and a collimator actuator problem crop up during these tests. They are all noted in the "problems" section. The results: 02:04:01 163 platedb loadCartridge cartridge=12 02:04:02 APO.elenam 163 platedb f text="Plate 3536 pointing None is not present in the plateDB" 03:30:42 170 guider loadCartridge cartridge=12 03:31:01 171 platedb showActive cartridge=12 03:33:50 172 sop doCalibs nbias=1, ff=101656 03:37:38 173 sop doCalibs nflat=1 narc=1 arcTime=2 ff=57,58 ========== collimation ========== 03:51:16 176 sop hartmann ff=59,60,61 idl>collimate, sp1> 257, sp2>1286 04:02:44 APO.elenam 177 boss i motorPosition=-7870,-8262,-8272,18448,17651,18201 boss moveColl spec=sp1 a=257 b=257 c=257 boss moveColl spec=sp2 a=1286 b=1286 c=1286 4:07:28 .boss 2886 boss i motorPosition=-7614,-8005,-8016,19734,18936,19487 Collect flats and arcs with different positions of collimation notors: ========= 1) collimate 0 (best focus position) 4:07:28 .boss 2886 boss i motorPosition=-7614,-8005,-8016,19734,18936,19487 04:11:43 183 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 fflat=101662,63,64,65; fflat=101666,67,68,69 -------------- 2) coll=500 away from best focus coll=0 4:32:53 APO.elenam 184 boss i motorPosition=-7614,-8005,-8016,19734,18936,19487 boss moveColl a=500 b=500 c=500 coll=500 04:34:19 APO.elenam 187 boss i motorPosition=-7114,-7458,-7514,20234,19436,19969 04:36:05 188 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101670,71,72,73; farc=101674,75,76,77 ----------- 3) coll=1000 away from best focus col=500 04:58:26 .boss 3018 boss i motorPosition=-7114,-7458,-7514,20234,19436,19969 boss moveColl a=500 b=500 c=500 coll=1000 boss i motorPosition=-6616,-6959,-7006,20734,19936,20469 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101678,79,80,81; farc=101682,83,84,85 -------- 4) coll=2000 away from best focus coll=1000 motorPosition=-6616,-6959,-7006,20734,19936,20469 boss moveColl a=1000 b=1000 c=1000 coll=2000 motorPosition=-5622,-5959,-6011,21735,20979,21469 05:25:36 197 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101686,87,88,89; farc=101690,91,92,93 ------------- 5) coll=5000 away from best focus coll=2000 motorPosition=-5622,-5959,-6011,21735,20979,21469 boss moveColl a=3000 b=3000 c=3000 coll=5000 05:46:48 APO.elenam 199 boss i motorPosition=-2624,-3005,-3003,24735,23938,24488 05:48:15 201 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101694,95,96,97; farc=101698,99,101700,01 ------------- 6) coll=10000 away from best focus coll=5000 05:46:48 APO.elenam 199 boss i motorPosition=-2624,-3005,-3003,24735,23938,24488 boss moveColl a=5000 b=5000 c=5000 coll=10000 06:11:26 .boss 3222 boss i motorPosition=2372,1995,1996,29735,28936,29469 06:12:56 205 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101702,03,04,05; ffarc=101706,07,08,09 --------------- 7) coll=0 (best focus) coll=10000 06:11:26 .boss 3222 boss i motorPosition=2372,1995,1996,29735,28936,29469 boss moveColl spec=sp1 a=-5000 b=-5000 c=-5000 boss moveColl spec=sp1 a=-5000 b=-5000 c=-5000 boss moveColl spec=sp2 a=-5000 b=-5000 c=-5000 boss moveColl spec=sp2 a=-5000 b=-5000 c=-5000 boss moveColl spec=sp1 a=-2000 b=-3000 c=-3000 coll=0 06:39:11 APO.elenam 213 boss i motorPosition=-7600,-8005,-8010,19734,18933,19469 -14 0 -6 0 3 18 - the difference between initial focus position and after multiple shifts and return 06:42:25 214 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101710,11,12,13; farc=14,15,16,17 --------------- 8) coll=-500 away from best focus coll=0 07:03:53 APO.elenam 215 boss i motorPosition=-7600,-8005,-8010,19734,18933,19469 boss moveColl a=-500 b=-500 c=-500 coll=-500 07:05:39 APO.elenam 218 boss i motorPosition=-8099,-8506,-8510,19234,18437,18969 07:06:46 219 sop doCalibs nflat=4 narc=4 flatTime=20 arcTime=2 ff=101718,19,20,21; farc=101722,23,24,25 ---------- 9) coll=0 (best focus) coll=-500 07:05:39 APO.elenam 218 boss i motorPosition=-8099,-8506,-8510,19234,18437,18969 boss moveColl a=500 b=500 c=500 coll=0 07:27:32 APO.elenam 221 boss i motorPosition=-7610,-8005,-8017,19734,18936,19469 ============================= Series of flats with different exposure times. 07:30:23 222 sop doCalibs nflat=4 flatTime=1; ff=101726,27,28,29 07:40:12 223 sop doCalibs nflat=4 flatTime=2; ff=101730,31,32,33 07:52:59 224 sop doCalibs nflat=4 flatTime=4; ff=101734,35,36, (37) boss displayed error and interrupted the sequence, no 101737 has been created. This was due to the sp1 reboot (see details in the pronblems section). 09:43:03 252 boss exposure flat itime=4; ff= 101741 (after debugging) ----------- 08:10:07 229 sop doCalibs nflat=1 flatTime=4, ff=101737 error again; did boss clearExposure boss iack sop restart (Craig was online by this time. See problems section). ============= 09:46:59 253 sop doCalibs nflat=1 flatTime=8;; ff=101742 10:01:12 262 sop doCalibs nflat=4 flatTime=8, ff=101743,44,45,46 10:11:34 267 sop doCalibs nflat=4 flatTime=16, ff=101747,48,49,50 10:22:22 271 sop doCalibs nflat=4 flatTime=32; ff=101751,52,53,54 -------------------- We spent the remainder of the night working on documentation, developing a new logTool and studying the flats and arcs taken for Kyle (results still pending). Sent out an updated version of the command-line interim document. endNight finished. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 8044 23:45Z 00:10Z 100 O 63.52 69.72 55 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:37Z 47 47 7 266 (W) 1725 - mostly cloudy 23:09Z 49 49 4 189 (S) 2449 - ' 23:41Z 48 47 7 201 (SSW) 1644 - ' 00:15Z 46 45 6 127 (SE) 959 - ' 00:43Z 45 44 3 100 (E) 1044 - ' 01:15Z 46 44 8 159 (SSE) 1199 - ' 01:47Z 45 43 10 130 (SE) 1078 - ' 02:19Z 46 43 13 141 (SE) 1457 - ' 02:52Z 46 43 11 126 (SE) 1636 - ' 03:24Z 45 43 15 125 (SE) 1408 - ' 03:56Z 45 43 13 128 (SE) 1330 - ' 04:28Z 44 42 6 133 (SE) 1059 - ' 05:00Z 45 43 5 155 (SSE) 1331 - ' 05:32Z 46 43 12 169 (S) 1761 - ' 06:04Z 46 43 11 197 (SSW) 1565 - ' 06:37Z 46 43 7 185 (S) 1700 - ' 07:09Z 45 43 6 181 (S) 1688 - ' 07:41Z 45 43 4 200 (SSW) 1650 - ' 08:13Z 45 43 9 172 (S) 1833 - ' 08:45Z 45 43 9 184 (S) 1521 - ' 09:17Z 46 42 7 181 (S) 1967 - ' 09:49Z 45 43 8 184 (S) 1999 - ' 10:22Z 45 43 7 185 (S) 1785 - ' 10:54Z 45 42 2 199 (SSW) 1399 - ' 11:26Z 45 41 7 190 (S) 919 - ' ---=== TELESCOPE STATUS ===--- Never ventilated. Never opened. Telescope stowed at 30 deg alt with cartridge 12 mounted. Counterweights at 235 Cartridge 12 mounted No interlocks bypassed Spectro autofill enabled 11:26Z - Spectro dewar: 198 lb; 18 psi 11:26Z - Imager dewar: 120 lb; 18 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_40_0 Watcher: v2_36_0 MCP: NOSVN TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.11.0 2009-06-24 sdssProcedures: v2_44 SoS: v5_1_8 plate-mapper: ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -0.6940 -0.6940 Axial B -0.0300 -0.0410 Axial C -0.3710 -0.3710 Trans D 11.6140 11.6170 Lateral E 10.3632 10.3505 Lateral F 11.9253 11.9253 GALILS Commanded: 9900. 400. 5850. 1200. 2950. 2750. Actual: 9917. 401. 5832. 1205. 2954. 2757. SETMIR VALUES PriDesOrient: 0.00 -3.30 5.10 278.08 178.06 PriOrient: 0.00 -3.04 5.26 277.70 178.07 SECONDARY: ---------- Focus: 0.00 Air Temp.: 7.4 Alt.: 29.999993 MIGS TONIGHT NOMINAL Axial A 18.6070 18.6150 Axial B 15.6130 15.6120 Axial C 13.8240 13.8170 Trans D -11.7730 -11.7780 GALILS Commanded: 1597665. 1566232. 1588071. -7600. -12400. Actual: 1597721. 1566188. 1588095. -7597. -12380. SETMIR VALUES SecDesOrient: 1257.00 -4.00 -9.00 10.00 234.42 SecOrient: 1256.99 -3.99 -8.97 10.21 234.68 ---=== PROBLEMS IN DETAIL ===--- Received the following error in the STUI log; seems to be associated with an auto-mechStatus. This occurred between exposures, when nothing was in process. 03:44:26 .boss 2804 boss f text="command failed: ValueError("invalid literal for int() with base 10: 'NaN'",) at /home/sdss3/products/Linux64/boss/v1_2_5/python/boss/Commands/MechCmd.py:76" ---------- 04:23Z - During a sequence of four flats and four arcs, we got the "critical alert" sound cue, but no message in the "Alerts" GUI or in the log. Perhaps this is a camcheck communication problem during these ongoing exposures? The problem repeated itself periodically during the tests we did for Kyle, and we always had repeated exposures running. ---------- While changing focus, note that on sp1, two sequential moves of 5000 ticks each did not put the actuators at the right positions. Actuator a was 2000 ticks short, b was 3000 ticks short and c was 3000 ticks short. A third command to move the remainder was needed. That worked. We were not near a limit. Is this related to the "1000 tick" bug noted in a previous log? We did NOT use the piston command. ---------- sp1 spontaneous reboot mayhem --- The following happened between two exposures: 07:59:45 APO.elenam.sop 550 boss f text="command failed: CommError(, "TDS(TCPPort(name=None:2100 to t-g-sdss-2.apo.nmsu.edu:2100) ok=True initialized=False) send ''E'', received ''D''") at /home/sdss3/products/Linux64/boss/v1_2_5/python/boss/Controllers/Forth.py:320" The first boss ping gave CmdText="ping" 08:05:48 APO.elenam 227 boss i text="sp2daq : OK" 08:05:49 APO.elenam 227 boss i text="sp1cam : DEAD" 08:05:50 APO.elenam 227 boss i text="sp1mech : v2_9_0" 08:05:51 APO.elenam 227 boss i text="sp2mech : v2_9_0" 08:05:52 APO.elenam 227 boss i text="sp2cam : LIVE" 08:05:52 APO.elenam 227 boss i text="sp1daq : OK" But a subsequent ping showed sp1cam to be "LIVE" We did a boss iack. We still couldn't read out sp1 exposures, so did a clearExposure, then we got a serious alert: Serious boss.SP1ExecBoot Reported by camCheck Serious boss.SP1PhaseBoot Reported by camCheck We did an iack again and tried an exposure again. No good. Restarted sp1 and restarted the boss Actor. No good. Craig came online and found the boss ICC "splat" when the sp1 reboot happened. He says we should only have to do an iack when a reboot happens, but for now, sop may become hosed, in which case a sop status will tell you if you need to do a sop restart. The command line document was updated with this info.