APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday October 4, 2005 (MJD 53648) ---=== OBSERVING TEAM ===--- Swing: John Barentine Night: Mike Harvanek Dan Long Support: Fritz Stauffer Jon Brinkmann Steph Snedden Rick Kessler (by phone) Visitors: Damien Christian (Queen's University, Belfast) ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- After a 2 hour humidity delay, we imaged 82 S between lambda of 115 and 164 (excluding ramps) under variable clouds, before having to close again for humidity around 7:00Z. We did not reopen. ---=== OBSERVING LOG ===--- Afternoon --------- We're plenty good on disk space for tonight. sdsshost2:/home has 17% (19GB) free, and sdssfiles1:/data1/data has 46% (640GB) free. Also, Fritz has installed another drive on 'host2 in the /local partition, offering an additional 349GB. Problems today: 1) Murmur server died hard, requiring a reboot of sdsshost2 2) Communication with id1 failed. Crates were rebooted. Also noticed that the azimuth windscreen pointer is a little off-center, although the rails look pretty much aligned. The engineers might want to have a look. Night ----- 00:58Z Run 5696 On telescope bias Watcher is complaining of no udp packets for all nodes. Rebooted crates and tried again. 01:20Z Run 5697 Another on telescope bias Still NaN in the headers. A call to Fritz and now all is well. The process on sdsshost2 which relays the udp packets had died due to an incorrect address being given as part of the restart of sdsshost2 following the reboot today. 02:30Z Doors and Louvers Open, Fans On 03:10Z Enclosure Open after humidity delay 03:27Z Run 5698 82 S From 113 to 166 (including ramps) Skippy and seeing good from the start. Focus converged at frame 17 Clouds roll in a few frames later :( Clouds thinning :) Waves of cloud come and go for much of the run Thicker clouds between frames 18, 114 --> 33, 116 57, 119.7 --> 63, 120.7 108, 127.4 --> 113, 128.1 163, 135.7 --> 176, 137.6 193, 140.2 --> 221, 144.3 Seeing for this run varied between 0.8 and nearly 3 arcseconds though there were fairly long stretches between frames 50 and 150 and again between 250 and the end where the seeing was under 2 arcseconds and the clouds weren't too bad. Humidity right near the limit for most of the run. Now going over the limit. 07:03Z endDrift Closing due to clouds, humidity, and uncomfortably close radar returns 07:15Z Enclosure Closed, Fans Off Humidity stays high 10:50Z We through in the towel 11:15Z endNight rejoiced spooling failed for reasons not terribly clear. It seemed to go wrong after getting the oplogs [sn_user@sn10 ~]$ spool_photo_wrapper.cmd 5698 53648 . . . Get oplogs from : sn_user@sdsshost2.apo.nmsu.edu:/home/skycat/oplogs/astrolog/53648 bla: Subscript out of range. Full output from spooling script in the problems section We called Rick and he is going to look at it. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 5694 22:17Z 22:23Z 100 O 57.07 58.66 25 ignore 5696 01:02Z 01:21Z 100 O 98.51 103.27 46 bias 5697 01:26Z 01:46Z 100 O 104.52 109.59 48 bias no entry for 5698 making a hand entry (note that the run info is in the idReport file.) 5698 ~3:27Z ~7:03Z 82 S ~113 ~166 ~377 science supernova ---=== IMAGING RUN DETAILS ===--- Run 5698: 82 S 115 to 164 (excluding ramps) Frame Lambda Notes ----- ------ -------------------------------------------------------- 10 113 Skippy and seeing good from the start (start of ramp) 17 114 Focus converged ( wasn't far off at the start anyway) 18 114 Thicker cloud 25 115 Start of science 33 116 Clouds thin 57 120 Cloud thicken 63 121 Clouds thin 108 127 Clouds thicken 113 128 Clouds thin 163 136 Clouds thicken 176 138 Clouds thin 193 140 Clouds thicken 221 144 Clouds thin 355 164 Start of ramp (ending drift because we need to close) 368 166 End of ramp and run. ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 5698 15 4 76 -0.300 0.140 -2.500 0.131 -0.00504 218 51 5698 76 4 129 -0.200 0.146 -2.900 0.132 -0.00482 218 51 5698 128 4 83 0.200 0.181 -3.100 0.159 -0.00481 218 51 5698 180 4 81 0.600 0.203 -3.200 0.163 -0.00523 218 51 5698 233 4 59 0.000 0.200 -3.600 0.133 -0.00501 218 51 5698 286 4 60 0.900 0.152 -3.400 0.112 -0.00513 218 51 5698 338 4 52 1.200 0.144 -3.400 0.137 -0.00541 218 51 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 5698 51 1 51 322 8 -1.078 0.144 0.213 0.138 -0.00072 5698 51 1 51 322 14 -0.749 0.202 0.259 0.212 -0.00087 5698 51 1 51 322 14 -0.412 0.108 0.633 0.205 -0.00213 5698 51 1 51 322 16 -0.091 0.137 0.749 0.140 -0.00252 5698 51 1 51 322 8 0.225 0.115 1.053 0.363 -0.00354 5698 243 1 51 322 25 -0.784 0.384 0.336 0.188 -0.00113 5698 243 1 51 322 25 -0.461 0.200 0.792 0.213 -0.00266 5698 243 1 51 322 24 -0.317 0.242 1.216 0.450 -0.00409 5698 243 1 51 322 19 0.460 0.219 0.789 0.270 -0.00265 5698 243 1 51 322 27 0.644 0.271 1.334 0.309 -0.00448 5698 262 1 51 322 17 -0.547 0.236 0.392 0.286 -0.00132 5698 262 1 51 322 12 -0.334 0.214 0.771 0.200 -0.00259 5698 262 1 51 322 14 0.222 0.241 0.813 0.229 -0.00273 5698 262 1 51 322 21 0.665 0.226 0.891 0.385 -0.00300 5698 262 1 51 322 15 0.711 0.350 1.117 0.432 -0.00376 5698 296 1 51 322 11 -0.646 0.262 0.261 0.248 -0.00088 5698 296 1 51 322 23 -0.451 0.219 0.537 0.311 -0.00181 5698 296 1 51 322 13 0.209 0.242 0.925 0.145 -0.00311 5698 296 1 51 322 25 0.035 0.247 1.339 0.417 -0.00450 5698 296 1 51 322 15 0.766 0.281 0.872 0.123 -0.00293 5698 314 1 51 322 21 -0.944 0.320 -0.108 0.260 0.00036 5698 314 1 51 322 17 -0.548 0.302 0.176 0.183 -0.00059 5698 314 1 51 322 17 -0.087 0.331 0.327 0.299 -0.00110 5698 314 1 51 322 13 0.175 0.384 0.707 0.119 -0.00238 5698 314 1 51 322 14 0.541 0.308 0.779 0.175 -0.00262 5698 347 1 51 322 17 -0.511 0.308 -0.034 0.276 0.00011 5698 347 1 51 322 11 -0.328 0.292 0.467 0.379 -0.00157 5698 347 1 51 322 13 0.068 0.319 0.944 0.232 -0.00317 5698 347 1 51 322 19 0.207 0.299 0.884 0.331 -0.00297 5698 347 1 51 322 6 0.785 0.241 0.979 0.390 -0.00329 5698 357 1 51 322 18 -1.115 0.150 -0.295 0.354 0.00099 5698 357 1 51 322 15 -1.019 0.182 0.272 0.397 -0.00092 5698 357 1 51 322 17 -0.321 0.229 0.248 0.286 -0.00084 5698 357 1 51 322 14 -0.091 0.199 0.543 0.351 -0.00182 5698 357 1 51 322 17 0.306 0.211 0.591 0.271 -0.00199 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n):y QA Procedures Done (y/n):n UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 34113, plate -9999 ------- 21:33 34113 0.0 bias 21:43 34114 0.0 bias 23:32 34115 0.0 bias ----- sequence 34116, plate 2258 ------- 23:37 34116 10.0 flat 23:39 34117 2.0 arc 23:46 34118 2.1 arc 23:48 34119 2.1 arc ----- sequence 34120, plate -9999 ------- 00:00 34120 0.0 bias 01:05 34121 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 04:00Z imager -37.60 0.0000 50.80 0.0000 58.90 0.0000 1.000000 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL Stays: JL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 04:00Z imager 1.00000 0 -347 -261 -38 50.8 13.7 10 186 23-i' 1.02 05:04Z imager 1.00000 0 -370 -284 -38 50.8 13.7 12 197 23-i' 1.42 05:57Z imager 1.00000 0 -350 -263 -38 50.8 13.0 10 178 23-i' 1.71 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 16:13Z 49 50 10 234 (SW) -9999 - Partly Cloudy 16:45Z 48 47 0 218 (SW) -9999 - Partly Cloudy 20:21Z 59 48 11 192 (SSW) -9999 - Partly Cloudy 20:53Z 61 52 6 188 (S) -9999 - Partly Cloudy 21:25Z 63 52 11 226 (SW) -9999 - Partly Cloudy 22:07Z 63 50 14 222 (SW) -9999 - Partly Cloudy 22:39Z 62 53 14 222 (SW) -9999 - Partly Cloudy 23:10Z 60 53 11 232 (SW) -9999 - Partly Cloudy 23:41Z 60 53 8 199 (SSW) -9999 - Partly Cloudy 00:20Z 58 52 15 192 (SSW) -9999 - Partly Cloudy 00:50Z 56 52 14 193 (SSW) -9999 - Mostly Cloudy 01:20Z 55 50 13 200 (SSW) -9999 - Partly Cloudy 01:51Z 55 49 14 196 (SSW) -9999 - Partly Cloudy 02:23Z 55 49 15 211 (SSW) -9999 - Mostly Clear 02:55Z 55 49 12 199 (SSW) -9999 - Mostly Clear 03:26Z 56 49 12 204 (SSW) -9999 - Mostly Clear 03:56Z 56 49 11 187 (S) -9999 - Partly Cloudy 04:29Z 56 50 10 202 (SSW) -9999 - Partly Cloudy 05:01Z 56 50 12 185 (S) -9999 - Partly Cloudy 05:33Z 55 50 12 180 (S) -9999 - Partly Cloudy 06:05Z 55 49 11 179 (S) -9999 - Partly Cloudy 06:37Z 54 48 9 173 (S) -9999 - Mostly Clear 07:09Z 53 49 7 179 (S) -9999 - Mostly Clear 07:42Z 52 50 0 207 (SSW) -9999 - Mostly Clear 08:14Z 55 49 9 212 (SSW) -9999 - Clear 08:46Z 55 49 7 210 (SSW) -9999 - Clear 09:18Z 55 49 9 214 (SW) -9999 - Clear 09:50Z 54 49 10 184 (S) -9999 - Clear 10:26Z 53 48 12 205 (SSW) -9999 - Mostly Clear 10:59Z 51 48 8 224 (SW) -9999 - Mostly Clear 11:31Z 50 48 2 181 (S) -9999 - Partly Cloudy ---=== TELESCOPE STATUS ===--- Stowed at 30 deg alt, cartridge 4 installed. Spectro auto-fill connected. Spectro dewar 11:00Z Dewar wt 128 lb Imager dewar 11:00Z Dewar wt 164 lb No interlocks bypassed. ---=== SOFTWARE USED ===--- IOP/SOP: v4_5_0 Watcher: v2_30_2 MCP: v5_26_0 TPM: tpm_v3_1_0 AstroDa: v15_4 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.8420 -5.8440 Axial B -5.6740 -5.6770 Axial C 0.9420 0.9420 Trans D -9.9920 -9.9940 Lateral E 10.3759 10.3759 Lateral F 11.9380 11.9380 GALILS Commanded: 4700. -2450. 3000. -7800. 3300. 3000. Actual: 4700. -2450. 3000. -7800. 3300. 3000. SETMIR VALUES PriDesOrient: 0.00 -1.68 7.05 385.25 92.02 PriOrient: 0.00 -1.52 7.02 384.99 93.81 SECONDARY: ---------- Focus: 0.00 Air Temp.: 16.6 Alt.: 29.999997 MIGS TONIGHT NOMINAL Axial A 1.5540 1.5420 Axial B 1.0590 1.0470 Axial C 1.1850 1.1650 Trans D -0.0990 -0.1240 GALILS Commanded: 1593450. 1603636. 1554295. -6800. -5900. Actual: 1593450. 1603650. 1554300. -6800. -5900. SETMIR VALUES SecDesOrient: 1257.00 15.87 0.00 -50.00 136.76 SecOrient: 1256.99 15.88 0.00 -49.56 136.73 ---=== PROBLEMS IN DETAIL ===--- Murmur Server Out To Lunch -------------------------- Before checkout this afternoon, Jon and I found that the murmur server had stopped functioning, and attempts at stopping and restarting the process didn't work. A "defunct" murmur_log process owned by jcb seemed to be the problem and I tried killing with a "kill -9" command. That didn't work. Fritz suggested I log out of sdsshost2 entirely, which did make the defunct process go away. However, we could not get a new murmur server to run with the dscRun stop_mursrvr/start_mursrvr commands. In the end, we had to reboot sdsshost2 and start servers again. That worked. Error On goDrift ---------------- On attempting the first goDrift -bias of the afternoon, I got this error: Waiting for command response from id1 (crate 1, ICC 0), control server (client 0 ) Waiting for command response from id1 (crate 1, ICC 0), control server (client 0 ) Waiting for command response from id1 (crate 1, ICC 0), control server (client 0 ) Waiting for command response from id1 (crate 1, ICC 0), control server (client 0 ) 2005-10-04 21:43:31Z: beginData has failed: {id1 {couldn't read file "/p/astroba se/fitsHeaders.tcl"}} {id2 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {id3 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {id4 {couldn't read fi le "/p/astrobase/fitsHeaders.tcl"}} {id5 {couldn't read file "/p/astrobase/fitsH eaders.tcl"}} {id6 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} {ad1 {co uldn't read file "/p/astrobase/fitsHeaders.tcl"}} {ad2 {couldn't read file "/p/a strobase/fitsHeaders.tcl"}} {ad3 {couldn't read file "/p/astrobase/fitsHeaders.t cl"}} {ad4 {couldn't read file "/p/astrobase/fitsHeaders.tcl"}} Error: goDrift ends with error in beginData Jon and I rebooted both photo crates and the astro crate to see if this would solve the problem. Eventually the DA came back and we took a test bias (Run 5694) to make sure the goDrift would work again. Jon thinks the references to "/p/astrobase/" etc. above refer to old 'host, erroneously of course. Spectro Focus.... Or Not ------------------------ Same problem again today with the inability to grabInstruments, with the same complaint as before: ************************************************** ====>>>>>>RELAY SERVER FAILED TO START!!!!<<<===== ************************************************** requestListen spectro: Neither 30310 nor 30307 appears to exist ************************************************** requestListen spectro: Neither 30310 nor 30307 appears to exist requestListen spectro: Neither 30310 nor 30307 appears to exist while executing "error "requestListen $type: Neither $port1 nor $port2 appears to exist"" etc. But I found this only when logged in as myself (jcb). If I log in as observer, I can grabInstruments without any complaints. Motor numbers today: sp1 Coll_motor_A 1259 Coll_motor_B 1258 Coll_motor_C 1756 sp2 Coll_motor_A 8514 Coll_motor_B 5447 Coll_motor_C 6008 Air temps at time of focus: tpm_TM_PMSSAvg 14.3436 tpm_TM_TelairAvg.VALA 4.4825 Forecast for tonight calls for lows about ~7 C, so I will focus for ~10. goSpecFocus gives: sp1 red l-r = 0.390 pixels , tolerance = 0.15 pixels, not focused yet Predict a movement of -3569 steps for sp1 red Issue: sp1; mechPiston -3569 Calculating pixel shift maps now... Done calculating pixel shift maps... sp2 red l-r = 0.432 pixels , tolerance = 0.15 pixels, not focused yet Predict a movement of -3953 steps for sp2 red Issue: sp2; mechPiston -3953 Calculating pixel shift maps now... Done calculating pixel shift maps... sp1 blue l-r = -0.289 pixels , tolerance = 0.15 pixels, not focused yet Predict a movement of 10 steps for sp1 blue If red is in focus then you need to rotate blue camera ring by 10 degrees Calculating pixel shift maps now... Done calculating pixel shift maps... sp2 blue l-r = -0.426 pixels , tolerance = 0.15 pixels, not focused yet Predict a movement of 15 steps for sp2 blue If red is in focus then you need to rotate blue camera ring by 15 degrees collimate gives: Reading /data/spectro/53648/sdR-b1-00034118.fit Reading /data/spectro/53648/sdR-b1-00034119.fit COLLIMATE: Min offset = -0.98 pix COLLIMATE: Max offset = 0.27 pix COLLIMATE: RMS across CCD = 0.33 pix COLLIMATE: Mean offset = -0.29 pix COLLIMATE: Camera b1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 10.3 degrees for spectro-1 COLLIMATE: (if red is already in focus) Reading /data/spectro/53648/sdR-b2-00034118.fit Reading /data/spectro/53648/sdR-b2-00034119.fit COLLIMATE: Min offset = -0.90 pix COLLIMATE: Max offset = 0.08 pix COLLIMATE: RMS across CCD = 0.30 pix COLLIMATE: Mean offset = -0.37 pix COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 13.3 degrees for spectro-2 COLLIMATE: (if red is already in focus) Reading /data/spectro/53648/sdR-r1-00034118.fit Reading /data/spectro/53648/sdR-r1-00034119.fit COLLIMATE: Min offset = -0.21 pix COLLIMATE: Max offset = 1.08 pix COLLIMATE: RMS across CCD = 0.38 pix COLLIMATE: Mean offset = 0.35 pix COLLIMATE: Camera r1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict (red) piston movement of -3162 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston -3162" Reading /data/spectro/53648/sdR-r2-00034118.fit Reading /data/spectro/53648/sdR-r2-00034119.fit COLLIMATE: Min offset = -0.23 pix COLLIMATE: Max offset = 0.66 pix COLLIMATE: RMS across CCD = 0.25 pix COLLIMATE: Mean offset = 0.35 pix COLLIMATE: Camera r2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict (red) piston movement of -3187 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston -3187" Predicted moves are within the expected difference between current air temperature and expected nighttime temperature, so no adjustment is made. Spooling Error: --------------- [sn_user@sn10 ~]$ spool_photo_wrapper.cmd 5698 53648 ###################################### spool_photo_wrapper.cmd PREVIEW ###################################### ------ SPOOL ------ RUN : 5698 MJD : 53648 RAW_RERUN: 5000 ------ PHOTO ------ RUN : 5698 RERUN : 5001 FIELDs : 0 to 1100 ------------------------------------------------ COLUMNS : 1 2 3 4 5 6 MACHINES : sn01 sn02 sn03 sn04 sn05 sn06 ------------------------------------------------ Preview of commands: > spool_master.cmd 5698 53648 NOPROMPT > photo_master.cmd 5698 5001 NOPROMPT SUBMIT COMMANDS ? y/[n] => y ---------------------------------------------------- spool_master.cmd : spool raw data for RUN : 5698 RERUN : 5000 MJD : 53648 COLUMNS : 1 2 3 4 5 6 PROMPT : no GETLOGS : yes GETDATA : yes Problems => call Rick ASAP Get oplogs from : sn_user@sdsshost2.apo.nmsu.edu:/home/skycat/oplogs/astrolog/53648 bla: Subscript out of range. ... will launch photo in 20 seconds ... [1] 1580 PHOTO has been launched; will run when spool is done Please check data copy (spool) with : > spool_mon.cmd If there are problems => call Rick ASAP [sn_user@sn10 ~]$ spool_mon.cmd cat: /data/sn10.a/data/photo/5698/5000/status/SPOOL1.BEGIN: No such file or dire ctory bla: Subscript out of range. [sn_user@sn10 ~]$