APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday September 15, 2009 (MJD 55090) ---=== OBSERVING TEAM ===--- Audrey Simmons Dan Oravetz Paul Harding (guider) Kyle Dawson (BOSS) Matt Olmstead (BOSS) Vaishali Bhardwaz (BOSS) Jon Brinkmann (support) Steph Snedden (support) Robert Lupton (guider support) ---=== OBSERVING PLAN ===--- 1) Imaging, if weather permits. 2) Testing the guider on sky. 3) Obtaining BOSS exposures on sky. 4) BOSS cloudy night tasks. ---=== OBSERVING SUMMARY ===--- We imaged 83 S from lambda = 144 to 173, including ramps, under photometric conditions and excellent seeing. We imaged the flat-field scan (0 O) at RA = 30 for 1 hour with excellent seeing. The flavor of this scan was "calibration" instead of "science" since IOP was not happy with the stripe and stip we gave it; the error details are in the Log and the Problems section. Also, during the last seven minutes of this scan some light clouds moved in at the southern edge of the cloud camera, but were never more than ~ 30 degrees below where we were imaging according to the cloud camera. Paul and Robert did some guider testing in RA/DEC instead of ALT/AZ, but we have nothing conclusive to report. We had lots of imager.heartbeat errors, servers misbehaving, gang transfer issues, daqBusy problems, and disk space issues; details in the Log and Problems section. Most of the misbehavior seemed to be a consequence of sdssfiles1 being down and sdsshost2 having to accommodate more data than it was prepared to. A special thanks goes to Jon for managing disk space until twilight so that we could image. endNight unsurprisingly had problems tonight. The errors are in the Problems section. ---=== OBSERVING LOG ===--- Afternoon ========= 23:30Z - Restarted the servers on MONX3. 23:41Z - Started an imager bias. 23:50Z - Verified the STUI actors were alive and responsive. 23:50Z - Tried to check the disk space with `df -h' on host2, but it hung since sdssfiles1 is down. 00:06Z - We took some hartmann exposures with `sop hartmann' after moving the collimator motors by +5000. Original positions: sp1: 6167, 5797, 5791 sp2: 6860, 7043, 6615 Current positions (following +5000 move in both sp1 & sp2): sp1: 11168, 10797, 10790 sp2: 11858, 12040, 11595 00:15Z - Checked out the telescope and the lower enclosure. Checked fiducial crossings and mirror numbers. The 45-deg. altitude fiducials were not read again. We mounted cartridge 12 in preparation for opening at sunset. 00:30Z - Began ventilation. Since sdssfiles1 is down, Jon and Steph modified a handful of configuration files, created /data/photo/frames, /data/photo/fields, and /data/photo/md5sum on sdsshost2, and adjusted some other things so that we could write imaging data somwhere. Night ===== 01:30Z - Opened to partly cloudy skies. Topped off the spectro dewars just in case we could open later for imaging (didn't want to have to stop and fill spectro dewars in the middle of imaging). Cartridge 12 ------------ platedb i activePlugging=12,3536,A,55090,1,318.0000,0.0000,-15,5.0 02:00Z - gotoField. We had trouble finding guide stars during twilight which we attributed to the guider CCD's high response in the blue causing the twilight sky to dominate. Further into twilight we did find guide stars. Paul said he is aware that the blue cutoff for the guider still needs to be put in place and will talk to the right people. We began testing the guiding in RA/DEC instead of ALT/AZ. The sky cleared and the cloud cam was showing photometric conditions, so we decided to mount the imager. We tried to image the flat-field scan to begin with, but IOP complained "Error: can't read "stripe": no such variable" when we entered 0,O for the stripe and strip. The trace-back for this error is in the Problems section. We also tried different stripe and strip values (e.g. 82,O) and none of our attempts succeeded. We decided to start imaging the next highest priority stripe, 83 S, while we were trying to come up with a work-around for this cross scan. 83 S, Run 8038 ************** 06:34Z - Started `goDrift' at lambda = 142.5. We started at an altitude of 50 degrees. The seeing, according to the DIMM, was ~0.7". 06:44Z - Focus good from frame 16. Started ramp-up at lambda = 144. 06:52Z - Started science at frame 26, lambda = 146. 07:02Z - Steph provided us with some work-arounds that we will try if the weather still permits after 83 S is completed. 08:32Z - End science, start ramp-down at frame 196, lambda = 171. 08:40Z - End ramp at frame 209, lambda = 173. Since `skippy' isn't working, we kept an eye on the fiducial corrections and all were well during the run. The median seeing as reported by seeingRun was well below 1.5" and the cloud camera reported photometric skies throughout the entire run. After the run finished up we had problems with the servers and with gang transfers, as well as with a nearly full disk unable to accept anymore transfers from the DA. Jon was aware of this and was rsync-ing data off of sdsshost2 as of 07:00Z and subsequently deleting the transferred data off of sdsshost2 throughout the night in order to create space for the photo data. As space opened up on host2, the DA was writing while things were still transferring off of host2 and the high disk activity really bogged things down. Once things settled down with the host2 disk writing/transferring/deleting, then things became more responsive again and we were able to move on to more imaging. Flat-field scan, Run 8039 ************************* We didn't have any problems with IOP complaining about not reading the stripe because we called the flavor "calibration" instead of "science"; 0,O worked fine this time as the stripe and strip. LMST = 30 degrees. We issued: TRACK 30,-5.0 FK5 /SCANVELOCITY=(0.0029543432,0.0029543432) 09:25Z - Started `goDrift' at RA = 30. Focus good from the start, frame 8. 10:17Z - We noticed some clouds at the south edge of the cloud camera ~30 degrees lower altitude than where we were pointed. We decided to go the extra eight minutes and make it an even hour on this scan. The clouds never came any closer. (04:17Z corresponds to approximately frame 98). We kept an eye on the fiducial crossings throughout the run and all of them looked fine. The median seeing was again < 1.5" throughout the entire run. After we ended the drift and verified stripe, strip, etc., there were complaints about setting a whole bunch of parameters to "NaN". These errors are reproduced in the Problems section. We tried to take the imager bias and ran into problems because of a lack of disk space (error in Problems section). We had a bunch of imager.heartbeat errors in STUI that were a result of failed ftelnet attempts by the imager server since Audrey still had control after drifting. Once Audrey did a `grabInst -drop im', then the critical alerts stopped. Imager Bias, Run 8042 ********************* We were unable to take a bias because of the daqBusy complaints that looked like a result of a failed bias attempt, but we freed things up with: im> iccExec [vmeList] readoutAbort This freed things up and we were able to proceed with a bias. Amidst all of the disk space issues we checked that the pool did not contain any unarchived frames, and it returned how we had hoped: im> iccExec [photoList] archListQ {id1 {}} {id2 {}} {id3 {}} {id4 {}} {id5 {}} {id6 {}} 11:17Z - `goDrift -bias'. This is an on-telescope tbars-unlatched bias. 11:50Z - Unmounted the imager, mounted the corrector, mounted cartridge 12 and then closed up. We ran endNight for 55089 since we were unable to last night and it succeeded. Then we proceeded to trying endNight for tonight and we ran into some more complaints, so we ran it manually and reproduced the errors in the Problems section. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 8032 23:42Z 00:13Z 100 O 59.81 67.48 65 ignore 8035 02:54Z 03:22Z 100 O 107.79 114.90 50 ignore 8035 02:54Z 03:22Z 100 O 107.79 114.90 50 ignore 8035 02:54Z 03:33Z 100 O 107.79 117.54 50 ignore 8035 02:54Z 03:22Z 100 O 107.79 114.90 50 ignore 8035 02:54Z 03:33Z 100 O 107.79 117.54 50 ignore 8035 02:54Z 05:27Z 100 O 107.79 146.35 50 ignore 8036 06:27Z 06:35Z 83 S 142.63 142.63 27 ignore Not sure why the Run numbers are so screwy here. 8038 - 83 S 8039 - Flat-field scan. Supposed to be 0 O. Called "calibration" for flavor. 8042 - On-telescope imager bias. ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- 8036 11 4 0 0.000 0.000 0.000 0.000 -0.00000 0 0 8038 8 4 0 0.000 0.000 0.000 0.000 -0.00000 0 0 8038 60 4 0 0.000 0.000 0.000 0.000 -0.00000 0 0 ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ 8038 35 1 51 330 18 -0.577 0.294 2.060 0.281 -0.00693 8038 35 1 51 330 35 -0.242 0.127 2.229 0.281 -0.00749 8038 35 1 51 330 30 0.144 0.187 2.153 0.226 -0.00724 8038 35 1 51 330 21 0.643 0.233 2.217 0.246 -0.00745 8038 35 1 51 330 29 0.743 0.131 2.128 0.201 -0.00715 ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ The focus started at around -70 during the first run. The `Focus' button in logTool did not grab the info. ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:57Z 63 33 12 262 (W) 391 - partly cloudy 23:29Z 66 39 9 266 (W) 457 - partly cloudy 00:02Z 62 37 3 251 (WSW) 500 - partly cloudy 00:34Z 61 36 6 278 (W) 485 - partly cloudy 01:07Z 59 37 2 268 (W) 517 - partly cloudy 02:08Z 59 35 2 252 (WSW) 490 - partly cloudy 02:41Z 59 36 1 224 (SW) 503 - partly cloudy 03:14Z 59 37 5 224 (SW) 553 - partly cloudy 03:46Z 59 37 8 234 (SW) 613 - clear 04:18Z 58 36 9 224 (SW) 634 - clear 04:50Z 58 36 2 223 (SW) 627 - clear 05:23Z 58 35 5 262 (W) 603 - clear 05:55Z 57 36 5 269 (W) 650 - clear 06:27Z 57 37 5 267 (W) 730 - clear 06:59Z 57 38 3 271 (W) 750 - clear The weather info stopped writing, but clear/mostly clear the rest of the night. ---=== TELESCOPE STATUS ===--- 00:30Z - Began ventilation. Doors and louvers opened, fans on. 01:30Z - Enclosure off. 11:50Z - Enclosure on, doors and louvers closed, fans off. No interlocks bypassed. Status at: 12:05Z Telescope stowed at: 30,121,00 mount Instrument mounted: Cartridge 12 Counterweights at: 250 lbs LN2 autofill systems: Connected and turned on. 180L LN2 dewar scales: Spectro 164.30 lbs, 24.18 psi Imager 119.60 lbs, 19.58 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_40_0 Watcher: v2_36_0 MCP: NOSVN TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.11.0 2009-06-24 sdssProcedures: v2_44 SoS: v5_1_8 plate-mapper: v4_3_1 STUI: Version 0.2.1 2009-09-02 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -0.6940 -0.6940 Axial B -0.0310 -0.0410 Axial C -0.3710 -0.3710 Trans D 11.6140 11.6170 Lateral E 10.3759 10.3505 Lateral F 11.9380 11.9253 GALILS Commanded: 9900. 400. 5850. 1200. 2950. 2750. Actual: 9917. 401. 5832. 1205. 2954. 2757. SETMIR VALUES PriDesOrient: 0.00 -3.30 5.10 278.08 178.06 PriOrient: 0.00 -3.04 5.26 277.70 178.07 SECONDARY: ---------- Focus: 0.00 Air Temp.: 17.0 Alt.: 30.000001 MIGS TONIGHT NOMINAL Axial A 18.6200 18.6150 Axial B 15.6260 15.6120 Axial C 13.8420 13.8170 Trans D -11.7880 -11.7780 GALILS Commanded: 1597772. 1566131. 1588077. -7600. -12400. Actual: 1597721. 1566188. 1588095. -7597. -12380. SETMIR VALUES SecDesOrient: 1257.00 -4.00 -9.00 10.00 234.42 SecOrient: 1256.99 -4.00 -9.03 10.17 234.69 ---=== PROBLEMS IN DETAIL ===--- Collimate results ----------------- IDL> collimate, 101485L Reading /data/spectro/55090/sdR-b1-00101485.fit SDSSPROC: Read noise = 1.68881 1.65849 1.65670 1.85020 electrons Reading /data/spectro/55090/sdR-b1-00101486.fit SDSSPROC: Read noise = 1.70354 1.66450 1.65530 1.85060 electrons COLLIMATE: Min offset = -1.20 pix COLLIMATE: Max offset = 1.80 pix COLLIMATE: RMS across CCD = 0.69 pix COLLIMATE: Mean offset = 0.38 pix COLLIMATE: Camera b1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of -13.5 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-55090-b1-00101485.log PostScript file = Collimate-55090-b1-00101485.ps Reading /data/spectro/55090/sdR-b2-00101485.fit SDSSPROC: Read noise = 1.78148 1.93740 1.97971 1.95663 electrons Reading /data/spectro/55090/sdR-b2-00101486.fit SDSSPROC: Read noise = 1.78103 1.90528 1.98192 1.96797 electrons COLLIMATE: Min offset = -0.40 pix COLLIMATE: Max offset = 2.00 pix COLLIMATE: RMS across CCD = 0.66 pix COLLIMATE: Mean offset = 0.49 pix COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of -17.3 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-55090-b2-00101485.log PostScript file = Collimate-55090-b2-00101485.ps Reading /data/spectro/55090/sdR-r1-00101485.fit SDSSPROC: Read noise = 2.29660 2.43170 2.41219 5.59314 electrons Reading /data/spectro/55090/sdR-r1-00101486.fit SDSSPROC: Read noise = 2.28583 2.42320 2.40617 5.58489 electrons COLLIMATE: Min offset = -1.80 pix COLLIMATE: Max offset = 0.80 pix COLLIMATE: RMS across CCD = 0.71 pix COLLIMATE: Mean offset = -0.22 pix COLLIMATE: Camera r1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict (red) piston movement of 2001 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 2001" Log file = Collimate-55090-r1-00101485.log PostScript file = Collimate-55090-r1-00101485.ps Reading /data/spectro/55090/sdR-r2-00101485.fit SDSSPROC: Read noise = 2.47250 2.64433 2.54644 2.62141 electrons Reading /data/spectro/55090/sdR-r2-00101486.fit SDSSPROC: Read noise = 2.46796 2.64516 2.54912 2.65249 electrons COLLIMATE: Min offset = -2.00 pix COLLIMATE: Max offset = 0.80 pix COLLIMATE: RMS across CCD = 0.49 pix COLLIMATE: Mean offset = 0.12 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of -1057 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston -1057" Log file = Collimate-55090-r2-00101485.log PostScript file = Collimate-55090-r2-00101485.ps % Program caused arithmetic error: Floating illegal operand IOP flat-field stripe error --------------------------- im> goDrift Flavor of this run (science, bias, calibration, engineering, ignore, help) []: science Stripe, Strip (e.g, 22,N or SS,N) [0,O]: 0,O Please provide a one line comment [] Flatfield scan Error: can't read "stripe": no such variable im> tb can't read "stripe": no such variable while executing "eqToStripe $ra $dec $stripe..." invoked from within "lindex [eqToStripe $ra $dec $stripe]..." invoked from within "keylset output stripe [lindex [eqToStripe $ra $dec $stripe]..." invoked from within "if {[info exists segue]} { keylset output stripe [lindex [eqToStripe $ra $dec $stripe -segue] 0] } else { keylset output stripe [lindex [eqToStr ..." (procedure "getStripeFromTCC" line 24) invoked from within "getStripeFromTCC" invoked from within "set standardReferenceList [getStripeFromTCC]..." (procedure "checkStripeAgainstTCC" line 19) invoked from within "checkStripeAgainstTCC" invoked from within "if {![checkStripeAgainstTCC]} { puts "Input stripe and strip do not match actual telescope pointing." puts -nonewline "Continue anyway? " ..." invoked from within "if {[cequal $scanType(flavor) science]} { if {![checkStripeAgainstTCC]} { puts "Input stripe and strip do not match actual telescope pointing." (procedure "goDrift" line 38) invoked from within "goDrift" At the end of the flat-field scan after we verified the flavor, stripe, strip, etc., we got the following: im> endDrift 2009-09-16 10:25:15Z: stopping drift with: iccExec id1 id2 id3 id4 id5 id6 ad1 ad2 ad3 ad4 readoutEnd 2009-09-16 10:25:53Z: closed shutters 2009-09-16 10:25:53Z: endData completed Flavor of this run (science, bias, calibration, engineering, ignore, help) [calibration]: Stripe, Strip (e.g, 22,N or SS,N) [0,O]: Mountain estimate of the overall data quality (unknown, bad, acceptable, good) [unknown]: good Please provide a one line comment [] 2009-09-16 10:26:13Z: idReport: problem in calcStripeParams, setting many params to NaN. Error was: invalid keyed list format or inconsistent field name scoping: no value associated with field "no" continuing with idReport anyway Error: error flushing "file14": No space left on device When we tried to take the imager bias, no space to work with: im> goDrift -bias 2009-09-16 10:28:37Z: No heartbeat found from id1 in last 500 of murmur log 2009-09-16 10:28:37Z: 2009-09-16 10:28:37Z: Murmur not working for all id1 node - Losing information!! Error: error writing "file15": No space left on device endNight Problems ----------------- We think the errors in 8039 are a result of an inability to write parameters after the drift because of a lack of disk space? There is an error up above that shows what happened after we ended the drift on that run. Run 8040 was a hung bias attempt that we knew had issues to begin with. We never saw anything read out on the monitors and it seemed doomed to fail from the beginning. im> verifyNoUntransferredGangsInPool 55090 -verbose struct IMAGINGRUN - length mismatch between typedef and input Offending line: imagingrun drifting 0 O 8039 calibration mean 2000 95.0 -25.0 -9999 -9999 117 -9999 -9999 MEAN 2000.0imagingrun drifting 100 O 8040 bias mean 2000 91.32351389 0.05578362 49.92 -9999 -9999 0.0000 0.0000 MEAN 2000.0 26322 15.041060 unknown 55090.44542 START Error: Trouble in param2Chain with args=/astrolog/55090/idReport-55090.par hdr im> checkForMissingRun 55090 struct IMAGINGRUN - length mismatch between typedef and input Offending line: imagingrun drifting 0 O 8039 calibration mean 2000 95.0 -25.0 -9999 -9999 117 -9999 -9999 MEAN 2000.0imagingrun drifting 100 O 8040 bias mean 2000 91.32351389 0.05578362 49.92 -9999 -9999 0.0000 0.0000 MEAN 2000.0 26322 15.041060 unknown 55090.44542 START Error: Trouble in param2Chain with args=/astrolog/55090/idReport-55090.par hdr im> verifyReportWithObserver 55090 id This is what the idReport file thinks happened on 55090: struct IMAGINGRUN - length mismatch between typedef and input Offending line: imagingrun drifting 0 O 8039 calibration mean 2000 95.0 -25.0 -9999 -9999 117 -9999 -9999 MEAN 2000.0imagingrun drifting 100 O 8040 bias mean 2000 91.32351389 0.05578362 49.92 -9999 -9999 0.0000 0.0000 MEAN 2000.0 26322 15.041060 unknown 55090.44542 START Error: Trouble in param2Chain with args=/astrolog/55090/idReport-55090.par reportFileHeader The commands below gave the exact same errors as above: im> verifyImagerFilesOnDisk $mjd -verbose im> createIdApoDiskLog 55090 -verbose im> mailAllRunSummaries $mjd -verbose im> mailAllRunlogs $mjd -verbose