APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday February 16, 2005 (MJD 53418) ---=== OBSERVING TEAM ===--- Atsuko Nitta (swing) Night Dan Long Steph Snedden Support (doing the hard work): Craig Loomis JB ---=== OBSERVING PLAN ===--- Shakeup! ---=== OBSERVING SUMMARY ===--- Clouds/sprinkles precluded opening tonight. This shakeup night was spent battling ptvme problems and testing the calibrator procedure. The day started with strange iop error messages related to dscStatus and pool full errors. This was traced to ptvme connectivity troubles. After a fair amount of swapping and swearing, we got up and running around 03:30Z on the original host and ptvme card (those in use prior to last weeks problems). We proceeded with the standard bias and flat checks which completed normally. Next we went through the calibrator procedure and set up to take QE data on r1. It got about 2/3 of the way through and stopped taking data. After a FATAL dscArch error message, the imager/host/DA stopped talking to each other properly such that the imager could not be read out. The premature end to the sequence means no useful calibration data were acquired. Attempting a bias drift from other iops stalled at initData. We finally got a drift to work after quitting the iop which first had trouble performing initData for a drift. ---=== OBSERVING LOG ===--- Afternoon ============ JB and Norm put the imager on the telescope so that we can start doing the imager calibration. But upon rebooting host and initNighting, I found that I could not start a drift because of : im> goDrift -bias Error: {id1 { No such status entry defined}} im> Then another try got: im> goDrift -bias No such status entry defined Error: Aborting because pool is full So we pooInit. But still the errors persisted. We rebooted the photo1 crate, but then we saw the same error for ad1. im> goDrift -bias Error: {ad1 { No such status entry defined}} im> At this point, we decided to ask help from Craig. After an hour of groaning, Craig found a bus error from the clue that was given to him via the tb message when Dan's initNight failed in sop. Steph made the observations on how we are like the road repair people where you often find one person digging while 5 people watch -- while Craig worked, we stood around him and watched him work. Since it is a shake night and very cloudy outside, we were not in hurry to get on sky. So Craig decided to debug to find out what is really wrong. He with help from JB reconfigured one of the spare ptvme boards. Then he swapped this one into the spareHost we have been using this past week or so. spareHost did not come up with the ptvme board Craig configured. spareHost had a prom failure. Cloomis will call SGI tomorrow. So we are back on original host with the the new configured ptvme board. This booted up fine. To be safe, JB rebooted the crates too. Craig and JB then tried to start the servers, but we got ptvme failure errors despite of the fact ptvmeTest reports no problem. Craig and JB headed back to the computer room. They fiddled with the configuration of the ptvme card. No change. :( :( :( Craig is perplexed. Craig and JB then put in the card that was in host last week. Final configuration is the real original host with the real original host ptvme card. The net effect since before the astro crate problem last week is that the astro crate ptvme card was swapped. Tonight, Craig and JB found that one of our spare ptvme card is bad and that the spare Challenge needs service. Cloomis has already placed a service call. Night: ====== 03:20Z Host is back to the original machine with original ptvme card and we are up and running. What follows is Craig's summary of the host/ptvme work: We started the evening running on the spare SGI with PTVME # 540651. Pervasive problems with connectivity to all crates, easily repeatable by running some statusDump (oh, ccd*, pool*, scd*) a number of times: on all crates, the command would occasionally return no status entries (i.e {}). After the usual stupid reboots, we tried swapping in PTVME # 540659. The spare SGI then repeatedly failed to boot, due to a very early PROM exception. I had seen the same PROM error once before with an earlier 'spare configuration, so we gave up on 'spare, and installed 540659 into the real 'host ('host had been taken off line during last week's astro crate work). 'host booted fine and the usual ptvmeTests worked, but no iops/servers would start, complaining about ptvme errors. Some quick tests revealed that no addresses above 0x180000fff could be accessed on any crate. We tried a variant PTbug configuration (note that the config page http://sdsshost.apo.nmsu.edu/astroda/doc/www/href/ptvme/doc/www/ ptvme.install.hw.html is ambiguous), but that did not help. So we swapped in PTVME # 541015 -- the card which had been in 'host before the astro hoo-hah. That appeared to work: biases are scrolling by. Net effect, from before the astro crate work: - the astro crate PTVME card has been swapped - PTVME card 540659 is likely broken. - 'spare needs a service call. - craig 03:40Z We start the monthly imager checkBias biasCheck checkFlat. Which brings to mind the age old question... How many biases could a biasCheck check if a biasCheck could checkBias? checkBias errors: none checkFlat errors: {42 WARN: leftGain } {42 FAIL: } biasCheck Results: 45. 53389 46. 53418 Select index of reference mjd [46]: 45 Writing file /astrolog/53418/idBiasCheck-005160.par CCD 11 Offset. Left: -0.46 +/- 0.50 Right: -0.88 +/- 0.33 CCD 21 Offset. Left: -1.12 +/- 0.32 Right: -1.54 +/- 0.50 CCD 31 Offset. Left: -2.00 +/- 0.00 Right: -0.39 +/- 0.50 CCD 41 Offset. Left: -2.00 +/- 0.00 Right: -1.00 +/- 0.00 CCD 51 Offset. Left: 3.78 +/- 0.49 Right: 3.86 +/- 0.38 CCD 12 Offset. Left: -1.12 +/- 0.32 Right: -2.00 +/- 0.00 CCD 22 Offset. Left: -2.00 +/- 0.00 Right: -2.00 +/- 0.00 CCD 32 Offset. Left: 13.77 +/- 0.42 Right: -0.89 +/- 0.31 CCD 42 Offset. Left: -2.00 +/- 0.00 Right: -1.00 +/- 0.00 CCD 52 Offset. Left: -1.00 +/- 0.00 Right: 0.00 +/- 0.00 CCD 13 Offset. Left: -3.00 +/- 0.00 Right: -3.00 +/- 0.00 CCD 23 Offset. Left: -1.76 +/- 0.43 Right: -2.00 +/- 0.00 CCD 33 Offset. Left: 2.00 +/- 0.00 Right: -0.75 +/- 0.44 CCD 43 Offset. Left: -0.51 +/- 0.50 Right: 0.00 +/- 0.00 CCD 53 Offset. Left: -1.00 +/- 0.00 Right: -1.00 +/- 0.00 CCD 14 Offset. Left: -1.00 +/- 0.00 Right: -1.00 +/- 0.00 CCD 24 Offset. Left: -1.44 +/- 0.50 Right: -2.00 +/- 0.00 CCD 34 Offset. Left: 2.78 +/- 0.48 Right: 2.82 +/- 0.41 CCD 44 Offset. Left: -1.00 +/- 0.00 Right: -0.22 +/- 0.42 CCD 54 Offset. Left: -2.00 +/- 0.00 Right: -2.00 +/- 0.00 CCD 15 Offset. Left: -1.34 +/- 0.47 Right: -2.00 +/- 0.00 CCD 25 Offset. Left: -3.15 +/- 0.36 Right: -3.14 +/- 0.35 CCD 35 Offset. Left: -0.42 +/- 0.94 Right: -0.21 +/- 0.41 CCD 45 Offset. Left: -1.00 +/- 0.00 Right: 0.00 +/- 0.00 CCD 55 Offset. Left: -1.61 +/- 0.49 Right: -1.15 +/- 0.36 CCD 16 Offset. Left: -4.00 +/- 0.00 Right: -4.00 +/- 0.00 CCD 26 Offset. Left: -2.00 +/- 0.00 Right: -1.90 +/- 0.31 CCD 36 Offset. Left: -0.75 +/- 0.46 Right: -1.00 +/- 0.00 CCD 46 Offset. Left: -2.00 +/- 0.00 Right: -2.78 +/- 0.41 CCD 56 Offset. Left: -2.00 +/- 0.00 Right: -2.09 +/- 0.29 CCD 60 Offset. Left: -8.33 +/- 0.47 Right: -8.39 +/- 0.49 CCD 61 Offset. Left: -4.00 +/- 0.00 Right: -4.00 +/- 0.00 CCD 62 Offset. Left: -9.79 +/- 0.41 Right: -9.86 +/- 0.35 CCD 63 Offset. Left: -3.77 +/- 0.43 Right: -3.83 +/- 0.37 CCD 64 Offset. Left: -7.21 +/- 0.52 Right: -7.33 +/- 0.56 CCD 65 Offset. Left: 0.05 +/- 0.42 Right: 0.45 +/- 0.50 CCD 71 Offset. Left: -21.89 +/- 0.31 Right: -22.00 +/- 0.00 CCD 72 Offset. Left: -8.08 +/- 0.27 Right: -8.03 +/- 0.37 CCD 73 Offset. Left: 3.79 +/- 0.41 Right: 3.72 +/- 0.45 CCD 74 Offset. Left: -12.13 +/- 0.34 Right: -12.00 +/- 0.00 CCD 75 Offset. Left: 2.00 +/- 0.00 Right: 2.00 +/- 0.00 CCD 76 Offset. Left: -3.00 +/- 0.00 Right: 13.35 +/- 129.95 CCD 81 Offset. Left: -7.00 +/- 0.00 Right: -7.17 +/- 0.38 CCD 82 Offset. Left: -9.21 +/- 0.41 Right: -9.19 +/- 0.39 CCD 83 Offset. Left: -9.24 +/- 0.43 Right: -9.27 +/- 0.45 CCD 84 Offset. Left: -5.00 +/- 0.00 Right: -4.78 +/- 0.42 CCD 85 Offset. Left: -4.48 +/- 0.51 Right: -4.47 +/- 0.50 CCD 86 Offset. Left: -5.82 +/- 0.41 Right: -5.88 +/- 0.34 CCD 90 Offset. Left: -4.27 +/- 0.44 Right: -4.00 +/- 0.00 CCD 91 Offset. Left: -3.59 +/- 0.49 Right: -3.59 +/- 0.49 CCD 92 Offset. Left: -5.00 +/- 0.00 Right: -4.87 +/- 0.33 CCD 93 Offset. Left: -7.15 +/- 0.36 Right: -7.18 +/- 0.39 CCD 94 Offset. Left: -6.00 +/- 0.00 Right: -6.00 +/- 0.00 CCD 95 Offset. Left: -10.00 +/- 0.00 Right: -10.08 +/- 0.27 Calibrator Procedure Shake: Once the biases and flats were finished, we moved on the to only shake item, going through the calibrator procedure. After a little confusion on the starting point for the manual stage we got the calibrator set up to do QE on r1. There are some questions about the need to relabel tapes and initialize the pool prior to taking data. Are these just for tidiness or are there other reasons to do this? Regarding clearing the pool, it seems an unnecessary and potentially harmful step (and may have indirectly caused the halt of the script). If the frames are archived they should be flagged for being overwritten, so the pool should never run out of space. If the pool must be initialized, it needs to be made clear that the calibration scripts clear the pool immediately on starting. This is important because if one doesn't transfer the biases and darks taken _prior_ to running a calibration they will be lost. There seems to be a considerable lull in activity after frame 37. I would have thought it was finished except that I haven't got a prompt back yet. The last command was at 08:19:53Z, and was scheduling gangs2Unix transfers. The next command should have been the readoutBegin. There are no error messages. The murmur log shows the "starting da..." line, but no dLine entries follow Are we having more DA/ptvme problems? The murmur log has an unhappy message just before the last frame of the calibration sequence. 2005-02-17 08:16:00Z sdssid3 dCtrlSrv dsc_S_poolInited Frame Pool initialized and empty 2005-02-17 08:16:00Z sdssid3 dArch dsc_E_ivFrameCtx Invalid Frame context 2005-02-17 08:16:00Z sdssid3 dArch TEXTONLY *** FATAL *** dscArch: While double writing frames 2005-02-17 08:16:00Z sdssid3 dArch TEXTONLY ---> Frame 5162 37.53 not archived This was the only sign of discontent. All attempts at goDrift met with silence - no errors, but no activity. We should put the port for the MSP in the calibration procedure so it can be logged out in the event of the iop session with the instruments grabbed dying. We logged out the imager port and the servers were able to talk to it. All seemed well. Then we tried to take a normal bias. it stalled after saying "DA not setup for drifting, performing an initData". This was at 09:43Z and Steph's iop IDA was 3568. We logged the imager port out again. Two other attempts met with similar results. At this point Steph closed the window she tried a bias drift in and hey-presto the iop I had tried to start a drift in went on to set PBINSET etc and start drifting. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 5160 03:23Z 03:40Z 100 O -92.75 -88.57 41 bias 5161 03:45Z 04:01Z 100 O -87.39 -83.23 41 calibration 5163 10:58Z 11:07Z 100 O 21.38 23.48 27 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 30916, plate -9999 ------- 03:54 30916 0.0 bias 03:58 30917 0.0 bias ----- sequence 30918, plate -9999 ------- 10:23 30918 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JLcloudy1 Stays: JLcloudy2 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:36Z 40 28 7 270 (W) -9999 - Cloudy 00:09Z 39 29 6 270 (W) -9999 - Cloudy 03:50Z 39 21 7 263 (W) -9999 - Cloudy 04:22Z 41 20 8 270 (W) -9999 - Mostly Cloudy 04:54Z 41 20 8 275 (W) -9999 - Partly Cloudy 05:27Z 41 18 8 280 (W) -9999 - Mostly Cloudy 05:59Z 40 17 12 317 (NW) -9999 - Cloudy 06:31Z 40 22 6 280 (W) -9999 - Cloudy 07:03Z 38 21 2 208 (SSW) -9999 - Mostly Cloudy 07:36Z 41 20 8 267 (W) -9999 - Cloudy 08:08Z 41 22 11 275 (W) -9999 - Cloudy 08:40Z 39 23 8 266 (W) -9999 - Sprinkles 09:12Z 38 27 14 238 (WSW) -9999 - Cloudy 10:17Z 38 26 9 258 (WSW) -9999 - Cloudy 10:49Z 38 24 14 278 (W) -9999 - Cloudy 11:21Z 37 24 15 251 (WSW) -9999 - Cloudy ---=== TELESCOPE STATUS ===--- Enclosure never opened Stowed at 30 deg alt, cartridge 3 installed. Spectro auto-fill connected. Spectro dewar 11:40Z Dewar wt 284 lb Imager dewar 11:40Z Dewar wt 239 lb No interlocks bypassed. ---=== SOFTWARE USED ===--- IOP/SOP: v3_143_0a Watcher: v2_30_0 MCP: v5_26_0 TPM: tpm_v2_47_0 AstroDa: v14_47 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v1_97 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5790 0.5790 Axial B 0.5880 0.5880 Axial C 0.9460 0.9460 Trans D -8.2000 -8.1950 Lateral E 10.0457 10.0076 Lateral F 11.6078 11.5824 GALILS Commanded: 4300. -1900. 2850. -13300. -7600. -7650. Actual: 4299. -1917. 2868. -13303. -7614. -7660. SETMIR VALUES PriDesOrient: 0.00 4.57 1.18 64.25 -249.76 PriOrient: 0.00 4.56 0.88 65.00 -249.58 SECONDARY: ---------- Focus: 0.00 Air Temp.: 4.4 Alt.: 30.000611 MIGS TONIGHT NOMINAL Axial A 1.5700 1.5710 Axial B 1.0520 1.0490 Axial C 1.1600 1.1610 Trans D -0.1920 -0.2070 GALILS Commanded: 1620915. 1602269. 1527114. -13800. -9950. Actual: 1619188. 1604973. 1528571. -13820. -9964. SETMIR VALUES SecDesOrient: 1257.00 27.97 -10.00 -90.00 257.89 SecOrient: 1256.36 28.04 -11.39 -90.85 257.45 PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5810 0.5790 Axial B 0.5900 0.5880 Axial C 0.9450 0.9460 Trans D -8.1960 -8.1950 Lateral E 10.0457 10.0076 Lateral F 11.6078 11.5824 GALILS Commanded: 4300. -1900. 2850. -13300. -7600. -7650. Actual: 4300. -1900. 2850. -13300. -7600. -7650. SETMIR VALUES PriDesOrient: 0.00 4.57 1.18 64.25 -249.76 PriOrient: 0.00 4.56 0.88 65.00 -249.58 SECONDARY: ---------- Focus: 0.00 Air Temp.: 5.4 Alt.: 30.000716 MIGS TONIGHT NOMINAL Axial A 1.5700 1.5710 Axial B 1.0480 1.0490 Axial C 1.1590 1.1610 Trans D -0.1910 -0.2070 GALILS Commanded: 1619202. 1604953. 1528550. -13800. -9950. Actual: 1619200. 1604950. 1528550. -13800. -9950. SETMIR VALUES SecDesOrient: 1257.00 27.97 -10.00 -90.00 257.89 SecOrient: 1256.99 27.97 -10.01 -89.95 257.50 ---=== PROBLEMS IN DETAIL ===---