APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday December 25, 2002 (MJD 52634) ---=== OBSERVING TEAM ===--- Steph Snedden (day) Howard Brewington (day) Scot Kleinman Atsuko Nitta French and Ruth Leger Chefs: Steph Snedden Howard Brewington Jurek Krzesinski Scot Kleinman Atsuko Nitta Russet McMillan Bill Ketzeback Hercules Ruth Sabiers ---=== OBSERVING PLAN ===--- Shake and Science! ---=== OBSERVING SUMMARY ===--- It actually cleared after our Christmas Feast, but strong enough winds kept the cold, powdery snow freely blowing, causing the telescope to remain in its building. We finished the shake items we could. We did checkBias and checkLED. I am afraid I did not quite understand the output. Connie, could you check and make sure all is well? The data are kept in /data/photo/ank/52634 for now. Landru crashed twice. We lost site power once and were on the generator as this went out. We ran endNight to test the new version, but do not plan to ship tapes. Please ignore any such request. ---=== OBSERVING LOG ===--- You don't really want to read what we did tonight, right? We know you really came here for the Feast Menu (all home made, of course): Egg rolls Squash soup Fresh green salad Fruit salad ala French Roasted Turkey and Gravy Fresh Cranberry sauce Sourdough Stuffing (what other kind is there?) Mashed potatoes Sweet potato and pecan casserole Cranberry bread Nut bread Christmas Stoellen Polish pancakes Apple sparkling drink Pear sparkling cider Hot Snedden apple cider Pumpkin pie with whipped cream Carrot cake Kenyan coffee Christmas cookies from Jackie (thanks Jackie!) Brownies from Craig who shovelled snow off Russet's house and lit the furnace, then gave her brownies. Soup's a brewin' OK- now for the boring stuff. Around 07:20Z, landru crashed hard (sorry Craig). I could not even ping it. My X session just stopped responding and the machine appeared dead. We did a reset and continued along. Sob. The sky eventualy cleared up, but there was still so much snow around and winds were gusting to 25mph, blowing enough of it around, that we decided not to open. Around 07:38Z, We took a bias and then issued a "getAllFrame" command to copy a frame over so that I can run checkBias command. But before any frames were transferred, we saw the following message over and over in the iop window I used the "getAllFrame" command. im> getAllFrames 3544 20 /data/photo/ank/52634 pgxwin_server: Another server is already active. file /da filling out runlog /astrolog/52634/runlog-3544.txt for run 3544. Waiting for command response from id1 (crate 1, ICC 0), data server (client 1) Waiting for command response from id1 (crate 1, ICC 0), data server (client 1) Waiting for command response from id1 (crate 1, ICC 0), data server (client 1) Waiting for command response from id1 (crate 1, ICC 0), data server (client 1) Few minutes after the message about id1 started to scroll on the iop window, the watcher started to complain about "no gang transfer" so we issued "teamster 60". At 07:55Z, I issued ptvmeTest commands to see if anything was wrong, but all errno=0 and nothing was reported bad. I also issed another "teamster 60 -force" command since I got more "no gang transfer" message in Watcher. Starting up new teamster only resulted in 2002-12-26 08:01:50Z sdsshost TMS 509127 TEXTONLY Gangs transfer: id1:1/1 id2:1/1 id3:1/1 id4:1/1 id5:1/1 id6:1/1 ad1:1/1 ad2:1/1 ad3:1/1 ad4:1/1 -> unix message in the murmurlog which is natural since the imager was not doing anything, hence there should not be any gangs that needed to be transferred. 08:07Z I issued "ptvmeReset". I was still getting "Waiting for ..." message in my iop window and the getAllFrame command issused almost 30min ago was still not happening. "ptvmeReset" seemed to have done nothing. ptvmeTest commands reported nothing wrong. So I decided to reboot the crates. When I returned from the computer room to the control room, the iop window said: Waiting for command response from id1 (crate 1, ICC 0), data server (client 1) Access to shared data area at 0x18000000 lost (errno=0x5 I/O error) Read failed from PT icc10 CSB=0x80420010 rmt=0x1808ABDC cnt=0x00000004 ******************************************************************************** * * * Backplane communications between the ICC/OAC are no longer possible. * * * * Perform the following steps until operations are possible. * * * * 1. Check connections between the machines. * * 2. Reboot (reset) the ICC. * * 3. Restart this (OAC) program. * * 4. Reset the local PT-VME940 (ptvmeMode -R). * * 5. Power-cycle the VMEbus crates. * * 6. Reboot the local machine (host). * * * ******************************************************************************** which I have been expecting to see when I first saw the id1 message on iop. Once the crates came back up, I restarted the servers and reissued the getAllFrame command which worked. Here are the output from the checkBias command: im> checkBias 3544 20 /data/photo/ank/52645 -doghouse Error: /data/photo/ank/52645 does not exist im> checkBias 3544 20 /data/photo/ank/52634 -doghouse {64 WARN: leftBiasLevelComp rightBiasLevelComp } {64 FAIL: } {64 WARN: leftBiasLevelComp rightBiasLevelComp } {64 FAIL: } {65 WARN: } {65 FAIL: oddEvenAmp } {65 WARN: } {65 FAIL: oddEvenAmp } {71 WARN: } {71 FAIL: leftBiasLevelComp rightBiasLevelComp } {71 WARN: } {71 FAIL: leftBiasLevelComp rightBiasLevelComp } {73 WARN: oddEvenAmp } {73 FAIL: } {73 WARN: oddEvenAmp } {73 FAIL: } {12 WARN: rightBiasVector} {12 FAIL: } {56 WARN: rightBiasVector} {56 FAIL: } {83 WARN: leftBiasLevelComp rightBiasLevelComp } {83 FAIL: } {83 WARN: leftBiasLevelComp rightBiasLevelComp } {83 FAIL: } h1 im> Displaying the mosaic bias image did not show anything obviously wrong. Plotting the bias vectors for the ccd that got warning when running checkBias (ccd 64, 65, 71, 73 12, 56 83) showed that there are few high/low points. But I am not sure how significant they are and what we are to do about them. Connie, could you check tell us what you think of the results? I took another bias to make sure all is well after crate reboot, and got the message that the makeRunLog for the previous bias run (which was used for checkBias) failed. Here is the actual message: makeRunLog failed for run 3544. I have the following diagnostics plus check the murmur log and your screen history: ipVectorGetFromRegion: row/col out of range We placed the LED on the imager and did a goDrift (Run 3546). After transferring a frame to HOST, we issued checkLED which gave the following: im> checkLED 3546 20 /data/photo/ank/52634 {43 WARN: } {43 FAIL: leftGain } {46 WARN: leftGain } {46 FAIL: } 0 im> Unlike checkBias, checkLED did not give me the info on the display region. So I did not display the mosaic image of the LED data. The plotFlatvec showed few high and low points for ccd 43 and 46 which got "warn" message from checkLED, but I also saw some high and low points on other CCD that did not get any warning. So again, Connie, can you look at the LED data and make sure all is well? 10:15Z Scot rebooted Landru again after it froze on him again. The same time, we lost site power. We went to the observers' trailer to shut down all computers there since our trailor is not on backup generator. Scot's computer, hello, was already down - apparently his UPS needs help - a site PR was filed (this is important since our Observing and shake plans are on hello). We also took a bias (Run 3547) in doghouse with tbar unlatched. We ran checkBias on this unlatched doghouse bias and got: im> checkBias 3547 20 /data/photo/ank/52634 {64 WARN: leftBiasLevelComp rightBiasLevelComp } {64 FAIL: } {64 WARN: leftBiasLevelComp rightBiasLevelComp } {64 FAIL: } {65 WARN: } {65 FAIL: oddEvenAmp } {65 WARN: } {65 FAIL: oddEvenAmp } {71 WARN: } {71 FAIL: leftBiasLevelComp rightBiasLevelComp } {71 WARN: } {71 FAIL: leftBiasLevelComp rightBiasLevelComp } {73 WARN: oddEvenAmp } {73 FAIL: } {73 WARN: oddEvenAmp } {73 FAIL: } {11 WARN: } {11 FAIL: leftBiasVector rightBiasVector } {12 WARN: rightBiasVector} {12 FAIL: } {15 WARN: rightBiasVector} {15 FAIL: } {36 WARN: leftBiasVector rightBiasVector} {36 FAIL: } {51 WARN: } {51 FAIL: leftBiasVector rightBiasVector } {53 WARN: rightBiasVector} {53 FAIL: } {56 WARN: rightBiasVector} {56 FAIL: } {83 WARN: leftBiasLevelComp rightBiasLevelComp } {83 FAIL: } {83 WARN: leftBiasLevelComp rightBiasLevelComp } {83 FAIL: } {94 WARN: oddEvenAmp } {94 FAIL: } {94 WARN: oddEvenAmp } {94 FAIL: } {95 WARN: leftBiasLevelComp } {95 FAIL: } {95 WARN: leftBiasLevelComp } {95 FAIL: } h59 At 11:00Z, we decided to put the telescope to bed. The manipulator had problems moving back and forth. The cartridge elevator also did not move. So we did not get a cartridge from he bay to mount on the telescope. Instead, we left the telescope with only corrector lens on. The counter weight is at the lowest level to balance the telescope. We filed a PR. We ran endNight to test the new IOP's endNight. It worked, but produced an error/warning at the end: Error: Found a problem with ioptape.archlog! The tapes may not have written correctly! Unless this error proves to be real, we recommend running this run with the previous version of IOP. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 02:31 17556 0.0 bias 07:29 17557 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL NONE Stays: JL ALL ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 01:58Z 24 -21 7 349 (NNW) 8 - 02:29Z 22 -14 13 10 (N) 46 - 03:03Z 22 -4 11 355 (NNW) 61 - 03:35Z 20 2 15 25 (NNE) 83 - 04:07Z 20 7 15 12 (NNE) 88 - 04:39Z 21 7 4 58 (ENE) 103 - 05:12Z 21 5 11 1 (N) 98 - 05:44Z 22 2 13 320 (NW) 96 - 06:16Z 20 4 12 351 (NNW) 87 - 07:45Z 20 4 14 346 (NNW) 94 - 11:51Z 18 0 16 321 (NW) 61 - ---=== TELESCOPE STATUS ===--- in the enclosure with corrector mounted only open with fans running most of the night, but with enclosure on closed ~11:00Z ---=== SOFTWARE USED ===--- IOP/SOP: v3_101_0 Watcher: v2_20_0 MCP: v5_17_0 TPM: tpm_v2_19_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_59 SoS: v4_9_13 hoggPT: v1_5_8 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0880 0.0790 Axial B 0.8010 0.7970 Axial C 0.8180 0.8240 Trans D -9.0790 -9.1130 Lateral E 1.8300 1.8870 Lateral F 1.3970 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: -5.1 Alt.: 30.061305 MIGS TONIGHT NOMINAL Axial A 1.5140 1.5170 Axial B 1.1560 1.1560 Axial C 1.0990 1.0990 Trans D -0.6980 -0.7070 GALILS Commanded: 1619869. 1555177. 1576405. -3400. -7900. Actual: 1619650. 1555250. 1576350. -3400. -7900. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1257.06 -0.02 -20.10 -0.54 130.90 ---=== PROBLEMS IN DETAIL ===---