APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday January 21, 2003 (MJD 52661) ---=== OBSERVING TEAM ===--- Swing: Howard Brewington Night: Dan Long, Pete Newman Support: Scot Kleinman (phone), Eric Neilsen (FNAL, phone) ---=== OBSERVING PLAN ===--- Shakedown night 1. ---=== OBSERVING SUMMARY ===--- Did not open but tested in the enclosure. Nasty problem with the critical alerts system encountered. ---=== OBSERVING LOG ===--- Humidity, clouds and a snow flurry kept us closed all night, but we tested what we could in the enclosure. Light leaks in spectrograph sp1 seem solved. A 15-min dark (exposure # 18253) was taken after the end of evening twilight with no cartridge mounted but the slithead doors closed and only stop-lights on in the enclosure. The counts were reassuringly low and consistent between sp1 and sp2, and examination of the raw frames shows no sign of any flares. Both mirrors were homed and relaxed at alt=70deg using relax20 procedure. The mrror numbers look good, but we still need to get on the sky with the imager to check collimation more precisely. sdssProcedures v1_61 tests: --------------------------- All changed noted in the change log seem to be in place. IOP v3_103_0 tests: ------------------- PR 4604 - Add a legend to seeingRun - closed. Works as required. PR4501 - endNight trapping of tape failures - PR updated but state unchanged. This test involves making endNight deliberately fail. As noted in the shakeup at the end of the last run, we think it is pointlessly dodgy to test this at night when we might jeopardize real data. So, we ask that the test be done during the day. PR 4676 - Critical error emails. That PR unchanged, and a new one opened on the watcher. After modifications to the way the aleaerts email was (to be generated, we repeated the test of forcing a fill of the spectrograph secndary dewars with the valve to the 180L LN2 dewar closed. Confusion ensued! See problems section #2 for details. TPM v2_22_0 tests: ------------------ PR 4494 - TPM alarms - unchanged yet. Most of the "false" alarms seen in recent runs are indeed suppressed (at least with the telescope in the enclosure). We did see the following flagged by watcher: dewar (MAJOR: 2003-01-21 23:58:13Z) tracking (MINOR: 2003-01-21 23:56:28Z) m2 mig (MAJOR: 2003-01-22 01:13:51Z) dewar (MAJOR: 2003-01-22 03:08:01Z) The tracking alarm may have been while recording mirror positions with the telescope not really tracking. The M2 positions, however, gave us no cause for concern and the LN2 dewars had plenty of gas in them, so we are not sure what that alarm means (failure to read the weight scales?). We did NOT complete the caGet1 command asked for in the shake request, as we did not open for normal operation yet. Imager checkBias and checkLED tests: ------------------------------------- We got a worrying number of error messages from the checkLED test, although we do not know what they mean. We will email these to Connie for edification. The CCD T-bars were unlatched, and we visually confirmed the LEDs were on before starting (and that was obvious from the video monitors, too). We also tried a checkBias in the doghouse with the T-bars unlatched and got a similar number of errors. The errors reported with and without the -doghouse flag on the checkBias command were slightly different. So, we tried again in the doghouse with the T-bars latched and got much more sensible results, although also a few errors. See problem section #1 for details. We ran endNight on the imager and spectro biases taken tonight as a test of endNight for version v3_103_0. endNight completed normally for the sample biases on the imager and spectrograph. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3593 21:06Z 21:22Z 100 O 146.59 150.52 40 ignore 3594 03:15Z 03:33Z 100 O -120.97 -116.41 44 bias DH, unlatched 3595 04:00Z 04:30Z 100 O -109.59 -101.99 65 engineering DH, LEDs 3596 05:08Z 05:18Z 100 O -92.69 -90.05 30 ignore 3597 05:20Z 06:24Z 100 O -89.65 -73.51 122 ignore DH, latched ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 18247, plate -9999 ------- 19:30 18247 0.0 dark 21:06 18248 0.0 bias ----- sequence 18249, plate -9999 ------- 21:15 18249 0.0 bias 02:25 18251 0.0 bias 02:28 18252 0.0 bias ----- sequence 18253, plate 1 ------- 02:46 18253 0.0 dark ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- No tapes to ship. Spectro/gangs data written to the cloudy night tapes. ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 20:30Z 37 28 9 269 (W) 76 - Variable cloud 21:07Z 41 27 6 285 (WNW) 82 - " 21:39Z 39 26 4 243 (WSW) 98 - " 22:11Z 39 26 5 284 (WNW) 95 - " 22:43Z 39 27 7 302 (WNW) 65 - " 23:15Z 35 24 1 301 (WNW) 70 - " 01:39Z 32 27 7 351 (NNW) 59 - " 02:15Z 32 26 2 340 (NNW) 69 - " 02:48Z 32 25 2 266 (W) 61 - " 03:20Z 33 25 1 298 (WNW) 52 - " 03:52Z 33 26 6 333 (NNW) 50 - " 04:24Z 33 24 2 322 (NW) 64 - " 04:57Z 33 23 1 355 (NNW) 72 - " 05:29Z 33 23 0 239 (WSW) 81 - " 06:01Z 33 24 6 267 (W) 97 - " 06:34Z 33 24 7 267 (W) 86 - " 07:06Z 33 20 10 232 (SW) 76 - " 07:42Z 33 20 9 32 (NNE) 61 - " 08:15Z 34 24 6 348 (NNW) 53 - " 08:49Z 32 21 5 334 (NNW) 66 - " 09:22Z 33 23 9 316 (NW) 81 - Snow? 09:54Z 31 24 9 8 (N) 102 - Variable clouds 10:26Z 29 27 18 34 (NE) 114 - " 10:58Z 31 25 11 6 (N) 52 - " 11:31Z 30 24 16 34 (NE) 21 - " 12:03Z 31 22 7 334 (NNW) 30 - ---=== TELESCOPE STATUS ===--- Fans were on and doors open for various times, but we did not open. Status at 12:14Z: Telescope stowed at: 30 deg Instrument mounted: Engineering camera Counterweights at: 110 Autofill systems: On 180L LN2 dewar weights: Spectro 28 lb (just filled) Imager 41 lb (just filled) Tthe grade 2 nitrogen that presurizes the LN2 dewar lost pressure rapidly again today and was replaced. ---=== SOFTWARE USED ===--- IOP/SOP: v3_103_0 (test) Watcher: v2_20_0 MCP: v5_17_0 TPM: tpm_v2_22_0 (test) AstroDa: v14_47 TCC: 2.6.8 sdssProcedures: v1_61 SoS: v4_9_13 hoggPT: v1_5_8 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0860 0.0790 Axial B 0.8030 0.7970 Axial C 0.8180 0.8240 Trans D -9.0780 -9.1130 Lateral E 1.8780 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 0.2 Alt.: 29.999993 MIGS TONIGHT NOMINAL Axial A 1.5190 1.5170 Axial B 1.1550 1.1560 Axial C 1.0950 1.0990 Trans D -0.7020 -0.7070 GALILS Commanded: 1619598. 1555044. 1576342. -3400. -7900. Actual: 1619650. 1555250. 1576350. -3400. -7900. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1256.93 -0.06 -20.05 -0.51 130.92 ---=== PROBLEMS IN DETAIL ===--- Problem #1 - Failures and warnings from checkLED and checkBias -------------------------------------------------------------- The following is a list of the error output and locations of data. Ran a bias in the doghouse with tbars unlatched ( 3594 ). Put frame 20 in /data/photo/optics/biascheck. Complaints about the data whether I used the -doghouse option or not. No doghouse: im> checkBias 3594 20 /data/photo/optics/biascheck -astro {65 WARN: } {65 FAIL: oddEvenAmp } {65 WARN: } {65 FAIL: oddEvenAmp } {11 WARN: } {11 FAIL: leftBiasVector rightBiasVector } {51 WARN: } {51 FAIL: leftBiasVector rightBiasVector } {56 WARN: rightBiasVector} {56 FAIL: } {photo h5} {astro h6} -doghouse: im> checkBias 3594 20 /data/photo/optics/biascheck -astro -doghouse -outputDir dashdoghouse {65 WARN: } {65 FAIL: oddEvenAmp } {65 WARN: } {65 FAIL: oddEvenAmp } {26 WARN: rightBiasVector} {26 FAIL: } {56 WARN: rightBiasVector} {56 FAIL: } {photo h54} {astro h124} Run 3595 is drift taken with LED illuminator on. Frame 25 is in /data/photo/optics/checkLED checkLED threw even more errors. tbars were unlatched for this run as well. im> checkLED 3595 25 /data/photo/optics/checkLED {11 WARN: } {11 FAIL: rightGain } {12 WARN: } {12 FAIL: leftGain rightGain } {13 WARN: leftGain } {13 FAIL: rightGain } {14 WARN: leftGain } {14 FAIL: rightGain } {16 WARN: leftGain } {16 FAIL: rightGain } {21 WARN: } {21 FAIL: rightGain } {22 WARN: } {22 FAIL: rightGain } {23 WARN: } {23 FAIL: leftGain rightGain } {24 WARN: } {24 FAIL: leftGain rightGain } {25 WARN: } {25 FAIL: leftGain rightGain } {26 WARN: } {26 FAIL: leftGain rightGain } {32 WARN: } {32 FAIL: leftGain } {34 WARN: } {34 FAIL: leftGain } {35 WARN: } {35 FAIL: leftGain } {36 WARN: leftGain } {36 FAIL: } {41 WARN: leftGain } {41 FAIL: rightGain } {42 WARN: } {42 FAIL: leftGain } {43 WARN: } {43 FAIL: leftGain rightGain } {44 WARN: } {44 FAIL: rightGain } {45 WARN: } {45 FAIL: leftGain rightGain } {46 WARN: } {46 FAIL: leftGain } {51 WARN: } {51 FAIL: leftGain rightGain } {52 WARN: } {52 FAIL: leftGain rightGain } {53 WARN: rightGain } {53 FAIL: } {55 WARN: rightGain } {55 FAIL: leftGain } {56 WARN: leftGain rightGain } {56 FAIL: } 0 Run 3597 is a bias taken in the doghouse with the tbars latched. Saved frame 20 in /data/photo/optics/checkBias Still a few complaints from checkBias. im> checkBias 3597 20 /data/photo/optics/checkBias -astro -doghouse {65 WARN: } {65 FAIL: oddEvenAmp } {65 WARN: } {65 FAIL: oddEvenAmp } {71 WARN: leftBiasLevelComp rightBiasLevelComp } {71 FAIL: } {71 WARN: leftBiasLevelComp rightBiasLevelComp } {71 FAIL: } {56 WARN: rightBiasVector} {56 FAIL: } {photo h146} {astro h59} Problem #2 - Watcher failed to report deliberate critical errors. ----------------------------------------------------------------- Here is a copy of the text in PR 4707 filed on the watcher: Something mighty strange and mighty disturbing is going on with the watcher and critical errors. We can't seem to get the watcher to complain when one is occurring. This may or may not be a watcher problem. If not, please reassign this critical high PR to where it belongs. We attempted to test the critical error alert system by forcing a critical error on the spectrographs. We closed the liquid valve on the 180 L dewar and forced a fill. The secondary dewar pressure soon dropped to zero and 2NDARY_DEWAR_PRESS FAULT!!! messages began showing up in the murmur log. The watcher remained silent. No system errors of any kind. The camchecks from the servers were clearly causing the error messages to show up in the murmur log. Since this same scenario caused critical errors to be flagged by the watcher during shakeup only a week ago, we decided to revert to the software versions used during shakeup and see if the watcher started to send out the alarm. Using version v3_102_0 of iop for both the servers and the sop session, v2_20_0 of watcher ( and later reverting the tpm to tpm_v2_19_0 as well ), we still could not get the critical error to fire in the watcher. We should have seen secondary dewar pressure, primary fill fault and even ccd and ln2 temperatures out of spec, and never did. A possible clue to this behavior is that while plots of various parameters of CCD and LN2 temperatures showed new data coming in over time, the values seen in the iop_ccd window ( active or not ), never changed, and for some watcher clients the squares were blank, showing no data at all. This was true of watcher run both on host and commish. Packets were still coming in to the watcher as the time of last packet was always very close to the current time, and things like mcp, tcc, and weather errors continued to come in. Another clue to the problem is that when putting the iop_ccd window to active, the following errors show up in the watcher client window. The error is the same on either host or commish. watch> can't read "voltage_info(voltage_CCD_name)": no such element in array can't read "voltage_info(voltage_CCD_name)": no such element in array while executing "set type $voltage_info(voltage_CCD_name)..." invoked from within "if {$type == "ccd"} { set type $voltage_info(voltage_CCD_name) } elseif {$type == "volt"} { set type $voltage_info(CCD_voltage_name) ..." (procedure "show_voltages" line 4) invoked from within "show_voltages ccd $just_update" (procedure "show_voltages_ccd" line 2) invoked from within "show_voltages_ccd 1" ("eval" body line 1) invoked from within "eval $procname $args" show_voltages_ccd is scheduled, further errors sent to murmur It may be that rebooting host will cure the problem. We chose not to do this so that the current state could be investigated.