APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday January 24, 2002 (MJD 52299) ---=== OBSERVING TEAM ===--- Presenting Swinging Pete Newman Starring Dan Long Eric Neilsen Also Starring Scot Kleinman Cameo Appearances by Jurek Krzesinski Atsuko Nitta Steph Snedden Mike Harvanek ---=== OBSERVING PLAN ===--- Shakeup! ---=== OBSERVING SUMMARY ===--- High humidity early on, and snow coming off the trees and blowing around kept us closed all night. Despite the lack of sky, several PRs were tested and closed. sop/467: Wrong exposure number in sdR file name. sop/2918: SOP doesn't monitor the mechanicals serial lines. sop/2975: Offsets I section should include axis position. iop/2919: Loading saved nightlog into logTool causes " and others to show up iop/2927: A proposal to allow an easy way to insert timestamps in logTool. iop/2872: Can logTool's imaging summary include start/stop times? iop/2859: Could exiting out of IOP treat a running logTool a little nicer? iop/2738: logTool should not changed entered text. mcp v5_9_0 and v5_9_1 have bugs which cause bus errors within a few minutes of booting. Reverting to v5_8_7 restored normal operation. TCC 2.6 tested as well as could be done in the enclosure. It showed no problems initializing, tracking, setting instruments, moving mirrors. No software declared current. MCP v5_9_1 doesn't run. IOP v3_66_4 depends on astroda and so will remain -test pending on sky testing. Here is the status of the software. current latest IOP/SOP: v3_64_3c v3_66_4 Watcher: v2_13_2a v2_13_2a MCP: v5_8_7 v5_9_1 TPM: v2_4_5 v2_4_5 AstroDa: v14_40 v14_41 TCC: 2.6 2.6 sdssProcedures: v1_38 ---=== OBSERVING LOG ===--- This afternoon setting up the new DA, after dscSwSys only id1 and id4 came up. Each of the other nodes were rebooted individually by dsclat and ctrl-x. Should dscSwSys reboot all the nodes or only the master nodes on each crate? In the instrument portion of the afternoon check, the following was seen in response to an attempt to take a frame. Flushing charge for 15 sec creating BEGIN appending END EXPOSURE TIME is: 0 1011908951 1011908951 creating opConfig parameter file /astrolog/52299/opConfig-52299.par There already is a teamster, PID 30551070, owned by long The running teamster is too old, but cannot be stopped automatically due to permissions problems. Please get rid of the running teamster! Seq 12866 Exp 12866 Frame 12866 Killing the process listed by hand worked, but why couldn't the observer account iop kill a teamster started by long? logTool did not come back when switching away and back to the desktop it was on. The window just remained a blank gray for several minutes until the window was quit and a new logTool started. This was almost certainly due to host bogging down. It seemed that several times during the first few hours of the night that host would get very slow. goStare failed early on due to a new check for DA .vs. SOP exposure number. Apparently there are times in the goStare process where they are *supposed* to differ by one. Eric chased this down and created a new version of IOP with this fix in. The mcp would not stay running using either version v5_9_1 or v5_9_0. It would hang on bus errors within 10 minutes or so of being launched. The detailed sequence of events and output from the console port showing the bus errors is in the problems section. Reverting to v5_8_7 restored normal operation. There was more inconclusive temperature testing of the M2 actuators. The secondary was moved many times during the night as we were testing Scot's scaleChange code. The secondary always moved properly. Unfortunately the cold night which was forecast never really materialized. Temperatures warmed to about -3 and stayed there. Not quite the -5 or colder which was desired. The exhaust fan was run from about 05UT until 11:30UT with the west door slightly cracked to cool the enclosure as much as possible and still minimize the risk of blowing snow entering. sop/467: Wrong exposure number in sdR file name. ------------------------------------------------ An oldie but a goodie. Problem fixed. PR closed. First attempt at a fix failed because it was assumed that the DA and SOP should always agree on the exposure number. Apparently there are times when they are supposed to differ. When this was taken into account goStare worked properly. The new code was tested by purposely setting the DA and SOP exposure numbers to different values and then running a goStare. First SOPs exposure number was incremented. When goStare was issued, it correctly detected the difference and asked if the user wanted to increment the lower number to make them the same again. Yes was given and the exposure indeed had agreement between filename and header info when it was written. Next the DA exposure number was manually incremented. A goStare was issued and again responded properly to the error and offered to fix the discrepancy. It did so correctly as seen in the agreement between header and filename of the bias frame which was written out. sop/2918: SOP doesn't monitor the mechanicals serial lines. ------------------------------------------------------------ Tested by issuing a mechSend s and checking the reply in the murmur log. It appears that the desired information is now logged. Jon is this what you wanted to see in the murmur log? Jan 24 19:28:38 sdsshost IDA 313546 TEXTONLY MECH: sp1: sent s Jan 24 19:28:41 sdsshost IDA 313546 TEXTONLY MECH: sp1: received spMechVersion v2_0_1 Bootup 666639483 SpectroID 1 SlitID 35 Air On Shutter_open_sensor Off Shutter_... Jan 24 19:28:41 sdsshost IDA 313546 TEXTONLY ...closed_sensor On Left_open_sensor Off Left_closed_sensor On Right_open_sensor Off Right_closed_sensor On Coll_motor_A -4... Jan 24 19:28:41 sdsshost IDA 313546 TEXTONLY ...163 Coll_motor_B -4163 Coll_motor_C -4163 Requested_exp.time 0 Exp_time_left 0 Last_exp.time 3 sop/2372: make scaleChange in SOPGUI work for values > |.01%|. --------------------------------------------------------------- Tested to the extent possible without being on sky. It does indeed accept arguments larger than .01%, and does wait between steps to implement each change. Thanks to Scot who stayed late tweaking things to get the code to interpret the TCC output satisfactorily. A word of caution: The trigger for scaleChange to know the change in scale is complete is a change in the TCC parameter scaleFac. This value is broadcast at the completion of the SET SCALE command. However if a SHOW SCALE is issued at anytime after SET SCALE is issued and before it finishes, scaleFac will broadcast the destination value, and scaleChange will say it finished before it actually did. We will wait until we are on sky and test it while guiding before signing off on it. It will also allow us to experiment and find the magic maximum value of scale change from which the guider can recover. iop/2919: Loading saved nightlog into logTool causes " and others to show up in the log... --------------------------------------------------------------------------- Special characters are now preserved when night log is reloaded. PR closed. iop/2927: A proposal to allow an easy way to insert timestamps in logTool. ------------------------------------------------------------------------- One can now insert the time in the ISO short form with the key sequence ctrl-shift-t. The long form with date and time can be added with the sequence ctrl-shift-d. Note that this is slightly different than described in the PR fix where the sequence was alt-t and alt-d respectively. iop/2872: Can logTool's imaging summary include start/stop times? ------------------------------------------------------------------ i4l now includes start and stop times for each run. PR closed. iop/2859: Could exiting out of IOP be made to treat a running logTool a little nicer? --------------------------------------------------------------------------- Quitting an IOP which is running a logTool, will complain that this logTool is running and not exit. It does provide an option to force the quit if it is necessary. PR closed. iop/2738: logTool should not changed entered text. -------------------------------------------------- Double quotes now remain double quotes. PR closed. sop/2975: Offsets I section should include axis position. ---------------------------------------------------------- It does now. PR closed. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 2900 0015Z 0029Z 100 O -166.85 -166.85 36 ignore 2901 0032Z 0052Z 100 O -166.85 -166.85 47 ignore 2902 0630Z 0637Z 100 O -68.73 -67.12 25 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot --------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields nGood rowMean rowSig colMean colSig rot --------------------------------------------------------------- ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 12869, plate -9999 ------- 00:51 12869 0.0 bias 00:59 12870 0.0 bias 01:13 12873 0.0 bias 01:19 12874 0.0 bias 01:24 12875 0.0 bias 01:30 12876 0.0 bias 01:35 12877 0.0 bias 01:38 12878 0.0 bias 01:42 12880 0.0 bias 01:46 12882 3.0 bias 01:50 12884 0.0 bias ----- sequence 12885, plate -9999 ------- 02:16 12885 0.0 bias 02:23 12886 0.0 bias 02:43 12888 0.0 bias 02:50 12890 0.0 bias 04:42 12891 0.0 bias ----- sequence 12892, plate -9999 ------- 06:36 12892 0.0 bias 06:42 12893 0.0 bias 06:47 12894 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument getclock Az Alt Rot Scale ------------------------------------------------------------------------------ 0641Z 3 0 121.00 0.0010 30.01 -0.0020 0.00 0.0056 1.000000 0649Z imager 121.00 0.0010 30.01 -0.0020 0.00 0.0056 1.000000 0650Z 3 0 121.00 0.0010 30.01 -0.0020 0.00 0.0056 1.000000 0652Z 3 0 121.00 0.0010 30.01 -0.0020 0.00 0.0056 1.000000 0653Z 3 0 121.00 0.0010 30.01 -0.0020 0.00 0.0056 1.000000 0653Z 3 0 121.00 0.0010 30.01 -0.0020 0.00 0.0056 1.000000 Note that these were all tests of the formatting. Nothing real. ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: none Stays: none ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 00:23 21 20 3 46 (NE) 547 - Partly Cloudy 00:55 21 18 0 14 (NNE) 312 - Mostly Clear 01:27 19 16 7 12 (NNE) 248 - Clear 01:59 18 15 4 20 (NNE) 210 - Clear 02:31 19 13 7 34 (NE) 205 - Clear 03:07 21 11 8 40 (NE) 259 - Clear 03:59 21 9 1 331 (NNW) 209 - Clear 04:32 24 -5 12 57 (ENE) 157 - Clear 05:04 20 3 11 14 (NNE) 223 - Clear 05:36 22 -1 11 41 (NE) 164 - Clear 06:13 23 -10 9 62 (ENE) 172 - Clear 06:49 25 -14 7 84 (E) 196 - Clear 07:23 26 -16 5 88 (E) 178 - Clear 07:55 26 -16 2 198 (SSW) 189 - Clear 08:27 25 -15 1 166 (SSE) 181 - Clear 08:59 25 -14 0 173 (S) 167 - Clear 09:32 27 -17 0 327 (NNW) 176 - Clear 10:04 25 -15 5 338 (NNW) 135 - Clear 10:36 26 -17 1 271 (W) 184 - Clear 11:08 26 -21 6 353 (NNW) 146 - Clear 11:40 27 -22 9 28 (NNE) 159 - Clear 12:13 28 -24 7 352 (NNW) 83 - Clear ---=== TELESCOPE STATUS ===--- 05:05 West door cracked, Exhaust fan on. 11:25 Enclosure closed, fans off Stowed at 30 deg alt, cartrige 3 installed. Spectro auto-fill connected. Spectrograph dewar 12:30 Dewar wt 277 lb Imager dewar 12:30 Dewar wt 113 lb ---=== SOFTWARE USED ===--- IOP/SOP: v3_66_4 Watcher: v2_13_2a MCP: v5_8_7 TPM: v2_4_5 AstroDa: v14_41 TCC: v2.6 sdssProcedures: v1_38 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0760 0.0750 Axial B 0.8060 0.8020 Axial C 0.0740 0.0710 Trans D -1.4950 -1.4860 Lateral E 1.8880 1.8540 Lateral F 1.4440 1.4240 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: -5.5 Alt.: 29.999996 MIGS TONIGHT NOMINAL Axial A 1.5470 1.5500 Axial B 1.2300 1.2260 Axial C 1.2450 1.2480 Trans D 0.2200 0.2320 GALILS Commanded: 1599680. 1565318. 1586370. -10400. -9750. Actual: 1599600. 1565250. 1586350. -10400. -9750. SETMIR VALUES SecDesOrient: 1257.00 -3.40 -10.00 0.00 133.82 SecOrient: 1257.04 -3.39 -10.00 -0.24 133.69 ---=== PROBLEMS IN DETAIL ===--- MCP v5_9_1 and v5_9_0 bug: -------------------------- MCP did not come up after a switch to v5_9_1. The soft reboot did not complete. An mcpResetCrate was issued and the MCP did not reboot. We were logged into the console port and saw no activity. The PLC also became unavailable. The odd thing is that an mcpResetCrate didn't work either! I went down to the enclosure and reset the crate from there. The MCP came back up and the PLC did as well. This lasted about 5 minutes, then both became unavailable again, and the console port of the MCP became unresponsive again. This time a ctrl-x was done from the menu and the hard reboot option was selected. This worked. We tested mcpResetCrate a second time, and this time it worked. After about 20 minutes it became unavailable again. In the console port the following messages appeared. Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xfb990c "tNetTask" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xbb56cc "tmRot" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xb96a80 "ampMgt" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xbbf654 "tmAz" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : v5_9_0: mcp-> at end DRIFT azimuth: 121.001729 0.000000 14132.184294 at end DRIFT altitude: 30.004776 0.000000 14132.321788 at end DRIFT rotator: 0.001125 0.000000 14132.478030 START axis=0: time=2.180030 START axis=2: time=2.076903 START axis=4: time=1.870436 frame=0xf8eea4, nxt=0xf8f30c, nxt=0x0, frame_cnt=13 frame=0xf8f448, nxt=0xf8f2ac, nxt=0x0, frame_cnt=13 frame=0xf8f0b0, nxt=0xf8f050, nxt=0x0, frame_cnt=6 START axis=0: time=8.548356 START axis=2: time=8.376556 START axis=4: time=8.191431 Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xfb990c "tNetTask" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xbb7c48 "tmRot" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xb99c50 "ampMgt" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xb9751c "tUmbilical" Bus Error Program Counter: 0x0000010c Status Register: 0x3018 Access Address : 0x0db60006 Special Status : 0x0505 Task: 0xb958f4 "TCC" Decided to revert to v5_8_7. No bus errors after that.