APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday October 17, 2001 (MJD 52200) ---=== OBSERVING TEAM ===--- Swing: Scot Kleinman (APO) Night: Jurek Krzesinski, Pete Newman (APO) Support: Craig Loomis (APO), French Leger (FNAL), Jon Brinkman (APO) ---=== OBSERVING PLAN ===--- Science. Imaging if conditions allow, then spectroscopy. ---=== OBSERVING SUMMARY ===--- Variable cloud all night. Three plates done (640, 680, 705), usable data on two others (649, 744). Many old problems revisited, a couple of new PRs filed. The SoS slowdown cured. We lack plates for both start and end of night! Hope the recent imaging helps for the next drilling run. ---=== OBSERVING LOG ===--- Start of the nigh hassles: Jurek could not run logTool, getting caget errors from IOP. See problem section #1 for details. Also, Pete could not log in to telrun from any session. See problem section #2 for details. Together with these start-up problems, there was a small dust spike around sunset, so we opened at 30 minutes after sunset, but on the sky by the end of twilight. Fiducial problems: See problems section #3 for details. Summary: still have problems (missed fiducials, repeated wild errors), but not enough to stop us observing. French confirmed the problem is downstream from the MEI input, as all interrupt signals appear there. SoS sluggish (PR2805): We solved this problem. See problems section #5 for details. The full-step error continues on the secondary galil axis E. Pete's logViewer hung with a blank screen when switching to the Unexpected section (how apropos!) that also hung its starting IOP. Restarting iop then logViewer solved the problem. Not clear what happened, so no PR filed. endNight rejoiced greatly. Observing sequence: Variable cirrus before sunset, so start with spectroscopy. Annoying orographic clouds throughout. The satellite images showed the clouds densest right over us - renew those petitions to the weather gods, please! 19:25 Plate 640, cartridge 2. Done in 3 exposures, but postCals requested after we had finished, so not taken because SoS is being slow. Stars found on first gotoField. Good seeing. 21:02 Plate 649, cartridge 1. Not done in 3 exposures, but smear taken ( although in cloud, so another smear under good sky is advisable). Clouds increasing and seeing much worse as a front moves across from the east. gotoField did not find guide stars, but the fk5InFiber found its star about 8 arcsec south of the pointing (see notes above on fiducial errors). We stopped observing when we had gone far beyond the design time window for the plate. 22:43 Plate 680, cartridge 9. Done in 3 exposures. Stars found in the gotoField. Clouds too thick to observe for ~80 minutes, so final exposure after design time, but S/N good. 01:56 Plate 705, cartridge 4. Done in 3 exposures. Clouds clearing and seeing improving, then thick again during the second exposure, then varying a lot. S/N slightly over the minimum because of guess work in dodging clouds. Plate 544, cartridge 7. Not done in 1 exposure, and not smeared. We had problems that started with the guide fibers being off the guider image. See problem #4 for details. The first exposure did not achieve (S/N)^2 > 2 in r2 due to clouds, but other cameras ok. The second exposure was aborted due to clouds. ---=== IMAGING RUN SUMMARY ===--- Run Stripe Flavor Lambda Last Begin End Frame Comments ------------------------------------------------------------------ 2667 100 O ignore 77.39 89.26 91 doghouse bias ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot --------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields nGood rowMean rowSig colMean colSig rot --------------------------------------------------------------- ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 11258, plate -9999 ------- 00:39 11258 0.0 bias 01:06 11259 0.0 bias ----- sequence 11260, plate 640 ------- 15.6 16.8 20.0 20.2 DONE 01:44 11260 10.0 flat 01:46 11261 2.0 arc 02:13 11262 901.0 target 02:31 11263 900.0 target 02:49 11264 901.0 target 02:56 11265 240.0 smear ----- sequence 11266, plate 649 ------- 13.7 8.0 12.5 12.3 NOT DONE 03:20 11266 10.0 flat 03:22 11267 2.0 arc 03:42 11268 900.0 target 03:59 11269 901.0 target 04:33 11270 1802.0 target 04:40 11271 240.0 smear ----- sequence 11272, plate 680 ------- 25.2 14.9 27.0 17.2 DONE 05:04 11272 10.0 flat 05:06 11273 2.0 arc 06:41 11274 901.0 target 07:09 11275 1502.0 target 07:39 11276 1501.0 target 07:45 11277 240.0 smear 07:48 11278 10.0 flat 07:50 11279 2.0 arc ----- sequence 11280, plate 705 ------- 23.5 19.4 20.3 16.3 DONE 08:05 11280 10.0 flat 08:07 11281 2.0 arc 08:28 11282 901.0 target 09:44 11283 1800.0 target 10:00 11284 722.0 target 10:06 11285 240.0 smear 10:09 11286 10.0 flat 10:11 11287 2.0 arc ----- sequence 11288, plate 544 ------- 2.7 3.3 2.9 1.9 NOT DONE 11:00 11288 10.0 flat 11:02 11289 2.0 arc 11:31 11290 1201.0 target smear needed! ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument getclock Az Alt Rot Scale ------------------------------------------------------------------------------ 19:52 2 640 1003369923 0.001355 0.001016 0.002812 0.999800 21:35 1 649 1003376130 -0.001353 -0.001635 0.000000 0.999840 00:28 9 680 1003386528 -0.001353 -0.001635 0.000000 1.000110 02:12 4 705 1003392777 0.002306 0.001380 0.001073 1.000050 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL4553 Stays: JL4554 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 19:52 2 640 0.99980 1974 2821 0 18 48.3 11.2 10 249 -- 1.5'' 21:34 1 649 0.99984 1579 2444 -75 32 41.6 11.9 11 229 -- 2.5'' 22:32 1 649 0.99978 2171 2901 -90 13 46.3 11.5 9 261 -- 1.8'' 00:28 9 680 1.00011 -1085 87 -200 -44 48.3 12.0 14 285 -- 1.7'' 02:12 4 705 1.00005 -493 652 -100 -12 56.7 11.1 16 271 -- 1.6'' ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 17:25 57 24 10 264 (W) 2010 - cirrus 18:47 52 26 11 262 (W) 2285 - '' 19:19 52 27 12 237 (WSW) 1768 - '' 19:50 51 27 13 237 (WSW) 1766 - '' 20:23 52 27 11 238 (WSW) 1656 - '' 20:56 52 27 9 229 (SW) 1515 - '' 21:27 53 26 12 238 (WSW) 1187 - '' 22:00 52 27 10 225 (SW) 1433 - Thick 22:32 52 26 9 261 (W) 1300 - '' 23:04 52 25 10 264 (W) 1211 - '' 23:36 53 23 11 269 (W) 1219 - '' 00:09 54 19 12 279 (W) 1104 - Variable 00:41 53 21 13 291 (WNW) 1227 - '' 01:14 53 21 16 283 (WNW) 1193 - '' 01:46 52 21 15 284 (WNW) 1354 - '' 02:16 51 23 11 266 (W) 1206 - '' 02:49 52 23 12 275 (W) 1338 - Thick 03:22 51 25 15 273 (W) 1308 - '' 03:54 51 26 16 264 (W) 1452 - Variable 04:27 51 26 10 265 (W) 1335 - '' 04:59 49 25 13 238 (WSW) 1137 - '' 05:32 49 24 14 246 (WSW) 823 - Thick 06:08 48 24 12 260 (W) 892 - '' 06:44 50 24 15 268 (W) 980 - '' 07:16 50 24 13 271 (W) 1152 - ---=== TELESCOPE STATUS ===--- 17:00 Doors open, fans on 18:50 Enclosure off 06:15 Enclosure on, fans off Instruments on autofill. 180L dewar weights at 07:10 Spectro: 115 lb Imager: 63 lb (filling ---=== SOFTWARE USED ===--- IOP/SOP: v3_58_2 Watcher: v2_11_0 MCP: v5_6_3 TPM: v2_4_2 AstroDa: v14_39 TCC: 2.5.5 sdssProcedures: v1_31 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0770 0.0710 Axial B 0.7990 0.8030 Axial C 0.0660 0.0640 Trans D -1.4940 -1.4590 Lateral E 1.8640 1.8600 Lateral F 1.4230 1.4280 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 11.3 Alt.: 30.046142 MIGS TONIGHT NOMINAL Axial A 1.5680 1.5670 Axial B 1.2880 1.2900 Axial C 1.3220 1.3200 Trans D 0.2240 0.2310 GALILS Commanded: 1599605. 1565312. 1586345. -10400. -9750. Actual: 1599600. 1565250. 1586350. -10400. -9750. SETMIR VALUES SecDesOrient: 1257.00 -3.40 -10.00 0.00 133.82 SecOrient: 1257.01 -3.39 -9.98 -0.23 133.69 ---=== PROBLEMS IN DETAIL ===--- Problem #1 - logTool failed to start When Jurek tried to start logTool from IOP, he got a window saying 'Invalid command name .logtool.logviewer4.displayframe.yscroll' (I think, I only read it from the screen). He then got the following stack trace: Stack Trace invalid command name ''.logtool.logviewer4.displayframe.xscroll'' while executing ''.logtool.logviewer4.displayframe.xscroll set 0 0.00462963'' (horizontal scrolling command executed by text) im> logTool 730179:caget: rld: Fatal Error: Cannot Successfully map soname 'libezcaScan.so' under any of the filenames /lib/libezcaScan.so:/usr/lib//libezcaScan.so:/usr/ope nwin/lib/libezcaScan.so:/usr/lib32/libezcaScan.so:/usr/lib32/internal/libezcaSca n.so:/lib32/libezcaScan.so:/opt/lib32/libezcaScan.so: Error: child process exited abnormally After which he got the buttons (top) section of logTool window open, but not the lower text part. This was repeatable. Pete, on host via sdss-commish, had no problem. This seems to be related to PRs 2684 and 2711 (we updated 2711, but that is the duplicate). ------------------------------------- Problem #2 - TCC was not accepting new users I could not get any TCC sessions to give me TCC message traffic. With guidance from Craig Loomis, we did telstop and telrun, which cleared the messages but meant that all our IOP, SOP and TCCMON sessions had to be restarted. This delayed opening by at least 20 minutes. Note that issuiung a 'show users' in Jurek's working TCC session did not resolve the problem.. Craig prodcued the following diagnostic messages during investigation. Otherwise, see the tcc25m log under apotop@galileo. Script started on Wed Oct 17 18:15:42 2001 [prn@sdss-commish ~]$ telnet tcc25m^M Trying 192.41.211.156...^M Connected to tcc002.apo.nmsu.edu.^M Escape character is '^]'.^M Welcome to OpenVMS (TM) Alpha Operating System, Version V7.2-1 ^MUsername: tcc ^MPassword: Welcome to OpenVMS (TM) Alpha Operating System, Version V7.2-1 Last interactive login on Wednesday, 17-OCT-2001 23:36:51.29 Last non-interactive login on Tuesday, 16-OCT-2001 17:53:31.87 Your process name is _TNA1520:^MESC[?3hESC> ^@$ show system ^MOpenVMS V7.2-1 on node TCC002 18-OCT-2001 00:16:35.07 Uptime 132 01:44:44 Pid Process Name State Pri I/O CPU Page flts Pages 00000081 SWAPPER HIB 16 0 0 00:00:07.30 0 0 00000084 LANACP HIBO 13 -- swapped out -- 28 00000086 IPCACP HIB 10 9 0 00:00:10.42 591 22 00000087 ERRFMT HIB 8 361430 0 00:18:33.24 244 62 00000089 OPCOM HIB 9 17043 0 00:00:35.52 1881 41 0000008A AUDIT_SERVER HIB 10 539 0 00:00:02.17 311 75 0000008B JOB_CONTROL HIB 10 8184 0 00:00:16.72 520 42 0000008C QUEUE_MANAGER HIBO 9 -- swapped out -- 29 0000008D SECURITY_SERVER HIB 10 3292 0 01:07:57.52 12081 170 0000008E TP_SERVER HIB 10 760302 0 02:16:31.84 1625 51 0000008F TCPIP$INET_ACP HIB 10 8090 0 00:00:11.55 473 136 00000090 HDRAPER HIBO 4 -- swapped out -- 25 00000091 TCPIP$NTP LEF 10156377470 1 01:22:02.49 7913 118 N 00000092 LATACP HIB 14 8 0 00:15:12.55 905 34 0000119A TCC LEF 5 7085 0 00:00:11.13 500 219 0000101B T_MSGHUB LEF 4 5073953 0 00:31:52.86 767 82 0000119C TCCUSER LEF 5 6028 0 00:00:09.45 474 265 0000169D _TNA1508: LEF 4 5126 0 00:00:07.45 393 308 0000119E _TNA1509: LEF 5 5040 0 00:00:07.63 423 243 0000149F _TNA1510: LEF 5 179 0 00:00:01.40 468 257 00000A20 T_SLOW HIB 6 9539 0 00:00:33.53 363 119 00001621 _TNA1511: LEF 4 141 0 00:00:01.07 398 215 00000A22 T_MOCKCTRL LEF 6 104 0 00:00:00.94 213 63 00000BA4 T_COMPAPPGEO HIB 6 103 0 00:01:43.18 251 49 00001525 _TNA1513: LEF 4 610 0 00:00:02.91 1067 77 00000D26 _TNA1514: LEF 5 144 0 00:00:01.01 396 189 00001427 _TNA1515: LEF 4 141 0 00:00:00.98 439 183 000009A8 _TNA1516: LEF 4 141 0 00:00:01.02 442 183 00000F29 T_GUIDE LEF 6 102 0 00:00:01.02 263 23 000011AB T_GCAMUSER LEF 6 34839 0 00:00:35.51 439 83 00000F2C _TNA1518: LEF 6 211 0 00:00:02.09 435 260 0000122D T_HUBMONITOR LEF 6 1337 0 00:00:01.97 223 66 0000102F T_TRACK HIB 9 578530 0 00:22:37.04 935 148 00000FB0 _TNA1520: CUR 4 150 0 00:00:00.88 277 149 00001031 T_COLLIMATE HIB 6 204837 0 01:09:51.67 360 100 00001033 T_AUTOLOUVER HIB 6 102 0 00:00:01.74 261 38 00000D35 T_AXEDONE HIB 6 5047 0 00:00:04.03 218 39 00000FB7 T_JOBQUEUE LEF 6 2491 0 00:00:04.02 331 209 00001039 T_BRDTELPOS HIB 6 2161826 0 01:29:01.71 1029 69 0000103B T_STATUS HIB 5 680939 0 00:21:18.16 582 96 00000F40 T_PTERR LEF 6 18267 0 00:03:00.25 697 510 00001647 _TNA1493: LEF 5 212147 0 00:04:33.95 685 347 ^M$ tel;^M$ telrun ^M0 8 I YourUserNum=8 show uset^Hrs exit ^M$ ^M$ show proc/all./ful^H^H^H^H^H/full ^M%DCL-W-IVQUAL, unrecognized qualifier - check validity, spelling, and placement FULL ^M$ ^B^MESC[K^@^@^@$ show proc/all/ful^H^H^H^H 18-OCT-2001 00:18:27.23 User: TCC Process ID: 00000FB0 Node: TCC002 Process name: ''_TNA1520:'' Terminal: TNA1520: (Host: 192.41.211.168 Port: 4668) User Identifier: [TCC] Base priority: 4 Default file spec: SYS$SYSDEVICE:[TCC] Number of Kthreads: 1 Devices allocated: TNA1520: Process Quotas: Account name: CPU limit: Infinite Direct I/O limit: 150 Buffered I/O byte count quota: 63808 Buffered I/O limit: 150 Timer queue entry quota: 10 Open file quota: 100 Paging file quota: 47040 Subprocess quota: 8 Default page fault cluster: 64 AST quota: 248 Enqueue quota: 2000 Shared file limit: 0 Max detached processes: 0 Max active jobs: 0 Accounting information: Buffered I/O count: 180 Peak working set size: 4048 Direct I/O count: 34 Peak virtual size: 178448 Page faults: 511 Mounted volumes: 0 Images activated: 4 Elapsed CPU time: 0 00:00:01.37 Connect time: 0 00:02:01.29 Authorized privileges: GRPNAM GRPPRV LOG_IO NETMBX OPER PHY_IO SHARE TMPMBX Process privileges: GRPNAM may insert in group logical name table GRPPRV may access group objects via system protection LOG_IO may do logical i/o NETMBX may create network device OPER may perform operator functions SHARE may assign channels to non-shared devices TMPMBX may create temporary mailbox Process rights: TCC resource INTERACTIVE REMOTE System rights: SYS$NODE_TCC002 Auto-unshelve: on Image Dump: off Soft Affinity: off Parse Style: Traditional Process Dynamic Memory Area Current Size (bytes) 131072 Current Size (pagelets) 256 Free Space (bytes) 108524 Space in Use (bytes) 22548 Largest Variable Block 108204 Smallest Variable Block 8 Number of Free Blocks 6 Free Blocks LEQU 64 Bytes 4 There is 1 process in this job: _TNA1520: (*) ^M$ show mem ^M System Memory Resources on 18-OCT-2001 00:18:56.33 Physical Memory Usage (pages): Total Free In Use Modified Main Memory (64.00Mb) 8192 843 6796 553 Virtual I/O Cache (Kbytes): Total Free In Use Cache Memory 3200 0 3200 Granularity Hint Regions (pages): Total Free In Use Released Execlet code region 512 0 509 3 Execlet data region 192 0 185 7 S0/S1 Executive data region 254 0 254 0 Resident image code region 512 0 359 153 Slot Usage (slots): Total Free Resident Swapped Process Entry Slots 75 32 40 3 Balance Set Slots 73 34 38 1 Dynamic Memory Usage (bytes): Total Free In Use Largest Nonpaged Dynamic Memory 2031616 929728 1101888 595456 Paged Dynamic Memory 1499136 951424 547712 945744 Lock Manager Dynamic Memory 507904 303360 204544 Buffer Object Usage (pages): In Use Peak 32-bit System Space Windows (S0/S1) 0 0 64-bit System Space Windows (S2) 0 0 Physical pages locked by buffer objects 0 0 Memory Reservations (pages): Group Reserved In Use Type Total (0.00 Mbytes reserved) 0 0 Paging File Usage (blocks): Free Reservable Total DISK$AVMS_721:[SYS0.SYSEXE]SWAPFILE.SYS 4992 4992 8064 DISK$AVMS_721:[SYS0.SYSEXE]PAGEFILE.SYS 101952 -3056 139264 Of the physical pages in use, 2324 pages are permanently allocated to OpenVMS. ^M$ show zone ^M%DCL-W-ACTIMAGE, error activating image FTSS$UTIL -CLI-E-IMAGEFNF, image file not found TCC002$DKA300:[SYS0.SYSCOMMON.][SYSEXE] FTSS$ UTIL.EXE; ^M$ telrun^M$ ^M$ telru^H^Hsto^H^H^H^H^H^Hshow mem/full ^M System Memory Resources on 18-OCT-2001 00:21:13.83 Physical Memory Usage (pages): Total Free In Use Modified Main Memory (64.00Mb) 8192 1273 6423 496 Virtual I/O Cache (Kbytes): Total Free In Use Cache Memory 3200 0 3200 Granularity Hint Regions (pages): Total Free In Use Released Execlet code region 512 0 509 3 Execlet data region 192 0 185 7 S0/S1 Executive data region 254 0 254 0 Resident image code region 512 0 359 153 Slot Usage (slots): Total Free Resident Swapped Process Entry Slots 75 34 38 3 Balance Set Slots 73 36 36 1 Dynamic Memory Usage (bytes): Total Free In Use Largest Nonpaged Dynamic Memory 2031616 946752 1084864 595456 Paged Dynamic Memory 1499136 952512 546624 945744 Lock Manager Dynamic Memory 507904 306176 201728 Buffer Object Usage (pages): In Use Peak 32-bit System Space Windows (S0/S1) 0 0 64-bit System Space Windows (S2) 0 0 Physical pages locked by buffer objects 0 0 Memory Reservations (pages): Group Reserved In Use Type Total (0.00 Mbytes reserved) 0 0 Paging File Usage (blocks): Free Reservable Total DISK$AVMS_721:[SYS0.SYSEXE]SWAPFILE.SYS 4992 4992 8064 DISK$AVMS_721:[SYS0.SYSEXE]PAGEFILE.SYS 101952 3968 139264 Of the physical pages in use, 2324 pages are permanently allocated to OpenVMS. ^M$ show log tcc* (LNM$PROCESS_TABLE) ''TCC_EARTH_DATA'' = ''TDAT:EARTHPRED.DAT'' ''TCC_POSREF_CAT'' = ''TCAT:POSREF7_9.DAT'' (LNM$JOB_80A80C40) (LNM$GROUP_000100) ''TCC_GCAM'' = ''NLA0:'' ''TCC_GMECH'' = ''NLA0:'' ''TCC_ROT'' = ''NLA0:'' (LNM$SYSTEM_TABLE) ''TCC_ALT'' = ''LTA19:'' ''TCC_AZ'' = ''LTA19:'' ''TCC_CLOCK'' = ''LTA5:'' ''TCC_ENCL'' = ''LTA6:'' ''TCC_ENGCAM'' = ''LTA16:'' ''TCC_GCAM1'' = ''LTA11:'' ''TCC_GMECH'' = ''LTA6:'' ''TCC_MOCKCTRL'' = ''LTA7:'' ''TCC_PRIM'' = ''LTA9:'' ''TCC_ROT1'' = ''LTA19:'' ''TCC_SEC'' = ''LTA10:'' ''TCC_TEL1'' = ''TCC_AZ'' ''TCC_TEL2'' = ''TCC_ALT'' ''TCC_TERT'' = ''NLA0:'' (LNM$SYSCLUSTER_TABLE) ^M$ eoj ^M TCC logged out at 18-OCT-2001 00:27:32.17^MConnection closed by forei gn host.^M [prn@sdss-commish ~]$ ^^Hexit Script done on Wed Oct 17 18:27:02 2001 ----------------------------------------------- Problem # 3 - altitude fiducial errors During opening around 18:50, the altitude fiducial at 28.91 deg gave an error of 3608586 counts after giving good corrections eralier. Slewing with the TCC across the fiducial around 15deg alt gave a good correction, after which the 28.91 deg fiducial worked ok. We saw a missed 30 deg az fiducial around 20:58 during a slew to instrument change. We also saw the az position error climb above 600 between the 90 deg and 105 deg fiducials during the same slew. The position error then went to -60882 at 21:02 after the instrument change, then back to 858 on the 120.76 fiducial. We missed then missed the 90 deg and 45 deg fiducials on the slew to the field (at az ~ 29.25 deg). However, French checked the fiducial interrupts going through the MEI board during these slews and saw all of them, so suspicion now falls on the DIO316 or MCP main boards. Is it possible the MCP board is hanging? That might explain both missed fiducial interrupts AND the TCC echo timing problem. The position error did not go below 600 again during that first plate, but as we acquired guide stars with an 8 arcsec offset, we continued. We reset the fiducial error using mcpMenu during the next cartridge change. During the slew to plate 680's field, we again saw the 90 deg and 45 deg fiducials missed in the murmur log, but the posistion error stayed within bounds, so we continued observing. We later saw more very large errors on az and alt (around 01:50--02:00) and further missed az fiducials (-45, +45, +90) without halting operations. Some typical and relevant watcher and murmur log error messages: altitude (correction for altitude -3608613 is too large (Oct 17 18:50:25): Oct 17 18:50:32 MDT ) Not disabling MS.ON; error -3608613 (max allowed: 600) (Not disabling MS.ON; error -3608613 (max allowed: 600) (Oct 17 18:50:25): Oct 17 18:50:32 MDT ) Not sure if these are related, but they mention the DIO 316 card which we know is in the same hardware chain: Oct 17 18:50:03 sdssmcp tIdleTask dsc_E_trcErr NIST_INT bit is set in dio316_interrupt: 0x5 Oct 17 18:54:51 sdssmcp tIdleTask dsc_E_trcErr NIST_INT bit is set in dio316_interrupt: 0x5 Oct 17 19:15:30 sdssmcp barcodcan dsc_E_trcErr NIST_INT bit is set in dio316_interrupt: 0x5 Oct 17 21:00:14 sdssmcp tIdleTask dsc_E_trcErr NIST_INT bit is set in dio316_interrupt: 0xf Oct 17 22:46:45 sdssmcp tIdleTask dsc_E_trcErr NIST_INT bit is set in dio316_interrupt: 0x3 Later, we see very large errors again (from the tLatch section of logViewer): 23:46:52 dsc_I_trcInfo az fiducial -29.38 deg 23:46:52 dsc_I_trcInfo err = ??? poserr = -3859990 ticks 00:30:29 dsc_I_trcInfo az fiducial -44.36 deg 00:30:29 dsc_I_trcInfo err = ??? poserr = -3845916 ticks 01:28:29 dsc_I_trcInfo az fiducial -59.38 deg 01:28:29 dsc_I_trcInfo err = ??? poserr = -3859242 ticks but these cleared during the slew to instrument change. We _think_ there may have been small jumps in the telescope position during spectro guiding, as the guide stars shifted around on withing the guide fibers on two occasions we noticed (during the third exposure of plate 680). Sorry, not timed exactly. ----------------------------------- Problem #4 - startGuider failed, UDP port conflict While setting up on plate 544, cartridge 7, we had problems sthat started with the guide fibers being off the guider image. This happened with both the glat from the gotoField, then after the preCalibs flat was taken. preCalibs hung. Ctrl-C ended the preCalibs, then startDa -tcc again. We saw a message from the preCalibs saying the image size was 384 x 256 (should be 768 x 512), which was curious. After getting out of preCalibs, we did clearOffsets then another gotoField -gflat. This time, startGuider ran, but did not issue offsets (see PR 2802, filed yesterday). We centered the stars by hand, requiring a rotator offset of -150 arcsec plus small offsets in RA and Dec. startGuider then ran as normal, but in the process we restarted the fsaoImage process (stopMon, startMon, showGuiderFrame). Five minutes later, SOP lost the UDP port. We followed the folklore prescription for repair, which worked. So that seems to confirm that running startMon (and/or showGuiderFrame) after grabInst triggers the UDP port problem. PR 2217 updated. ----------------------------------- Problem #5 - SoS slow to reduce R frames (PR 2805) We solved this problem which we again experienced tonight, and implenented a fix, finding a bug in the aporsync.sh job that copies spectro frames from sdsshost to sos. The fix was applied to the live version. The observers do not have check-in priviledges to idlspec2d, so the CVS version still needs fixing. The sporadic nature of the failure seems to have been a synchronicity problem associated with the exposure time of guider. PR set to needtest, and David Schlegel asked to update the CVS version.