APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday January 29, 2003 (MJD 52669) ---=== OBSERVING TEAM ===--- Swing: Howard Brewington Night: Dan Long, Pete Newman Support: Craig Loomis, Atsuko Nitta (both by phone) ---=== OBSERVING PLAN ===--- Science. ---=== OBSERVING SUMMARY ===--- 5,760 spectra obtained, finished all 9 spectrographic plates by 2am and went to bed early. (Just kidding, in reality it took until 5:45am.) Several problems from which we recovered with little loss of time (but who cares? We finished 9 plates!). ---=== OBSERVING LOG ===--- Version v3_105_0 of IOP has been declared current. The spectrograph collimation and focus was checked, and the sp1 focus adjusted. We discovered a problem with the spectrographs' LN2 auto-fill system just after opening this evening. See problem section #1 for details. During fiducial checks (I think - in the enclosure anyway), watcher reported: altitude (Windscreen touched in altitude: UP (2003-01-29 23:54:06Z sdssmcp): 2003-01-29 23:54:19Z ) Conditions at sunset were photometric with light NW wind, but we started with spectroscopy to try to finish plate 1130 during twilight whose field is sinking in the west. As it turned out, the seeing was a bit variable and occasional clouds arrived from the NW soon after, so we continued with spectroscopy. Observing sequence: 01:11Z Plate 1130, cartridge 1. Munn program. Done in 1 one exposure when added to MJDs 52642 and 52647. See problem section #2. 02:08Z Plate 811, cartridge 3. Loveday program. Done in 4 exposures when added to MJD 52650. Dan's SOP session hung up after the instChange from the last plate. See problem section #2. We also had a possibly related spectro PTVME link error that confused the union out of us. See problem section #3. 04:10Z Plate 1203, cartridge 2. Survey. Done in 3 exposures. 05:24Z Plate 1204, cartridge 5. Survey. Done in 3 exposures. 06:35Z Plate 1198, cartridge 6. Survey. Done in 3 exposures. 07:59Z Plate 876, cartridge 8. Survey. Done in 3 exposures. 09:15Z Plate 1159, cartridge 7. Survey. Done in 3 exposures. 10:26Z Plate 1163, cartridge 9. Survey. Done in 3 exposures 11:32Z Plate 848, cartridge 4. Survey. Done in 3 exposures endNight was not the only one rejoicing! ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3638 21:09Z 21:33Z 100 O 155.27 161.35 53 ignore doghouse bias ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 18886, plate -9999 ------- 00:10 18886 0.0 bias ----- sequence 18887, plate 1130 ------- 13.7 10.1 18.1 11.1 MJDs 52645/7 01:26 18887 10.0 flat 4.9 4.0 5.8 4.3 Tonight 01:28 18888 2.0 arc ---- ---- ---- ---- 01:54 18889 900.1 target 18.6 14.1 23.9 15.4 DONE 02:02 18890 240.0 smear ----- sequence 18891, plate 811 ------- 4.3 6.1 4.1 5.3 MJD 52650 02:26 18891 10.0 flat 31.8 30.0 25.2 26.5 Tonight 02:28 18892 2.0 arc ---- ---- ---- ---- 02:47 18893 900.1 target 35.1 36.1 29.3 31.8 DONE 03:04 18894 900.1 target 03:25 18895 900.1 target 03:34 18896 0.0 bias During DA crate recovery 03:36 18898 0.0 bias " " " " 03:55 18899 900.1 target 04:03 18900 240.0 smear ----- sequence 18901, plate 1203 ------- 18.1 19.8 16.6 21.1 DONE 04:22 18901 10.0 flat 04:25 18902 2.0 arc 04:41 18903 800.1 target 04:56 18904 720.1 target 05:11 18905 720.1 target 05:19 18906 240.0 smear ----- sequence 18907, plate 1204 ------- 20.7 21.8 17.4 21.7 DONE 05:35 18907 10.0 flat 05:37 18908 2.0 arc 05:52 18909 720.1 target 06:07 18910 720.1 target 06:22 18911 720.1 target 06:30 18912 240.0 smear ----- sequence 18913, plate 1198 ------- 17.6 18.9 16.3 18.5 DONE 06:47 18913 10.0 flat 06:49 18914 2.0 arc 07:16 18915 720.1 target 07:31 18916 720.1 target 07:46 18917 720.1 target 07:54 18918 240.0 smear ----- sequence 18919, plate 876 ------- 21.2 16.5 17.0 15.8 DONE 08:09 18919 10.0 flat 08:11 18920 2.0 arc 08:29 18921 800.1 target 08:45 18922 800.1 target 09:01 18923 800.1 target 09:10 18924 240.0 smear ----- sequence 18925, plate 1159 ------- 26.0 24.6 25.1 20.1 DONE 09:23 18925 10.0 flat 09:26 18926 2.0 arc 09:42 18927 800.1 target 09:58 18928 720.1 target 10:13 18929 720.1 target 10:21 18930 240.0 smear ----- sequence 18931, plate 1163 ------- 18.5 24.1 19.6 17.5 DONE 10:33 18931 10.0 flat 10:35 18932 2.0 arc 10:51 18933 720.1 target 11:05 18934 720.1 target 11:20 18935 720.1 target 11:28 18936 240.0 smear ----- sequence 18937, plate 848 ------- 17.8 15.8 16.0 20.4 DONE 11:42 18937 10.0 flat 11:44 18938 2.0 arc 11:59 18939 720.1 target 12:18 18940 900.1 target 12:36 18941 900.1 target 12:43 18942 240.0 smear ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 01:29Z 1 1130 -49.92 0.0017 45.00 0.0020 139.89 -0.0022 1.000050 02:40Z 3 811 -26.90 0.0037 54.18 0.0015 157.58 -0.0001 1.000060 Cartridge 2 offsets not recorded. 05:48Z 5 1204 42.62 0.0020 78.27 0.0005 218.45 0.0013 1.000190 06:52Z 6 1198 138.81 -0.0020 80.84 -0.0002 -45.81 -0.0002 1.000120 08:24Z 8 876 154.38 -0.0034 69.79 0.0007 -34.74 -0.0182 1.000120 09:34Z 7 1159 142.83 -0.0034 51.38 0.0007 -69.98 -0.0182 1.000090 10:43Z 9 1163 140.18 -0.0016 55.82 -0.0005 -66.90 -0.0097 1.000090 11:46Z 4 848 -13.05 0.0038 61.45 0.0010 168.94 -0.0044 1.000060 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL5973 Stays: JL5980 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 01:29Z 1 1130 1.00005 -493 739 -50 -50 45.0 3.1 9 2 BG38 1.4 02:39Z 3 811 1.00006 -591 553 -125 -27 54.2 3.1 12 355 BG38 1.7 04:31Z 2 1203 1.00019 -1875 -537 -125 70 64.8 3.2 6 331 BG38 1.3 05:48Z 5 1204 1.00019 -1875 -608 -175 43 78.3 3.2 7 331 BG38 1.3 06:52Z 6 1198 1.00012 -1184 -38 -175 139 80.8 3.3 9 337 BG38 1.4 08:24Z 8 876 1.00012 -1184 -9 -160 154 69.8 2.9 10 283 BG38 1.6 09:34Z 7 1159 1.00009 -888 301 -140 143 51.4 2.5 5 328 BG38 1.8 10:43Z 9 1163 1.00009 -888 268 -160 140 55.8 3.1 9 326 BG38 1.9 11:46Z 4 848 1.00006 -591 528 -130 -13 61.4 1.9 8 345 BG38 1.8 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:27Z 40 15 1 252 (WSW) 264 - Occasional clouds 23:59Z 38 14 1 236 (SW) 263 - " 00:29Z 37 14 0 11 (N) 297 - " 01:00Z 37 14 13 1 (N) 240 - " 01:32Z 37 12 11 15 (NNE) 162 - " 02:03Z 37 11 8 0 (N) 168 - " 02:33Z 37 12 12 357 (NNW) 157 - " 03:04Z 37 12 10 359 (NNW) 139 - " 03:34Z 38 10 10 346 (NNW) 135 - " 04:08Z 37 12 7 6 (N) 150 - " 04:38Z 37 13 5 334 (NNW) 169 - " 05:08Z 37 13 4 1 (N) 177 - " 05:40Z 37 13 4 303 (WNW) 198 - " 06:12Z 37 12 9 305 (NW) 176 - " 06:43Z 37 12 7 337 (NNW) 193 - " 07:14Z 38 9 9 315 (NW) 157 - " 07:45Z 37 12 11 294 (WNW) 165 - " 08:16Z 37 12 9 292 (WNW) 171 - " 08:47Z 36 12 8 291 (WNW) 220 - " 09:17Z 37 12 9 284 (WNW) 244 - " 09:52Z 36 12 11 309 (NW) 236 - " 10:22Z 36 12 8 340 (NNW) 303 - " 10:52Z 37 11 8 336 (NNW) 230 - " 11:23Z 36 11 7 344 (NNW) 240 - " 11:53Z 34 11 10 329 (NNW) 267 - " 12:26Z 36 12 9 324 (NW) 245 - " 13:03Z 37 12 7 342 (NNW) 226 - " ---=== TELESCOPE STATUS ===--- 23:25Z Fans on, doors open 01:45Z Enclosure off 12:50Z Enclosure on, fans off Status at 13:02Z: Telescope stowed at: 30 deg Instrument mounted: Cartridge 4 (which needs re-plugging, too) Counterweights at: 275 Autofill systems: On 180L LN2 dewar weights: Spectro 173 lb (filling) Imager 190 lb Interlocks bypassed: None ---=== SOFTWARE USED ===--- IOP/SOP: v3_105_0 (declared current tonight) Watcher: v2_20_0 MCP: v5_17_0 TPM: tpm_v2_24_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_62 SoS: v4_9_13 hoggPT: v1_5_8 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0810 0.0790 Axial B 0.7970 0.7970 Axial C 0.8180 0.8240 Trans D -9.0800 -9.1130 Lateral E 1.8540 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 3.7 Alt.: 30.000004 MIGS TONIGHT NOMINAL Axial A 1.5170 1.5170 Axial B 1.1560 1.1560 Axial C 1.1000 1.0990 Trans D -0.6980 -0.7070 GALILS Commanded: 1619630. 1555202. 1576247. -3400. -7900. Actual: 1619650. 1555250. 1576350. -3400. -7900. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1256.96 0.01 -20.02 -0.49 130.87 ---=== PROBLEMS IN DETAIL ===--- Problem #1 - spectrographs's LN2 autofill system failed ------------------------------------------------------- The spectrographs autofilled sometime shortly before opening tonight. We removed the enclosure at 01:45Z, and there was still some frost on the LN2 lines. On moving the telescope to alt=6 deg to remove the enclosure, we saw the usual LN2 spillage from the vents on both secondary dewars. The rotator was at 30 deg, so the sp1 and sp2 dewars were level. We then tried our standard forced fill after opening. The fill lines were hooked up, and the "Start" buttons pressed on the autofillwere pressed, but no fill started. Both toggle switches were in the "Auto" setting. We checked the hoses and tried pressing the Start buttons again, and still nothing happened. We then forced the fill by moving the toggle switches to "Manual" and pressing start, then turning the system back off when the secondary dewar vents started to spit LN2. We found that the insulation is broken on the wire than runs from the LN2 quick-fit connector on sp1 to the interlocks circuit. The break is right by the terminator on the LN2 connector. This break is not surprising, as the wire is subjected to LN2 venting when the secondary dewar is full, then subject to twisting as the LN2 connector flexes when the hoses are removed. However, the conductor in the wire is not visibly broken (we did not put an ohmmeter on it), so we are not certain this is the cause of the failure. We did consider whther the dewars were simply full already when we tried to refill them, but in the past neither Dan nor I had seen this completely prevent any filling. Of course, just to be contrary, the spectros DID autofill themselves when we hooked up the LN2in the morning! French is aware of the situation. ++++++++++++++++++++++++++++++++++++++ Problem #2 - SOP session driving the instruments hung, other anomalies ---------------------------------------------------------------------- We had several anomalies that seemed to be associated with Dan's SOP session that was being used to drive the spectrographs. First, the teamster seemed to take an age to transfer the preCalibs arc exposure for plate 1130 (exposure number 18888). But it did. Then, Dan's SOP session froze up completely after he issued the instChange command from the SOPGUI at the same instant the servers from grabInstruments started up. We could get no response from the SOP session, even with ctrl-C. Here are the messages immediately before the hang: sp1> IOP running as Backup Relay Server requestListen: listening 1 SOPGUI: instChange mcpPut: SYSTEM.STATUS mcpPut: sp1 slitdoor.open sp2 slitdoor.open mcpPut: sp1 slitheadlatch.open sp2 slitheadlatch.open mcpPut: az brake.off mcpPut: az AXIS.STATUS mcpPut: alt brake.off mcpPut: alt AXIS.STATUS I think that we haven't started slewing To kill this message, type 'schedule watchTCCSlew {} 0' Recovery required killing the process (pid 80335). The third "funny" was that the windows on the sirrobert laptop at the telescope were unresponsive for about a minute when Pete tried to use them while the instChange slew was proceeding., although then then started to respond as normal. Was that a symptom of sdsshost being busy? The identical hang on instChange occured again with the instChange issued at 04:06:04Z. And again at 06:31:48Z, except this time the servers had finished their business. Note that in each case, the slew to instrument change position DOES complete normally. Why this should be happening tonight for Dan when the same version of the software worked without incident for Pete last night is a mystery. In fact, tonight, we issued one instChange command from a SOP session being run by Pete, and it worked successfully and returned to the sp1> prompt (although the watchTCCSlew command is confused by the use of tccPutAndWait for both of us). But there was one significant difference between Dan's command and Pete's: Pete's did NOT list all the mcpPut commands to control the FF screens, slitheads, etc, although those commands clearly executed. Then Dan tried instChange from his command line, saw the MCP messages, and eventually got control back. We are more confused than ever, but the problem does seem to be related to the tccPutAndWait, watchTCCSlew and SOPGUI components. We will see if this persists when Pete drives tomorrow night. ++++++++++++++++++++++++++++++++++++++++++++++ Problem #3 - Spectro crate ptvme error? --------------------------------------- This may be related to the previous symptom of a delayed transfer of a spectro frame from the pool to sdsshost. After fexposure 18890 transferred to sdsshost around 02:02Z, all frame transfers stopped. A ptvmeTest -n4 command returned an error (below) that persisted across ptvmeReset commands, although a ptvmeTest -n 4 -a 0x18000000 command returned no errors. In any case, we entered a self-induced veil of confusion at this point, thinking, mistakenly, that issuing a teamster 60 -force command from a SOP session would start a _spectro_ teamster, when in fact it by default always starts an imager teamster. WHen we realised our mistake and started a spectro teamster with the command teamster 60 spectro -force we again saw spectro frames transferring and all was well. The good news is tht we were observing a plate that needed (S/N)^2 > 30 to finish, so during most of the above we kept collecting photons, and eventually lost only ~10 minutes doing test bias frames after physically rebooting the spectro DA crate. The teamster type issue has been folklored and a change request filed against IOP for the teamster comand to start the approprate instrument's getGangs. We let messages for both Atsuko (as DA focal point) and Craig (as computer expert), but in fact we'd solved the riddle before they returned our calls. Here are the errors: prn:sdsshost:~/ > ptvmeTest -n 4 PID: 194274 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 4 --> /dev/ptvme0n4 remAddr: 0x00000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n4", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 0 1995 ms 19:57 errno = 5: read (fd=3, 0x7F7F2B20, 0x00000004) = -1 (I/O error) CSB status: 0x80430010 AUX = (Timeout - LAN idle > 150 msec - target inite d? or VMEbus error) CMP = (Command Completed With Error Status) INNA = 0 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> prn:sdsshost:~/ > prn:sdsshost:~/ > ptvmeReset ptvmeMode: INFO: controller 0 reset (/dev/ptvme0n0) ptvmeMode: INFO: cur debug mode: 0 NORMAL ptvmeMode: INFO: cur debug: 0x00000000 ptvmeMode: INFO: cur timeout: 900 ticks ptvmeMode: INFO: def T/I xfer mode: 0x22 A32:D32 Program, Non-Privileged / A32:D32 Program, Non-Privileged ptvmeMode: INFO: op mode: 1 SINGLE_IO ptvmeMode: INFO: VMEbus priority: 0x03 ptvmeMode: INFO: VMEbus vector: 0x01 ptvmeMode: INFO: intern sem val: 1 ptvmeMode: INFO: user sem val: 1 ptvmeMode: INFO: PT-VME940 node: 0 ---=== SPECTROGRAPH COLLIMATION AND FOCUS ADJUSTMENTS ===--- Details of the adjustments made to the spectrograph mirror positions were not recorded today before the relevant xterm was clsoed, however the data are still in the Collimate files for exposure 18882 in observer@sos. The mirrors in sp1 were moved by +2600 to bring everything into focus.