APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday September 28, 2007 (MJD 54372) ---=== OBSERVING TEAM ===--- Afternoon: kaike Pan, Audrey Simmons Night: Dmitry Bizyaev Dan Oravetz Support: Jon Brinkman Eric Neilsen (by phone) Fritz Stauffer (by phone) left messages on Kurt Biery's work and office answer machines. ---=== OBSERVING PLAN ===--- SEGUE Science ---=== OBSERVING SUMMARY ===--- During afternoon checkout, we had problem with id5's framecatcher, and chksum It ended up with replacing its VCI card fixing the probelm (see log). The night was humid, cloudy, windy, and sometimes rainy. We didn't get a chance to open and observe. ---=== OBSERVING LOG ===--- afternoon: ----------- Spectro focus checked and adjusted for 11.5 C (for problem section for information) Data for 54359-54362 were removed from host2. id5 was not happy at all today: When we started doghouse bias, watcher complined about id5 with the fellowing messages: 1) Status: Inactive, Run: 0, (and other run # 6991-6993), Frame: -1 (and many others, 12, 20) dsc W chksumMismatch Checksum mismatch (expected -2075405749 vs. computed 323857328) (dsc_W_chksumMismatch Checksum mismatch (expected -2075405749 vs. computed 323857328) : 2007-09-28 23:44:55Z ) 2) id5's archiver and tape drives frameCatcher: Invalid frame idRR-006993-g5-0016.fit ( : 2007-09-28 23:54:15Z) It looked like that files were corrupted somewhere during data transfer from the pool to host1 (file1). So ew tried 1) rebooted all photo and astro crates, that did not do anything good, 2) rebooted host2 without any luck again. 3) rebooted file1 and all crates. We got micro error messgae: Camera Serial Line server is not responding (Last packet: 2007-09-28 23:51:32Z) But when ping imager, cc camCheck, and cc imstat, it showed imager is communicating. After we rebooted sdssfiles1 and all crates, we did a bias run (6994). Every node was taking data, and id 5 stoped after the first frame. However, iccSoundOff said id 5 was available, and watcher did not complained about this. Jon sound that VCI has fallen behind (by doing dsclat id5). We tried again bias run (6995), id 5 did the same. During another attempt bias run (6996), we got this watcher message after Jon restarted readout for id5 (readoutBegin id5). dsc E monPriorityHigh The monitor task has identified a serious problem: Archiver has fallen behind (50 frames are not archived), and message from MCP. telnet reading on port 41788: S errno ECONNRESET (2007-09-29 01:36:11Z sdssmcp) (telnet reading on port 41788: S_errno_ECONNRESET (2007-09-29 01:36:11Z sdssmcp): 2007-09-29 01:36:23Z ). After Jon VCI rebooted, we rebooted crates again. During nother try of bias run, id 5 stoped at frame 17 with the same error message as we had in very beginning. Kurt was called and left messages on his machines in office and home. Then Eric and Fritz were called. Eric After Jon Clean contact in CPU and VCI card, we tried another bias run. We still had problem of chksum and framecatcher on id5. But its VCI behaves correctly. Because the bias levels for CCD25 and CCD 45 were not right, so we did an iack on imger, then tok bias. This did not work out. Then we replaced VCI card for id 5. After the replacement, we took two bias runs (7000 and 7001), they are looked good. Thanks to Jon for helping. Night: ------ The problem with imager gone around 04:10Z. It didn't hurt observations because of high humidity, clouds, lightnings, and sometimes rain. 10:44Z high humidity gone. Clouds are starting to go away. Nevertheless, sky remains partly cloudy, and the clouds move very fast. The worse thing, they come from the east, a blind spot for radars. A danger of rain is still present. We decided not to open 50- min. before the twilingh in this situation. 10:55Z EndNight complains about missing imager files. It's not a surprise since some files were missed due to the problem with "id5" crate. All other issues went smoothly through the endNight. ---=== IMAGING RUN SUMMARY ===--- 6993 23:42Z 23:49Z 100 O 72.16 73.75 25 ignore 6994 01:12Z 01:24Z 100 O 94.63 97.65 34 ignore 6995 01:26Z 01:35Z 100 O 98.13 100.43 30 ignore 6996 01:39Z 01:42Z 100 O 101.53 102.15 18 ignore 6997 02:05Z 02:17Z 100 O 108.01 110.99 33 ignore 6998 02:57Z 03:18Z 100 O 120.86 126.30 51 ignore 6999 03:28Z 03:34Z 100 O 128.66 130.31 25 ignore 7000 03:42Z 03:56Z 100 O 132.27 135.80 38 ignore 7001 03:57Z 04:15Z 100 O 136.11 140.42 43 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- ---=== LTMATCH RESULTS ===--- ---=== SPECTROSCOPY DATA SUMMARY ===--- ----- sequence 46437, plate -9999 ------- 23:58 46437 0.0 bias 03:48 46438 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- ---=== WEATHER LOG ===--- 23:41Z 59 50 12 172 (S) 860 - 00:13Z 58 50 14 213 (SSW) 937 - 00:45Z 57 51 18 173 (S) 1000 - 01:17Z 57 51 15 181 (S) 976 - 01:54Z 55 52 18 144 (SE) 1054 - 02:28Z 54 52 22 146 (SE) 1044 - 03:01Z 55 51 13 175 (S) 986 - 03:33Z 54 51 24 168 (SSE) 919 - 04:09Z 53 51 25 151 (SSE) 913 - 04:41Z 52 51 22 142 (SE) 943 - 05:13Z 52 50 21 149 (SSE) 858 - 05:45Z 51 50 22 158 (SSE) 844 - 06:18Z 51 49 22 150 (SSE) 818 - 06:50Z 52 48 26 159 (SSE) 700 - 07:22Z 53 46 19 159 (SSE) 519 - 07:54Z 52 50 12 174 (S) 895 - 08:26Z 51 49 15 178 (S) 766 - 09:02Z 50 47 16 183 (S) 643 - 09:34Z 49 47 15 163 (SSE) 615 - 10:07Z 51 44 27 152 (SSE) 392 - 10:39Z 52 43 20 180 (S) 518 - ---=== TELESCOPE STATUS ===--- Not opened this night because of weather problems. Status at: 11:05Z Telescope stowed at: 30,121,00 mount Instrument mounted: Cartridge 2 , plate 2805 Counterweights at: 270 lbs LN2 autofill systems: Connected and turned on. 180L LN2 dewar scales: Spectro 208 lbs, 20.6 psi Imager 161 lbs, 19.0 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_38_0 Watcher: v2_36_0 MCP: v6_4_0 TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.7.2.1 August 6 2004 sdssProcedures: v2_36 SoS: v5_1_8 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -5.2090 -5.2130 Axial B -0.2134 -0.2135 Axial C -0.0013 -0.0016 Trans D 11.6370 11.6400 Lateral E 9.9695 9.9568 Lateral F 11.5316 11.5316 GALILS Commanded: 4600. -2000. 2650. -9300. -9800. -9800. Actual: 4600. -2000. 2650. -9300. -9800. -9800. SETMIR VALUES PriDesOrient: 0.00 0.00 0.00 0.00 0.00 PriOrient: 0.00 0.00 0.00 0.00 0.00 SECONDARY: ---------- Focus: 0.00 Air Temp.: 14.9 Alt.: 90.000010 MIGS TONIGHT NOMINAL Axial A 0.7969 0.7973 Axial B 0.6082 0.6041 Axial C 0.5415 0.5425 Trans D 0.6000 0.5999 GALILS Commanded: 1800329. 1304973. 1638679. -9450. -4400. Actual: 1798628. 1309538. 1643409. -9472. -4376. SETMIR VALUES SecDesOrient: 1257.00 -60.00 -140.00 -190.00 200.12 SecOrient: 1254.99 -59.24 -142.10 -190.84 199.62 ---=== PROBLEMS IN DETAIL ===--- Spectro Focus Info: ------------------- Focus adjusted for the colder weather. Now set for about 11.5 C (using the spectro body temperatures) sp1> caget tpm_TM_PMSSAvg tpm_TM_TelairAvg.VALA tpm_TM_PMSSAvg 12.8 tpm_TM_TelairAvg.VALA 12.6 sp1 14.1 sp2 12.1 Mean offset (pix) Collimator Position sp1 sp2 b1 r1 b2 r2 A B C A B C old . .16 . -.11 1098 1099 1098 -585 -601 -584 new . -.14 . -.09 -1853 -1853 -1852 -91 -95 -93 Camera Serial Line server is not responding (Last packet: 2007-09-28 23:51:32Z)