APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday February 13, 2004 (MJD 53049) ---=== OBSERVING TEAM ===--- Day: Howard Brewington Night: Howard Brewington Scot Kleinman ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- COLD and blowing snow. No observing tonight folks. Theres nothing to see here. Move along. Well. OK, there was a pretty fun IOP/DA problem which gave us something to do for a couple of hours or so (or so it seemed). So you can come look at that if you really want to. ---=== OBSERVING LOG ===--- Afternoon checkout ------------------ Spectro focus was checked and found to be within spec; see problems section for details. Night ------------------ sometime around 05:21Z our servers died- we manually restarted them - waiting forever for them to get past the iopStartup feature we recently diagnosed. Problems continued with IOPs and servers etc. see Problems section. Given temperatures near what should be our new closing limite (~12F or -12C) and constant blowing snow, we never opened. Sigh. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4451 23:04Z 23:22Z 100 O -161.42 -156.85 42 ignore 4454 09:07Z 09:12Z 100 O -10.33 -9.03 20 ignore 4455 09:38Z 09:58Z 100 O -2.40 2.52 47 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 23:25 25005 0.0 bias ----- sequence 25006, plate 1362 ------- 23:50 25006 10.0 flat 23:52 25007 2.0 arc 23:58 25008 2.1 arc 00:00 25009 2.1 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL NONE Stays: JL NONE ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 23:27Z 1 1362 1.00000 0 1258 0 121 30.0 -11.8 29 26 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:22Z 11 8 2 339 (NNW) -9999 - 23:54Z 9 7 7 28 (NNE) -9999 - 00:26Z 8 7 15 2 (N) -9999 - 00:43Z 9 7 19 0 (N) -9999 - 01:15Z 8 6 8 20 (NNE) -9999 - 01:47Z 8 4 20 8 (N) -9999 - 02:20Z 8 4 17 8 (N) -9999 - 02:52Z 9 4 17 4 (N) -9999 - 03:25Z 10 4 19 4 (N) -9999 - 03:55Z 10 4 8 355 (NNW) -9999 - 04:28Z 10 4 7 7 (N) -9999 - 05:01Z 10 2 9 5 (N) -9999 - 05:34Z 10 2 19 38 (NE) -9999 - 06:06Z 10 1 10 1 (N) -9999 - 06:38Z 11 0 7 33 (NNE) -9999 - 07:10Z 11 0 21 8 (N) -9999 - 09:39Z 12 -1 17 18 (NNE) -9999 - 10:11Z 12 -1 10 36 (NE) -9999 - 10:43Z 12 -1 9 8 (N) -9999 - 11:16Z 12 -1 16 7 (N) -9999 - ---=== TELESCOPE STATUS ===--- never opened ---=== SOFTWARE USED ===--- IOP/SOP: v3_128_0 Watcher: v2_29_0 MCP: v5_25_0 TPM: tpm_v2_37_0 AstroDa: v14_47 TCC: TCC 2.7.1 July 22 2003 sdssProcedures: v1_84 SoS: v4_10_7 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5420 0.5420 Axial B 0.6330 0.6390 Axial C 1.0940 1.0900 Trans D -8.4520 -8.4610 Lateral E 10.8839 10.8966 Lateral F 10.8458 10.8712 GALILS Commanded: 4600. -2850. 850. -9250. 18400. 17850. Actual: 4600. -2850. 850. -9250. 18400. 17850. SETMIR VALUES PriDesOrient: 0.00 0.00 15.20 829.90 1.60 PriOrient: 0.00 0.00 14.91 830.49 3.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: -11.9 Alt.: 29.999822 MIGS TONIGHT NOMINAL Axial A 1.5160 1.5160 Axial B 1.1180 1.1180 Axial C 1.1150 1.1210 Trans D -0.0010 -0.0050 GALILS Commanded: 1628650. 1548279. 1574465. -3900. -8750. Actual: 1628650. 1548200. 1574500. -3900. -8750. SETMIR VALUES SecDesOrient: 1257.00 0.00 -25.00 0.00 146.27 SecOrient: 1257.02 0.04 -24.98 0.28 146.54 ---=== PROBLEMS IN DETAIL ===--- --== spectro focus ==-- ambT=2.18 focus checked but not adjusted sp1: A= -6884 B= -5885 C= -6973 sp2: A= -39579 B= 758 C= -2230 HOST/Crate problem? ------------------- We had several problems at the start of the night with failing teamsters, failing servers, slow to start IOPs, etc. which just kept getting worse. My IOP crashed, spewing the following: Read failed from PT icc10 CSB=0x00000000 rmt=0x18093638 cnt=0x00000004 ******************************************************************************** * * * Backplane communications between the ICC/OAC are no longer possible. * * * * Perform the following steps until operations are possible. * * * * 1. Check connections between the machines. * * 2. Reboot (reset) the ICC. * * 3. Restart this (OAC) program. * * 4. Reset the local PT-VME940 (ptvmeMode -R). * * 5. Power-cycle the VMEbus crates. * * 6. Reboot the local machine (host). * * * ******************************************************************************** ~ Host% I tried to endDrift the test bias on Howard's IOP, but it just hung. I tried to start a new IOP, but it kept spewing: Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) Waiting for command response from crate 1, ICC 0, control server (client 0) ptvmeTest showed everything was OK ~ Host%ptvmeTest -n 1 -a 0x18000000 PID: 628818 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 1 --> /dev/ptvme0n1 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n1", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 4 ms 01:41 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 B0 2C 64 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <..,d............> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> and ptvmeResets did nothing to improve the situation. We startd exiting IOPs, but that helped not at all - so we rebooted 'host which also helped not at all. We ended up rebooting the crates and starting everything over again. This worked.