APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday June 4, 2003 (MJD 52795) ---=== OBSERVING TEAM ===--- swing: Daniel Long night: Howard Brewington Jurek Krzesinski Connie Rockosi by phone ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Clouds, spectrograph problems (sp1 power supply) - exposure lost, then bias problem and servers failed to start on the last exposure, but exposure was read out, see the log and problem section for details. We completed one exposure on plate 1023. ---=== OBSERVING LOG ===--- Afternoon: ---------- The 2.5m checkout went smoothly. No problems. The PT was another matter. The filter wheel box is apparently not telling mop where it is, although it does move in response to commands, so it seems to think it knows where it is. This afternoon the spectrographs were not refocused. Below are the mean offset values from collimate. Before After b1: .16 - b2: -.14 - r1: -.11 - r2: -.17 - Final Piston Position sp1 sp2 motor A -560 -2380 motor B -560 -2390 motor C -560 -2380 Night: ------------------------------------- Clouds at the beginning of the night. We are waiting for better weather conditions. 04:57Z Cart 06 Plate 1314, stars on slew. Seeing variable, around 2" +-0.3" During the first exposure, sp1 complained about no reply from camera serial line. Exposure was lost. We called Connie for help. We tried cc ping on sp1 - no response (ping on sp2 gaved the answer: PING), we checked that sp1 power supply is on and that voltages are in spec, and checked if plugs are properly in sockets - they were ok. No idea what was wrong. Finaly we ended up cycling the sp1 power supply (see the problem section for details). This seemed to resolved the problem. While dealing with a problem we found out that heavy clouds are comming. 06:40Z Enclosure on because of rain approaching from northeast. Then we were testing spectrographs by taking biases , and this resulted in another problem: Error: sp1: camera not willing to stop scanning sp2: camera not willing to stop scanning But Connie thinks it's not connected with sp1 power supply failure (see the problem section). 09:52Z Cart 04 Plate 1023, Fk5 used to find guide stars. Seeing: 1.9" One exposure completed on this plate. 10:27Z Servers failed just after the exposure was read. Since it was twilight already and dust got high we closed the telescope and started to deal with the last problem - probably just a ftelnet connection failure, but see the problem section. 11:28Z endNight finished and was happy ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 3980 22:41Z 23:03Z 100 O -57.60 -52.08 51 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): y QA Procedures Done (y/n): y UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 21283, plate 1314 ------- 22:32 21283 0.0 bias 22:39 21284 0.0 bias 22:42 21285 0.0 bias 22:44 21286 0.0 bias 23:04 21287 10.0 flat 23:11 21288 2.1 arc 23:13 21289 2.1 arc ----- sequence 21290, plate -9999 ------- 00:42 21290 0.0 bias bias taken at the beginning of the night 06:31 21293 0.0 bias testing sanity of spectrographs 06:34 21294 0.0 bias -//- 06:48 21295 0.0 bias -//- 06:50 21296 0.0 bias -//- 06:54 21297 0.0 bias -//- 07:16 21298 0.0 bias -//- 07:24 21299 0.0 bias -//- 07:33 21300 0.0 bias -//- 09:16 21301 0.0 bias this bias was ok. ----- sequence 21302, plate 1023 ------- 10:01 21302 10.0 flat looks ok. 10:03 21303 2.0 arc looks ok. 10:25 21304 900.1 target 10:36 21305 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 05:18Z 6 1314 217.05 0.0034 50.92 0.0046 70.40 -0.0081 1.000160 10:15Z 4 1023 21.49 0.0004 55.32 0.0004 197.86 0.0079 0.999930 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL6481 Stays: JL6482 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 00:45Z 6 1314 1.00000 0 1257 0 121 30.0 19.8 19 230 gdr 1.9 10:14Z 4 1023 0.99993 690 1507 -230 22 55.3 13.3 16 131 gdr 1.8 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 21:30Z 75 29 13 246 (WSW) 930 - Mostly Clear 22:03Z 73 28 14 267 (W) 774 - Mostly Clear 22:34Z 74 27 14 260 (W) 765 - Partly Cloudy 23:05Z 68 25 12 275 (W) 806 - Mostly Cloudy 23:35Z 66 24 16 268 (W) 1715 - Mostly Cloudy 00:08Z 65 24 1 246 (WSW) 1124 - Mostly Cloudy 00:40Z 65 31 12 237 (WSW) 1467 - Mostly Cloudy 05:46Z 59 31 1 150 (SSE) 1905 - Mostly Clear 06:18Z 58 31 2 141 (SE) 2073 - Mostly Clear 06:48Z 57 31 1 168 (SSE) 1942 - Mostly Cloudy 07:18Z 58 31 4 173 (S) 1907 - Mostly Cloudy 07:49Z 58 30 7 163 (SSE) 1741 - Mostly Cloudy 08:20Z 59 30 9 191 (S) 1576 - Mostly Cloudy 08:50Z 59 30 11 132 (SE) 1480 - Mostly Cloudy 09:20Z 56 32 15 147 (SSE) 1579 - Mostly Cloudy 09:51Z 55 33 12 145 (SE) 1376 - Mostly clear 10:24Z 55 33 15 125 (SE) 2855 - Mostly clear 10:57Z 54 33 10 86 (E) 1534 - Mostly clear 11:29Z 55 32 7 74 (ENE) 2660 - ---=== TELESCOPE STATUS ===--- 04:00Z Doors open and fans on 04:42Z Enclosure off 06:49Z Enclosure on, doors cracked; telescope at 121, 30, 00 LN2 auto-fill hoses hooked up. 09:40Z Enclosure off 11:00Z Enclosure on, doors closed, fans off; telescope at 121, 30, 00 LN2 auto-fill hoses hooked up. 11:11Z Spetro Dewar 80.5 lbs 11:11Z Imager Dewar 79.0 lbs ---=== SOFTWARE USED ===--- IOP/SOP: v3_113_1 Watcher: v2_22_0 MCP: v5_18_0 TPM: tpm_v2_29_0 AstroDa: v14_47 TCC: TCC 2.6.8 November 13 2002 sdssProcedures: v1_69 SoS: v4_9_13 hoggPT: v1_6_8 plate-mapper: v4_2_0 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0990 0.0790 Axial B 0.8110 0.7970 Axial C 0.8280 0.8240 Trans D -9.0530 -9.0560 Lateral E 1.8520 1.8870 Lateral F 0.0000 1.4300 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 19.1 Alt.: 30.178431 MIGS TONIGHT NOMINAL Axial A 1.5120 1.5170 Axial B 1.1470 1.1560 Axial C 1.0940 1.0990 Trans D -0.7000 -0.7070 GALILS Commanded: 1619624. 1555290. 1576279. -3400. -7900. Actual: 1619650. 1555250. 1576350. -3400. -7900. SETMIR VALUES SecDesOrient: 1257.00 0.00 -20.00 0.00 130.98 SecOrient: 1256.99 0.02 -19.99 -0.47 130.87 ---=== PROBLEMS IN DETAIL ===--- Problem: one sp1 power supply problem: ------------------------------------- While reading out our first exposure (arc and flat were red ok.) we've got the following error message: IOP running as Backup Relay Server requestListen: listening 2003-06-05 05:21:12Z: registered (addr29) listener sdsshost:observer:2716 IOP running as Backup Relay Server requestListen: listening 2003-06-05 05:31:30Z 29 sp1 camcheck 2003-06-05 05:31:37Z 29 sp1 SHOWREADVOLTS 2003-06-05 05:32:01Z 29 sp2 camcheck 2003-06-05 05:32:03Z 29 sp2 SHOWREADVOLTS 2003-06-05 05:32:12Z: registered (addr30) listener sdsshost:observer:2716 =====endStare: beginning read out ============================= appending END EXPOSURE TIME is: 1200.1 sp1 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2100 sp2 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200 2003-06-05 05:34:30Z: sp1 is not idle...aborting 2003-06-05 05:34:30Z: sp1 has status no_response 2003-06-05 05:34:30Z: beginData has failed: sp1 is not idle...aborting; sp1 has status no_response The camera sp1 refused to talk to us about 05:30Z but we realized that something is wrong 4 minutes later while this reading exposure failed. And here is the murmur log excerpt: Murmur: 2003-06-05 05:34:09Z sdsshost IOP 103706 TEXTONLY MECH: sp1: sent s 2003-06-05 05:34:10Z sdsshost IOP 1926 TEXTONLY starting to fill out getWeather parameter file 2003-06-05 05:34:10Z sdsshost IOP 1926 iop_I_weatherData {time 1054791249} {param temp} {value 14} 2003-06-05 05:34:10Z sdsshost IOP 1926 iop_I_weatherData {time 1054791249} {param dewPoint} {value 30.8} 2003-06-05 05:34:10Z sdsshost IOP 1926 iop_I_weatherData {time 1054791249} {param dpTemp} {value 58.9} 2003-06-05 05:34:10Z sdsshost IOP 1926 iop_I_weatherData {time 1054791249} {param winds} {value 1.0} 2003-06-05 05:34:10Z sdsshost IOP 1926 iop_I_weatherData {time 1054791249} {param windd} {value 122.1} 2003-06-05 05:34:10Z sdsshost IOP 1926 iop_I_weatherData {time 1054791249} {param dustb} {value 1640} 2003-06-05 05:34:10Z sdsshost IOP 1926 TEXTONLY finished with getWeather parameter file 2003-06-05 05:34:12Z sdsshost IOP 103706 TEXTONLY MECH: sp1: received s spMechVersion v2_5_0 Bootup 719771169 SpectroID 1 SlitID 38 Air On Shutter_open_sensor Off Shutt... 2003-06-05 05:34:12Z sdsshost IOP 103706 TEXTONLY ...er_closed_sensor On Left_open_sensor Off Left_closed_sensor On Right_open_sensor Off Right_closed_sensor On Coll_motor_A... 2003-06-05 05:34:12Z sdsshost IOP 103706 TEXTONLY ... -565 Coll_motor_B -565 Coll_motor_C -565 Requested_exp.time 0.0 Exp_time_left 0.0 Last_exp.time 1200.1 Exp_state None... 2003-06-05 05:34:12Z sdsshost IOP 103706 TEXTONLY ...nShutter_open_transit .74 Shutter_close_transit .55 Coll_motor_A_status 0x85 Coll_motor_B_status 0x85 Coll_motor_C_status... 2003-06-05 05:34:12Z sdsshost IOP 103706 TEXTONLY ... 0x85 OK 2003-06-05 05:34:12Z sdsshost IOP 103706 TEXTONLY MECH: sp2: sent s 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=13 LDJ 52795 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:13Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=13 LDJ 52795 2003-06-05 05:34:15Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY MECH: sp2: received s spMechVersion v2_5_0 Bootup 719771169 SpectroID 2 SlitID 6 Air On Shutter_open_sensor Off Shutte... 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY ...r_closed_sensor On Left_open_sensor Off Left_closed_sensor On Right_open_sensor Off Right_closed_sensor On Coll_motor_A ... 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY ...-2379 Coll_motor_B -2391 Coll_motor_C -2382 Requested_exp.time 0.0 Exp_time_left 0.0 Last_exp.time 1200.1 Exp_state Non... 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY ...e Shutter_open_transit .73 Shutter_close_transit .65 Coll_motor_A_status 0x85 Coll_motor_B_status 0x85 Coll_motor_C_stat... 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY ...us 0x85 OK 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY endStare: beginning read out 2003-06-05 05:34:16Z sdsshost IOP 103706 TEXTONLY MECH: sp1: sent S 2003-06-05 05:34:16Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:16Z sdsshost IOP 2716 TEXTONLY proc_atexit(instrument): Failed to attach to instrument sp1 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=18 LDJ 52795 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:19Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=18 LDJ 52795 2003-06-05 05:34:19Z sdsshost IOP 103706 TEXTONLY MECH: sp1: received S OK 2003-06-05 05:34:19Z sdsshost IOP 103706 TEXTONLY MECH: sp2: sent S 2003-06-05 05:34:22Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:23Z sdsshost IOP 103706 TEXTONLY MECH: sp2: received S OK 2003-06-05 05:34:23Z sdsshost IOP 103706 TEXTONLY Can not find exposureRemaining in the event loop, so can not delete it 2003-06-05 05:34:23Z sdsshost IOP 103706 TEXTONLY Stopping exposureRemaining 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=23 LDJ 52795 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:23Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=23 LDJ 52795 2003-06-05 05:34:24Z sdsshost IOP 103706 TEXTONLY MECH: sp2: sent s 2003-06-05 05:34:24Z sdsshost IOP 2716 TEXTONLY CAMERA sp2: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2200: connection error in ftelnet: connection refused 2003-06-05 05:34:27Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:27Z sdsshost IOP 103706 TEXTONLY MECH: sp2: received s spMechVersion v2_5_0 Bootup 719771169 SpectroID 2 SlitID 6 Air On Shutter_open_sensor Off Shutte... 2003-06-05 05:34:27Z sdsshost IOP 103706 TEXTONLY ...r_closed_sensor On Left_open_sensor Off Left_closed_sensor On Right_open_sensor Off Right_closed_sensor On Coll_motor_A ... 2003-06-05 05:34:27Z sdsshost IOP 103706 TEXTONLY ...-2379 Coll_motor_B -2391 Coll_motor_C -2382 Requested_exp.time 0.0 Exp_time_left 0.0 Last_exp.time 1200.1 Exp_state Non... 2003-06-05 05:34:27Z sdsshost IOP 103706 TEXTONLY ...e Shutter_open_transit .73 Shutter_close_transit .65 Coll_motor_A_status 0x85 Coll_motor_B_status 0x85 Coll_motor_C_stat... 2003-06-05 05:34:27Z sdsshost IOP 103706 TEXTONLY ...us 0x85 OK 2003-06-05 05:34:28Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:28Z sdsshost IOP 2716 TEXTONLY proc_atexit(instrument): Failed to attach to instrument sp1 2003-06-05 05:34:30Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:30Z sdsshost IOP 103706 TEXTONLY CAMERA: sp2 camera: sent "CAMNAG ". Read<1: 1>: "CAMNAG IDLE ok " 2003-06-05 05:34:30Z sdsshost IOP 103706 TEXTONLY sp1 is not idle...aborting 2003-06-05 05:34:30Z sdsshost IOP 103706 TEXTONLY sp1 has status no_response 2003-06-05 05:34:30Z sdsshost IOP 103706 TEXTONLY beginData has failed: sp1 is not idle...aborting; sp1 has status no_response 2003-06-05 05:34:30Z sdsshost IOP 103706 TEXTONLY Stopping wait_for_endStare 2003-06-05 05:34:31Z sdsshost IOP 103706 TEXTONLY Stopping checkAndGuide 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=30 LDJ 52795 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD iy = 2003 im = 6 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT id = 5 djm = 52795 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT hr=5 min=34 2003-06-05 05:34:31Z sdssmcp tCmdLog dsc_I_trcInfo MJD CONT sec=30 LDJ 52795 2003-06-05 05:34:31Z sdsshost IOP 103706 TEXTONLY Stopping asyncCheck 2003-06-05 05:34:31Z sdsshost IOP 103706 TEXTONLY Can not find checkAndGuide in the event loop, so can not delete it 2003-06-05 05:34:31Z sdsshost IOP 103706 TEXTONLY Starting checkAndGuide 2003-06-05 05:34:31Z sdsshost IOP 103706 TEXTONLY guiderMon found /data/spectro/guider/gimg0010.fits 2003-06-05 05:34:32Z sdsshost IOP 2716 TEXTONLY CAMERA sp1: FAILURE on ftelnet t-g-sdss-2.apo.nmsu.edu 2100: connection error in ftelnet: connection refused 2003-06-05 05:34:32Z sdsshost IOP 2716 TEXTONLY proc_atexit(instrument): Failed to attach to instrument sp1 and we traced down the problem to the power supply: sp1> goStare 0 -init -bias A [cc ping] brings no response from sp1: electronics powered down? 2003-06-05 05:45:45Z: Choosing staring readout mode 2003-06-05 05:45:45Z: Set frames to Rows=(2048+21) and Cols/2=(20+1024+20) Binning=(1,1) But we chceked that the sp1 power supply was on!. Connie was called. LEDs on the power supplies were all green, voltages were in spec (all above 98.5%). Meanwhile Connie looked in the murmur log and there were no indications of a problem before sp1 stopped responding. And one more thing for Connie: ------------------------------ The readings for +5 V on the sp1 power supply unit were 98.5% before and after rebooting. Actualy we didn't notice any suspicious readings or differences before and after rebooting. Bias problem: ---------------------------------------------------------------------- Then, to check sanity of spectrographs, we run a few biases, and on the third attempt we've got an error: sp1> goStare 0 -init -bias 2003-06-05 06:37:10Z: Choosing staring readout mode 2003-06-05 06:37:10Z: Set frames to Rows=(2048+21) and Cols/2=(20+1024+20) Binning=(1,1) sp1 camstat {} t-g-sdss-2.apo.nmsu.edu 2100 sp2 camstat {} t-g-sdss-2.apo.nmsu.edu 2200 setupCamera {error writing "file8": Bad file number} t-g-sdss-2.apo.nmsu.edu 2100 sp1 setupCamera {} t-g-sdss-2.apo.nmsu.edu 2100 setupCamera {error writing "file10": Bad file number} t-g-sdss-2.apo.nmsu.edu 2200 sp2 setupCamera {} t-g-sdss-2.apo.nmsu.edu 2200 sp1 cameraDetach {} t-g-sdss-2.apo.nmsu.edu 2200 sp2 cameraDetach {} t-g-sdss-2.apo.nmsu.edu 2200 Error: sp1: camera not willing to stop scanning sp2: camera not willing to stop scanning But the next ones went ok. (5 of them). Connie thinks, the bias problem wasn't connected with sp1 power supply: Connie wrote: Jurek, It looks like the problem you had taking biases after you had to power-cycle the spectrographs is our old friend the tcl file-handle bug. Here is an excerpt from the murmur log: 2003-06-05 06:37:04Z sdsshost IDA 103706 TEXTONLY CAMERA: Bad status from sp1 camera micro. Wanted " " Returning "error writing "file8": Bad file number" So it is probably not related to whatever mysterious problem caused sp1 to stop talking to us. ------------------------------- Servers failed to start: _______________________________________________________________________ Waiting for read to finish Seq 21302 Exp 21304 Frame 21304 Do not schedule startFlushCharge sp1 200 Do not schedule startFlushCharge sp2 200 =====endStare: done =========================================== mcpPut: SYSTEM.STATUS ************************************************** ====>>>>>>RELAY SERVER FAILED TO START!!!!<<<===== ************************************************** couldn't fork child process: Resource temporarily unavailable ************************************************** sp1> Looks like ftelnet problem, but what does murmur says? Here you go a proper excerpt for that event: ------------------------------------------ 2003-06-05 10:25:38Z sdsssd1 dLine dsc_I_trcInfo readout complete 2003-06-05 10:25:39Z sdsssd1 dLine dsc_W_trcWarn VCI has more line(s) 2003-06-05 10:25:40Z sdsssd1 dLine dsc_I_trcInfo readout complete 2003-06-05 10:25:40Z sdsssd1 dLine dsc_W_trcWarn VCI has more line(s) 2003-06-05 10:25:40Z sdsshost IOP 1926 TEXTONLY PMM tpmHandler: caGet1 failed 2003-06-05 10:25:41Z sdsshost IOP 103706 TEXTONLY Seq 21302 Exp 21304 Frame 21304 2003-06-05 10:25:41Z sdsshost IOP 103706 TEXTONLY Stopping wait_for_endStare 2003-06-05 10:25:41Z sdsshost IOP 103706 iop_I_currentInstName cart 4 2003-06-05 10:25:42Z sdsshost IOP 103706 TEXTONLY Can not find wait_for_endStare in the event loop, so can not delete it 2003-06-05 10:25:42Z sdsshost IOP 103706 TEXTONLY Stopping wait_for_endStare 2003-06-05 10:25:42Z sdsshost IOP 103706 TEXTONLY couldn't fork child process: Resource temporarily unavailable 2003-06-05 10:25:44Z sdssmcp MEI_DC dsc_I_trcInfo Encoders for axis rotator differ by -624991 2003-06-05 10:25:44Z sdsshost IOP 2731 TEXTONLY PMM tpmHandler: caGet1 failed 2003-06-05 10:25:44Z sdsshost IOP 103706 TEXTONLY Stopping listenerToPort 2003-06-05 10:25:44Z sdsshost IOP 103706 TEXTONLY Stopping packeteer 2003-06-05 10:25:44Z sdsshost IOP 103706 TEXTONLY Stopping keepAlive 2003-06-05 10:25:44Z sdsshost IOP 103706 TEXTONLY Can not find listenerToPort in the event loop, so can not delete it 2003-06-05 10:25:44Z sdsshost IOP 103706 TEXTONLY Starting listenerToPort 2003-06-05 10:25:44Z sdsshost IOP 103706 TEXTONLY requestListen spectro: Neither 30310 nor 30307 appears to exist what is this (?): 2003-06-05 10:25:42Z sdsshost IOP 103706 TEXTONLY couldn't fork child process: Resource temporarily unavailable ----------------------------------------------------------------------------- Since it's twilight already, there is no time for another exposure, but we wanted to see if ptvme behaved properly and checked it with ptvmeTest: sp1> ptvmeTest -n 0 -a 0x18000000" PID: 464623 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 0 --> /dev/ptvme0n0 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n0", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 0 997 ms 10:31 errno = 5: read (fd=3, 0x7F7F2AD0, 0x00000004) = -1 (I/O error) CSB status: 0x80420010 AUX = (Timeout - Target failed to select within 1 second) CMP = (Command Completed With Error Status) INNA = 0 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ------------------ sp1> ptvmeTest -n 1 -a 0x18000000" PID: 461234 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 1 --> /dev/ptvme0n1 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n1", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 24 ms 10:26 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 AE B9 FC 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ----------------------------- sp1> ptvmeTest -n 2 -a 0x18000000" PID: 462618 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 2 --> /dev/ptvme0n2 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n2", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 92 ms 10:28 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 B0 2C 50 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <..,P............> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ---------------------- sp1> ptvmeTest -n 3 -a 0x18000000 PID: 462661 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 3 --> /dev/ptvme0n3 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n3", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 8 ms 10:29 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 AE FB C0 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ---------------------- sp1> ptvmeTest -n 4 -a 0x18000000 PID: 463427 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 4 --> /dev/ptvme0n4 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) errno = 0: open ("/dev/ptvme0n4", O_RDWR) = 3 errno = 0: ioctl (fd=3, PT_IOCTL_GET_PRIORITY, &ptPriority) = 0 ptPriority = 0x03 errno = 0: ioctl (fd=3, PT_IOCTL_GET_VECTOR, &ptVector) = 0 ptVector = 0x01 errno = 0: ioctl (fd=3, PT_IOCTL_GET_TIMEOUT, &ptTimeoutSave) = 0 ptTimeout = 900 ticks errno = 0: ioctl (fd=3, PT_IOCTL_GET_OP_MODE, &ptOpMode) = 0 ptOpMode = 1 SINGLE_IO errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG_MODE, &ptDebugMode) = 0 ptDebugMode = 0 NORMAL errno = 0: ioctl (fd=3, PT_IOCTL_GET_DEBUG, &ptDebugSave) = 0 ptDebug = 0x00000000 errno = 0: ioctl (fd=3, PT_IOCTL_GET_NODE_ID, 0) = 0x00 . 1 15 ms 10:29 errno = 0: lseek (fd=3, 0x00000000, SEEK_CUR) = 0x00000000 00000000: 01 AE F2 C8 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F <................> 00000010: 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F <................> 00000020: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F < !"#$%&'()*+,-./> 00000030: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <0123456789:;<=>?> ------------------ hmmm, at the end insteed ptvmeTest we issued ptvmeReset anyway, after that, we tested spectrographs with goStare 0 -init -bias, and it was ok. Jurek