TCCON > Sites > JPL > Instrument History

Instrument History

Table of contents
  1. 1. Operational Status Record
    1. 1.1. Operational Status Notes
      1. 1.1.1. July 2017
      2. 1.1.2. June, 2017
      3. 1.1.3. July, 2013
      4. 1.1.4. June 2013
      5. 1.1.5. May 2013
      6. 1.1.6. April 2013
      7. 1.1.7. February 2013
      8. 1.1.8. January 2013
      9. 1.1.9. December 2012
      10. 1.1.10. July 2012
      11. 1.1.11. June 2012
      12. 1.1.12. May 2012
      13. 1.1.13. April 2012
      14. 1.1.14. March 2012
      15. 1.1.15. February 2012
      16. 1.1.16. January 2012
      17. 1.1.17. December 2011
      18. 1.1.18. November 2011
      19. 1.1.19. October 2011
      20. 1.1.20. September 2011
      21. 1.1.21. August 2011
      22. 1.1.22. July 2011
      23. 1.1.23. June 2011
      24. 1.1.24. May 2011
      25. 1.1.25. April 2011
      26. 1.1.26. March 2011
      27. 1.1.27. February 2011
      28. 1.1.28. December 2010
      29. 1.1.29. October 2010
      30. 1.1.30. September 2010

Operational Status Record

SiteUser NameOperational?DescriptionDate
JPLcoleenYes2017-10-02
JPLcoleenYesAligned FTS 6/21/17 and collecting data since2017-07-24
JPLcoleenNoContainer packed up (7/1/13) for move to Dryden2013-07-03
JPLcoleenNoContained packed up (7/1/13) for move to Dryden2013-07-03
JPLcoleenYesInitial measurements (since container returned to JPL) underway2013-01-17
JPLcoleenNoTrailer returned to JPL on 12/19/12 ans is currently being re-assembled.2013-01-10
JPLcoleenNoInstrument shut down for move to IN on 7/24/122012-08-03
JPLcoleenYesFully operational2012-03-29
JPLcoleenYes2012-01-25
JPLcoleenYesFully operational2011-11-15
JPLcoleenYesFully automated, although periodic problem with IFS controller remains.2011-09-03
JPLcoleenYesFully operational, but not fully automated.2011-05-24
JPLcoleenNoNew FTS system for OCO-2 TVAC to be installed 3/28/112011-03-23
JPLcoleenNoAwaiting delivery of the new FTS system for OCO-2 TVAC2010-11-16
JPLdwunchNoThe instrument for OCO-2 TVAC testing is about to be purchased.2010-04-29

Operational Status Notes

July 2017

  • 2017-07-26
    • Scanner failure; reinitialized and started right back up.
  • 2017-07-25
    • The new Netgear router has been placed in service in the JPL container
      today, around 6 PM, after the sun went behind the hill but before the
      official sunset.  I chose this timing to not affect data taking or
      analysis.
      
      I had first configured the router in my office.  I had to download
      the user manual to figure out port forwarding.  For some bizarre
      reason, the router user interface does not use the words "port
      forwarding" to describe the feature.  Sigh.
      
      When I walked up to the container, the irrigation was in use on
      the slope below the container.  It is possible that fine mist may
      be blown by the wind up to the suntracker.  So my first action was
      to place the system in Hold.
      
      Then I shutdown IFSloop and transitioned to the new router: powered
      down the old router, moved the Ethernet cables one at a time into the
      same port number as the old router, power up the new router.
      
      I noticed that the JPL network connection was indicated at 1 Gbit/s.
      The IFS125 was at 100 Mbit/s, but the Herc and suntracker at 10 Mbit/s.
      So, I shutdown QNX6 and power-cycled the Herc: this set its port
      to 100 Mbit/s as indicated both by the router and the Herc panel.
      
      Unfortunately, I had not made note of the Herc network speed
      indication before powering down the old router.
      
      I was pleased to note that everything worked correctly, both from
      inside the container reaching out, and from outside the container
      reaching in via the port forwards.
      
      I had previously measured the transfer rate from the Herc to ccycle
      with the old router and obtained 820 kB/s.
      
      With the new router, transferring the same file now achieves 1086 kB/s.
      
      Of course, part of the timing load resides in the Herc having to encrypt
      the data during "scp".  I have not made any other kind of measurements.
      
      If the new router is reliable, we can switch to the same kind at Caltech.
      
  • 2017-07-23
    • Power cycled ST, ran links to connect, and started IFSloop again.
    • Measured pressure at 9:50 am LT, Zeno: 967.1 HPa, Temp = 26.1 C; Digiquartz: 967.707.
  • 2017-07-22
    • Suntracker failure occurred at 4:20 pm local time.  Could not reset with links; ST needs power cycling.
  • 2017-07-19
    • Scanner failure
  • 2017-07-16
    • Scanner failure

June, 2017

  • 2017-06-27
    • Scanner failure
  • 2017-06-21
    • Although basic alignment (adjusting the laser beam input, adjusting the lamp filament and solar disk input on the APT wheel, peaking the last flat at the interferometer exit, and peaking the detectors) was already done, the interferometer alignment needed to be checked prior to comparisons with the GOSAT and Caltech EM-27s tomorrow.  Initial lamp runs revealed terrible fringing and very broad, unsymmetric line shapes.  These issues persisted after venting the FTS and swapping HCl cells (Cell #23 was and is again, in place.
    • Using the external laser alignment kit, we made a slight adjustment of the fixed cube corner at or near ZPD (and with APT=10000) to correct for a small shear misalignment.  Then, we moved the carriage toward MOPD, while reducing the size of the entrance aperture, and found that the interferometer was not aligned with the entire FTS. (The laser fringes were no longer centered in the entrance aperture.)  The  IR Adjustment Screws of the first flat mirror directing the beam from the first field stop into the interferometer were adjusted to center the fringes.  Upon removing the alignment kit, the changes were compensated through adjustments of the IR Adjustment Screw on the Compensating Mirror to get the laser beam through the exit aperture.  (Linefit results to follow.)
  • 2017-06-06
    • Completed setup of Armstrong instrument at JPL. New scanner cable (failed once), cleaned bars.
    • After adjustment of the laser entrance mirror (the MOPD Laser Adjustment Screws) 
        LASA PK-PK Zero ME LASB PK-PK Zero ME
      ZPD

      2.45 V

      -7.85 V

        10.30 V
       -8.975 V  0.821

       8.45 V

      1.225 V

       7.225 V 0.025 V  0.751
      MOPD

      +2.125 V

      -7.625 V

       9.75 V

      -8.950  V  0.786

      8.275 V

      1.375 V

        6.90 V 0.050 V

       0.723

       
       

July, 2013

  • 2013-07-01
    • Barometer reading- 22.9 C, Princo = 729.6 mmHg, Setra = 966.1 mbar (8:47 lt)

June 2013

  • 2013-06-28
    • Took a lamp run, while instrument under vacuum, and it looks very much like the one we started with yesterday or perhaps even better (narrower and deeper). Will stop here (jf20130628lgee0x.001) and run Linefit.
  • 2013-06-27
    • Decided to use the new "alignment kit" to investigate the alignment of the JPL FTS before it heads to Dryden.  Before starting, took one lamp spectrum (under vac.) - jf20130627lgee0x.001.  Then vented, positioned alignment kit, adjusted the filament position slightly, removed the HCl, and looked at the camera images of the fringes.  They were very close to centerered, but small adjustments were made to both the cube corner and the horizontal and vertical adjustment screws of the IR Adjustment Mirror at various OPDs and aperture settings. 
    • It was necessary to adjust the last flat mirror to get the image on the camera, so this had to be adjusted back upon removing the alignment kit.  The optimized DC value of the InGaAs detector = -1.85 V( APT=1000).
    • The vented spectra were remarkably unsymmetric (.002), so we attempted to put the cube corner back were it was (.003).  This change was so small that there was little affect.  We looked at the Si decter at 0.3 cm-1 res. to see the change in the continuum (.004).  The continuum seemed to drop slightly with the cube corner in its original positions, so we returned the cube corner to the fringe-centered position (.005).  Then adjusted the horizontal (.006) and vertical (.007) screws on the IR Adjustment and Compensation mirrors.  No real change was seen in the spectra.  We even moved the HCl cell around quite a bit, since we had also touched this, and took another spectrum (.008) but it made no difference . 
    • Finally closed up the detector region, opened the dome and looked at the solar alignment (which was perfect), pumped down the instrument, and returned control to IFSloop.
  • 2013-06-26
    • At 8:00 local time, received a Scanner failure:
      14:54:59 Srvr: TMA: IFS Solar
      14:55:00 [ERROR] 14:54:55:219:f:CAN packet sender:GNS=1:Device disabled
      (Same as yesterday.)  The IFS125 detectors were in AC mode (although I am not positive I waited long enough) so I reset the instrument.  Now in DC.
  • 2013-06-25
    • Received  Scanner failure errore at 10 am.
      This was caused by this in the log:
      
        16:55:26 Srvr: TMA: IFS Solar
        16:55:27 [ERROR] 16:55:19:219:f:CAN packet sender:GNS=1:Device disabled
      
      I've (JF) re-init IFSloop.  It's still waiting for the clouds to clear before
      it tries to scan again.
      
    • At 13:50 local time, workers came to install hook for safety harness.  In the midst of measurements, a workman walked up the rolling stairs and across the top of the trailer (apparently, not having taken the Fall Protection class!)  I explained what we were doing and asked them to come back later.
  • 2013-06-23
    • Shuji Kawakami-san (JAXA) set up the Optical Spectrum Analyzer inside and the sun tracker / meteorological station outside the trailer in the hopes of doing comparisons with the IFS125.  Unfortunately, the weather has not been very cooperative with clouds during the afternoon and June-gloom and even drizzle forecasted for the next two days.

May 2013

  • 2013-05-21
    • The suntracker failed at ~ 5 pm, as was indicated by the suntracker stale counter.  Accessed the suntracker through "links http://suntracker".  Set to Position Sun and then to Tracking Mode.  Finally, returned to IFSloop and reinitialized the software to clear the error.
      • I think Debra caused the suntracker to fail by accessing its webpage remotely. She won't do that again.
  • 2013-05-18
    • There was a scanner failure, unrelated to container temperature this time.  Console indicated all was well with the IFS: re-init the software allowed the measurements to start again.  However, the detector ADCs remained in AC mode.  Went to Hold, did IFS125 software reset, and re-init the software again.  Data started in DC mode around 10:30 LT.
  • 2013-05-03
    • Cleaned the CaF2 window and checked the alignment of the solar beam.  Beam was well aligned on the input tube and on the entrance slit, so no changes were made.

April 2013

  • 2013-04-26
    • New AC unit was installed in the trailer, but all power to the trailer was cut in the process.  By 1 p.m. the power was back on, but the suntracker and IFS125 needed to be reset.  The power was cycled on the suntracker electronics box and the suntracker was reinitialized and set to Position Sun using links.  IFS125 appeared to be running (no errors) however the was zero intensity on the detectors.  I exited IFSloop, reset the 125 manually, and restarted IFSloop.  I think all is well.
    •  @ 14:15 PDT, T=24.5 C, p(barometer) = 735.3 mm Hg (raw), p(Setra) = 973.7 hPa
  • 2013-04-23
    • Laser B modulation "too small" error, causing a scanner error. Reinitializing the software seems to have done the trick for now.
  • 2013-04-20 through 22
    • Container was turned off because the air conditioner is not cooling. Back on on 2013-04-23, and hopefully the air conditioner will be fixed today.
  • 2013-04-09
    • Container has been in hold since yesterday due to high winds.  Jim noted on 4/6/13 and 4/7/13 that the suntracker seemed to loose the TTM due to either the passage of a cloud or when the time was syncronized.  I took the container out of hold ~3 pm (local time) and the suntracker went right into TTM.  I simulated clouds by blocking the solar beam for a minute or two, but the suntracker returned to TTM immediately after the "clouds" passed.
    • @ 15:20 PDT, T=28.3 C, p(barometer) = 729.1 mm Hg (raw), p(Setra) = 964.4 hPa. (Used this to update pressure calibration spreadsheet.)

February 2013

  • 2013-02-11
    • The sky was clear today, so I went to the container just before noon.  The solar beam was very well centered on the input tube.  I decided to block part of the solar beam with balloon tape.  I did this without stopping the interferometer.  The signal drop occurred during data slices b1111329.0 and b1111330.0, at 12:02 LT.  The InGaAs DC term went from 0.522 to 0.406.  The ZPD peak value went from 0.817 to 0.639.  So, this is only a reduction to 78% of the original level.  It may not be enough, and we'll re-visit this based on the zero-level offset in the spectra.
  • 2013-02-03
    • @ 8:50 AM PST, T=22.3 C, p(Princo barometer) = 734.8 mm Hg (raw), p(Setra) = 973.1 hPa.

January 2013

  • 2013-01-31
    • Clear day, so a good time to check the solar alignment.  Did all this near local noon with IFSloop in Hold...
    • Removed dead wasp from the CaF2 window (it was on the side that would illuminate the quadrant sensor).
    • Noticed that the sunlight was not centered on the input tube.  Removing the insulation restores the centering.  The problem is caused by the bungee cords pulling the interferometer unevenly towards the blue support frame.  This has no effect on the data quality because the quadrant sensor assembly is attached to the IFS125 and gets pulled in the same way.  However, I've been using the centering on the input tube as a proxy for good suntracker alignment check, given that the entrance field stop is not visible in normal operation.  So I'll use only two loose bungee cords to avoid pulling on the IFS125.
    • Moved suntracker electrical cables away from light beam, as one of them could cast a shadow just outside the tube.  This should have had no effect on the data.
    • Released vacuum and check solar image on the 4 mm field stop.  A very small adjustment was required on the OAP just before the field stop.  The previous alignment had been done with high cirrus clouds, so this is not surprising.
    • Pumped down and re-initialized IFSloop.  (Yes, I did take the index card out.  Checked it twice.)
  • 2013-01-21
    • Very dry day (Zeno_RH of 4% to 5%), highest InGaAs ZPD value of 0.900 !!!
  • 2013-01-20
    • Dry day (Zeno_RH of 9% to 10%), highest InGaAs ZPD value of 0.887.
  • 2013-01-19
    • Dry day (Zeno_RH of 8% to 10%), highest InGaAs ZPD value of 0.882.
  • 2013-01-18
    • Suntracker went into tracking mode without problems this morning and IFS was taking solar scans when we arrived.  Decided to take one more lamp run to investigate whether the bumping of the quadrant sensor plate and consequent adjustments changed the linefit (see below).  Quick look at OPUS lamp run looks very promising!
    • Pressure measurement: Barometer T = 21.6 C; P = 735.8 mm Hg, Setra P = 974.6 hPa.
  • 2013-01-17
    • Coaligned the interferometer and the IR, but found that it was already seemingly well-aligned.  Still must look at the linefits.  Had some trouble with the tracker going into the tracking mode and discovered that the insulation had bumped the top plate of the quadrant sensor.  Manuel adjusted this and tracking mode started up, but am not somewhat undersure of the alignment.  May want to redo in the near future.  In the meantime, started up IFSloop to attempt solar measurements this afternoon.
    • Took first pressure measurement: Barometer T = 24.8 C; P = 734.0 mm Hg, Setra P = 971.8 hPa.  (May want to open up hole on the barometer mount for slight sideways adjustment.)
    • Linefits looks good (will post when back to Caltech.)  Final lamp spectra jf20130117lgee0x.090 (vented) and .010 (evacuated).
  • 2013-01-16
    • Centered the solar beam on the entrance field stop (AP1) and exit field stop (AP2) with the adjustable flat mirror just before AP2. Optimized DC value of the InGaAs detector = -2.154 V; Si detector = -0.297 V (APT=1000).
    • Ran the ghost test with filter (70-3F-FA) using the 7kHz XPM file dictated by Messershmidt et al.  Started with file jf20130116-07-ECL00gpr.001.  The GPR's are shown in the table below:
      Scanner Frequency Number of Scans Ghost Amplitude Parent Amplitude GPR Parent Wavenumber Ghost Wavenumber
      7.5 kHz 113  0.000007709  0.2934928  0.000026

       5912.431

       9882.499
           0.000016089

       0.2933447

       0.000055  "  "
           0.000017983

       0.2936933

       0.000061  "  "
           0.000005968  0.2940833  0.000020  5912.447  9882.281
    • Started the co-alignment of the
  • 2013-01-15
    • The modulation efficiency done before ANY adjustments.  ME is calculated according to this calculation, assuming the zero is the same at MOPD as it is for ZPD. Discovered that one of the rails was actually loose when sent to MOPD.  The end limit switch was actually hitting.
    •   LASA PK-PK Zero ME LASB PK-PK Zero ME
      ZPD

      0.200 V

      -7.27 V

       7.47 V
       -9.0 V  0.68

      6.77 V

      1.775 V

      4.995 V 0.037 V  0.59
      MOPD

       -0.777V

      -6.15 V

       5.373 V

       -8.92 V  0.49

       6.07 V

      2.47 V

       3.6 V  0.011 V

       0.42

    • After optimization mirror that feeds laser into spectrometer:
        LASA PK-PK Zero ME LASB PK-PK Zero ME
      ZPD

      0.345 V

      -7.3 V

      7.645 V
       -8.915 V 0.70

      6.892 V

      1.728 V

       5.164V 0.012 V  .60
      MOPD

      0.142 V

      -7.105 V

      7.247 V

       -8.917 V  0.67

      6.777 V

      1.822 V

        4.955 V  0.011 V

       0.58

       
    • After optimization of fixed cube corner
        LASA PK-PK Zero ME LASB PK-PK Zero ME
      ZPD

      0.390 V

      -7.307 V

      7.697 V
       -8.914 V 0.705

      6.915 V

      1.725 V

       5.190V 0.012 V  .602
      MOPD

      0.158 V

      -7.095 V

      7.253 V

       -8.919 V  0.665

      6.766 V

      1.842 V

        4.924 V  0.011 V

       0.574

       
  • 2013-01-08
    • Unpacking and reassembling commenced.

Back to top

December 2012

  • 2012-12-19
    • Trailer on the ground at JPL

 Back to top

July 2012

  • 2012-07-19
    • Barometer T = 28 C; P = 731.8 mm Hg, Zeno P = 968.4 hPa.
    • Ran the ghost test with filter (70-3F-FA) using the 7kHz XPM file dictated by Messershmidt et al.  The GPR's are shown in the table below:
      Scanner Frequency Number of Scans Ghost Amplitude Parent Amplitude GPR Parent Wavenumber Ghost Wavenumber
      7.5 kHz 113  0.00003085  .35745  0.000086306  5905.442  9878.512
    •  1/4 CW, 1/4 CW, 1/4 CW, 1/2 CCW, 1/8 CCW
  • 2012-07-18
    • Norton reported:
      I updated the SunTracker driver to report ST_modus, ST_t_int and
      ST_flip, so we can see whether that works correctly throughout the day
      today. I also made several cosmetic improvements to the display,
      eliminating display of some channels which we do not collect.
      
      The Sun Tracker driver does not currently have the code to wrestle
      control of the Sun Tracker if does not receive data, nor the capability
      of putting it into tracking mode. For the time being, we will have to
      tend to those issues manually if they are a problem, although I don't
      anticipate any trouble implementing those commands either.
      
      Two of the data that I suppressed in the display were ST_off_azi and
      ST_off_ele. As far as I can tell, these are not reported by the new sun
      tracker, so they are not being collected. I also suppressed the
      ST_status word, which we didn't bother to decode in the old sun tracker.
      
      I installed osupdate patches on both caltech_ifs9 and jpl_ifs8 that
      should address the failure of ifsdoit to terminate gracefully. Please
      let me know if you continue to have hangups there under any circumstances.
  • 2012-07-10
    • At the end of the day, tried to take a lamp run with OPUS; FTS and/or OPUS got hung up and had to abort.  Switched to IFSloop to allow lamp runs to be run overnight.
    • First measurement of Mod. eff. with new REO laser

      Laser type                       min        max       dark
      Laser A  amp (V)          -7.675      0.675       -9.0       front
      Laser A  amp (V)           -7.45        0.375      -8.975    back
      Laser B ampl (V)            1.55       7.1375       0.0       front      
      Laser B ampl (V)            1.6875   6.9625       0.025   back 

      After adjusting fixed cube corner, but then discovered the laser was not locked.

      Laser type                   
         min        max       dark
      Laser A  amp (V)          -7.425       2.3           -8.975      front
      Laser B ampl (V)           1.8625      8.6375      0.037      front

      Waited until laser locked and repeated above.

      Laser type                   
         min        max     
      Laser A  amp (V)           -7.625     0.675      front
      Laser A  amp (V)           -7.35       0.375      back
      Laser B ampl (V)            1.5875   7.1625    front      
      Laser B ampl (V)            1.725     6.95        back 

      Adjusted fixed cube corner (looking at the min.) and epeated (end values).

      Laser type                   
         min        max     
      Laser A  amp (V)             -7.65      0.675     front
      Laser B ampl (V)             1.5875   7.1625    front 
      ME (LASA) = 0.76; ME (LASB) = 0.64
    • Replaced original SIOS SL-04 HeNe laser with REO spare.

 Back to top

June 2012

  • 2012-06-21
    • I came into the container around 2:20 pm PDT. Everything looks normal, but it's a bit warm (78F), so I turned down the A/C to 77F. I turned off IFSloop and rebooted the IFS computer. Upon reboot, I heard a clunking noise coming from under the IFS. It appears to be the laser shutter opening and closing. The laser power supply said error, and flipped between showing lock and not when the clunking occurs. I turned off the laser power, waited a few seconds, and turned it back on again.
    • After 5 minutes or so, the laser locked (although I heard a small clunking sound). The IFS came out of error, however, just to be sure, I rebooted it again.
    • Turned on IFSloop and pumped down the IFS to 0.77 hPa (although it was only at 1.47 hPa).
    • The X-windows server for oco-fts wasn't working, so there's no GUI for this machine right now. I will send Susan an email.
    • Within the first solar scan after reinitializing IFSloop, the laser started clunking and is in error again. I ran a savelog and turned off IFSloop. The laser and IFS have been left on; the laser is in error.
  • 2012-06-07
    • Normal day for data acquisition.  Performed these tasks after sunset behind the tree/hill:
      • Removed cardboard cover from the dome position sensor and installed aluminum cover of the same size (5" by 7").
      • Cleaned copper loops with a file.  (Sand paper might have been better.)
      • Added powdered graphite lubricant over the copper loops.  This was selected because graphite is a conductor, but the powder seems to blow easily in the wind.  Hopefully some of this will remain on the contacts.
      • Cleaned CaF2 window with Dust-Off only.  Removed dead wasp that had been on the window for quite some time, on the side that illuminates the suntracker quadrant sensor.
  • 2012-06-06
    • Dome got stuck at home.  Put system in Hold at 13:25 (LT) to manually move the dome, with restart at 13:30 (LT).  No data was lost, but intensity was slightly higher after restart
  • 2012-06-01
    • Dome was facing ENE.  Sent it Home then to a few degrees ahead of the Sun, Azimuth 85 degrees, at 15:23 UT (8:23 LT).  Dome actually undershot to Az of 87 degrees, but that was still in the dead band of the dome driver, with the Sun at Az 83 degrees.

 Back to top

May 2012

  • 2012-05-31
    • Dome got stuck at Home, with AZ delta starting to increase at 19:29 UT.  Manually operated the shutter several times, with incorrect indications of "Away" when the shutter stopped.  Manually moved the dome past Home.  Re-sent it home and opened shutter.  Software re-initialized at 21:34:30 UT (14:34:30 LT).
  • 2012-05-26
    • Set tracker time at 8:16LT. The sky is partly cloudy, so the tracker goes in and out of lock.
  • 2012-05-25
    • Rain in the morning, then overcast.
    • We've had the heater servo working since the insulation is complete, so restored checking of IFS125 temperature in "IFS.tma".  We're now using the "IFS.tma" unchanged from CVS:
      $ diff IFS.tma.CVS IFS.tma    
      632,633c632,633
      <    > Email Error Message IFS125 temperature error.
      <    +3600 Validate Monitor_IFSTemperature;
      ---
      > #   > Email Error Message IFS125 temperature error.
      > #   +3600 Validate Monitor_IFSTemperature;
    • The last remaining differences between the JPL container and the CVS repository are in file "IFSctrl".  This is caused by the IP numbers of the IFS125 and the webcam.  We will have to setup the router to fix this:
      $ grep "10\.10\.0" IFSctrl
      our $server = "10.10.0.1:80";
                  my $jpgresponse = $ua->get( "http://10.10.0.5:8080/netcam.jpg",
    • Updated "slice-ipp.f" to the current CVS version:
      $ diff slice-ipp.f ~/slice-ipp.f
      25c25
      <       parameter (progver=20110509.d0)
      ---
      >       parameter (progver=20080509.d0)
      2361c2361
      <           write(luns,'(3a)')'static const char *',
      ---
      >           write(luns,'(3a)')'static char *',
      $ make -f Makefile.qnx4-f2c
      ...
      $ make -f Makefile.qnx4-f2c distribution
      distribute /d1/home/citco2/ipp slice-ipp
      cp: Copying slice-ipp to /d1/home/citco2/ipp/slice-ipp
    • Verified that IFSloop still builds with the new slice-ipp:
      $ cd ../../TM
      $ cp resistors.h ~/
      $ make clean
      ...
      $ appgen && make
      ... completed with no errors
      $ diff resistors.h ~/resistors.h 
      7c7
      < static const char *InGaAs_R[] = {
      ---
      > static char *InGaAs_R[] = {
      14c14
      < static const char *Si_R[] = {
      ---
      > static char *Si_R[] = {
    • Committed change to support lack of removable drive in the overnight analysis script:
      $ cvs diff -r 1.16 -r 1.17 anal/exam.ksh
      ...
      27,28c27,33
      <    SPEC=/removable/citco2/spectra
      <    SPECTMP=/removable/citco2/spectra.tmp
      ---
      >    if [ -d /removable/citco2 ]; then
      >      SPEC=/removable/citco2/spectra
      >      SPECTMP=/removable/citco2/spectra.tmp
      >    else
      >      SPEC=/data/spectra
      >      SPECTMP=/data/spectra.tmp
      >    fi
  • 2012-05-18
    • Cloudy morning.  Exited IFSloop, turned off laser and IFS125.  Installed a power cord extension for the laser: we can now walk around the instrument without stepping on any cables!
    • Given that we have had no error slices for many days, I was really hoping the laser problems had been fixed by the warmer temperature, and that we would be able to run without laser controller errors.  The IFS125 is now servoed around 26C and the air temperature in the container was 21.5C.  Turned laser on and waited for it to come ready.  Turned IFS125 on.  Unfortunately, this was followed by a long series of loud clicking noises from the laser beam block mechanism, with the laser controller repeatedly losing lock and showing an error status. The laser lock eventually re-established itself. Re-started IFSloop.
  • 2012-05-17
    • Completed all Ethafoam insulation.  The panels are secured by orange sticks which makes the whole assembly look great, but it will be hard to take apart.  Should switch to bungee cords.
  • 2012-05-16
    • Suntracker time was behind by one minute. Set it before leaving home.
    • Added most of the side panels of Ethafoam insulation.
  • 2012-05-15
    • Added Ethafoam insulation on top of instrument.  Paused data recording in the morning to do so.
    • Dome got stuck at home, with azimuth delta starting to increase at 19:20 UT, although data was probably still good for a while after that.  Closed shutter and moved dome manually, with good data re-starting at 20:40 UT (13:40 LT).
  • 2012-05-14
    • Added temperature sensor inside the Zeno enclosure.  Increased heater supply voltage by 2 volts: when one heater circuit is on, the voltage was 11.8 V and is now 13.8 V.  Glued protective cups over the IFS125 temperature sensors.
  • 2012-05-11
    • Completed electrical connections to the IFS125 heaters.  Neatly routed all temperature sensors and heater cables.  With the supply set at 12 V, the total current with all heaters is 14 A.  This keeps the instrument 2 to 3 degrees above air temperature, but we'll need to install the insulation to get the servo to cycle.  I'll also try a higher voltage next week.
  • 2012-05-10
    • Used Citristrip - Safer Paint & Varnish Stripping Gel from Lowes to remove 5"x5" paint squares, two per lid.  Protected perimeter of those squares with glass tape.  This worked very well: better living through chemistry!
    • Drilled and tapped one hole per lid for #2 screw, with depth about half the thickness of the lids.  For most lids, the holes are 3 3/4" from edge and centered in the other dimension.  For the interferometer lid, the hole is 5 3/16" from the edge opposite to the scanner and 10 3/8" from the edge at the detector compartment.  These screws are 7/8" long and will hold the 4-pin CINCH male heater connectors.
    • Bonded all 10 Minco silicone rubber heaters to the lids.  Part number is HR5509R8.9L12B, same as those in Park Falls.  Noted that the Lamont pictures show Kapton heaters instead.
  • 2012-05-09
    • Jess has replaced the "REV" relay on the original JPL dome controller.  Removing the old relay also removed the hole plating annuli.  I had suspected weak hole plating might have been the source of the problem.  We used a new relay, of course.  We also added short sections of bare wire-wrap wires in the holes to replace the plating.  The dome is now running with its original controller and the spare is a spare again.  I've re-trained the dome to be ahead of the Sun by the same amount as we had with the spare controller.
    • I've fixed the connections to the IFS125 temperature sensors.  This turned out to be a bad ribbon cable inside Herc4.  Getting all temp sensors to work is a requisite before we can get the heater servo installed.
  • 2012-05-04
    • Dome didn't stick at home at noon today, perhaps due to the tightening of the wingnuts yesterday.
    • Barometer T = 23.2 C; P = 733.5 mm Hg, Zeno P = 971.9 hPa.

 Back to top

April 2012

  • 2012-04-27
    • Dome got stuck at Home, but caught the problem early enough to avoid losing data.  Waited for end of scan, as indicated by time remaining, then put the system in Hold which closed the dome.  Moved the closed dome manually with no problems.  Returned it Home, opened shutter, then sent it to the Solar azimuth.  Finally, ReInit the software with TimeWarp.
  • 2012-04-21
    • Dome got stuck at Home.  From the ZPD values, it looks like it was casting a shadow starting around 13:10 LT, although it is possible that it caused the tracker to drift from the center of the sun before that.  Went to the container.  Could not command the dome to move.  Closing the shutter worked and allowed the dome to rotate.  Good data re-started at 13:46 LT.
  • 2012-04-20
    • Dome lost knowledge of position, blocking part of the beam at 14:19 LT, and eventually turned North.  Fixed by sending it home then to the sun azimuth, with good data re-starting at 15:26 LT.
  • 2012-04-19
    • Tracker time was 1'20" behind.  Setting the time allowed it to transition from Sun Position to Tracking Mode.  Good data started at 8:33 LT.
    • Channel fringes in O2 A-band reported by the OCO team.  Red filter in front of Si detector is the source of the problem: put container in Hold at 15:10 LT, removed filter, re-started data at 15:26 LT.  (All spectra created from data obtained after this point have been renamed to reflect the "no filter" status.) 
  • 2012-04-10
    • Low values in slices in the morning.  Tracker is in sun position mode, but with low quadrant values: going to track mode has no effect, of course.  Webcam seems to be down too.  Walked up to the container to find the dome shutter closed, despite it showing "Open" on console.  Placed software in Hold which brought the dome back home.  Sent shutter open command: it worked, and the suntracker locked right away.  Did software re-init.  Data started around 8:35 LT.
  • 2012-04-06
    • Again noted that the suntracker was not quite right at 7:20 LT; it was in "Tracking Mode", but the intensities were not equal (i.e., Intensity x1 was an order of magnitude higher than the other three).  Hit "send" while remaining in the "Tracking Mode" and the problem fixed itself.
    • Suntracker time was 57" behind true time.  Set it at 18:20LT.  This should help acquire the sun tomorrow morning.
  • 2012-04-05
    • Suntracker was in "Position Sun" mode and not "Tracking Mode" when checked at 7:20 LT; sent to "Tracking Mode" and suntracker started tracking without problem.
  • 2012-04-03
    • HTML error files generated in the morning; problem fixed by resetting the IFS125, but worry about this detector now that TVAC-2 is underway.
    • Measured barometer pressure P= 732.9, T = 25.5; Herc = 967.7.
    • Put the "spare" detector back into FTS.  Using field stop from solar run (IFS Direct SRC = 104 for lamp), max V (start) = -1.855 V; optimized to max V (final) = -2.188 V.
    • Evacuated and resatrted IFSloop.
  • 2012-04-02
    • SPECTRA taken 120401 and 120402 (prior to changing the resistor) are corrupted.
    • New detector ran fine over the weekend, although the detector was saturated indicating that Bruker had not changed the pre-amp. Jean-Francois added the test point, changed the resistor, and replaced the detector in the FTS.  Max V (start) = -1.879 V; Max V (after optimization) = -2.131 V (These values are using the IFSloop settings for pre-amp gain and field stop.)  Using the gains set by .xpm OPUS file, max V = - 3.451 V.  Much better detector!  (jf20120402lgeeax.001)
    • Checked the centering of the sun on the entrance slit; no change was made.
    • Cleaned all the covers and started to evacuate.  Did another lamp run (jf20120402lgeeax.002)
    • Restarted IFSloop.

 Back to top

March 2012

  • 2012-03-30

    • Vented the instrument; cleaned the rods, although they looked pretty good.

    • Took lamp runs with "spare" InGaAs detector, which is currently in place (jf20120330lgeeax.001).  Max V (on InGaAs) = 0.907 V.  Optimized last flat mirror before exit stop (very touchy) and the mirror in front of the detector; max V = 0.977 V.  Finally, looked at the filament on the entrance slit and discovered that it was far from the geometric center.  Started optimization again there. -2.658 V.  Took lamp run (jf20120330lgeeax.002). 

    • Swapped in "new" detector (On packing: I23428 Sn:IGR336, modified for IRPQ 1556); no test point modifcation. Took lamp run (jf20120330lgeeax.003). 

    • Centered sun on entrance slit (IFS Direct SRC=201 for sun; put paper behind aperture wheel; SRC = 4000 for APT 4mm).  Remember to return to SRC = 1000 and remove paper.

    • Evacuated and took more lamp runs with this detector under vacuum (jf20120330lgeeax.004).  Intensity and lineshape unchanged, so will let the system run over the weekend to see how this detector works.

    • Leveled barometer somewhat; measured pressure P= 734.0, T = 24.0, Herc = 969.5

  • 2012-03-29

    • There were some errors today: an IFS125 soft reset email and "bad magic" messages in the nightly email. Sure enough, the laser error light was on (observed on the morning of 2012-03-30).
  • 2012-03-28

    • Performed general maintenance in preparation for TVAC-2.  Removed and cleaned the CaF2 window and cleaned suntracker optics.  Blew dust from 45 deg turning mirror sending the solar beam into the spectrometer.
    • Retrained the dome to be slightly ahead (6-7 deg) of the sun, thus assuring that the suntracker is not in its shadow. Home is now at 147.82 deg.
    • Put the wide angle lens on the webcam.
    • Installed the cover over the dome's sensing wheel to protect it from sunlight, taped over the hole cut in the side of the suntracker tower mount piece.
    • Jess drilled through the aluminum plates and I-beams to finally fix FTS to floor.
    • The laser and FTS were put back on the UPS.  Laser came up without any error ; we left the IFS125 turned off to allow the laser to become ready without any disturbance.  No error developed when FTS powered on ; we did hear the beamblock release.
    • Pump lines re-connected and checked by running the pump from IFSloop, including opening IFS125 pump valve.
    • Re-cabled and checked the suntracker.
    • Adjusted that A/C air flow to come out horizontally instead of being directed at the instrument (and people). Adjusted the A/C set point from 73F to 78F. IFS125 temperature was about 22C. Laser still had no errors at 16:30 LT.

 Back to top

February 2012

  • 2012-02-28
    • After yesterday's power cycling, the suntracker was left in "position local".  Sent it to "position sun" which caused the quadrant sensor to read high values.  Then placed the system in "track mode" which evened out the quadrant readings.
    • However, the data slices still showed negligible intensity.  Went up to the container.  Confirmed good solar tracking, as viewed on the input tube.  Exited IFSloop. Hard reset the IFS125 and waited for its initialization sequence to complete. Re-started IFSloop.  This resulted in good intensity values in the data slices: AC for the first scan pair, then DC afterwards.
    • Noticed that the laser controller "Error" light is again on.  I wonder if this is caused by the low temperature in the container: the IFS125 reads only 13 C.
  • 2012-02-27
    • Put FTS, suntracker, laser, Netgear network box, webcam, and monitor for the Herc on the UPS.  This cleared the "Error" condition of the laser controller, as it needed to be powered off to transition to the UPS.
    • Attempted to install "returned" InGaAs detector, only to discover that it was the wrong detector and lacked the dual channel acquisition connector.  Back to Bruker!
    • Tested two cable assemblies for IFS125 temperature sensors: they both work. Then determined that the sensors that read out of range are missing a GND connection inside the Herc4. Will fix this after we switch to the new Herc. Routed all temperature sensors and attached them to the IFS125 compartments.
    • The software was in Hold most of the day because of rain. Taken out of hold just before sunset to record the weekly lamp runs. These runs are missing last two info files.
  • 2012-02-17
    • Dome lags behind the sun because its training is about 3 or 4 degrees off. This is made worse by the fact that it also undershoots. Also, the elevation mirror is always crowding that side of the dome, so it would be better if the training error was ahead of the sun. There was some shadow on the elevation mirror, so commanded the dome to move ahead.
    • Tracker had not acquired the sun because its time was about one minute off. Setting the time got the tracker to find the sun and switch to tracking mode (9AM LT). Sun beam is well centered on the input tube.
    • IFS125 started beeping. The Hercules screen showed Scanner Fail, then all subsystems Fail. Exited IFSloop and pressed the IFS125 reset button. Waited for IFS125 to initialize, then re-started IFSloop. Data recording okay, with both detectors in DC mode after the first scan.
    • Noticed that the laser box "Error" light is on. We will cycle the power on it when we switch to the UPS and clean up the AC wiring.
    • Thin high clouds.
  • 2012-02-16
    • Very windy: placed container in Hold to protect dome. The software should have done this on its own, but it seems there is something wrong with the wind speed measurement: it reads zero for long stretches of times while branches are flying by my office window. When it does read non-zero, the values are high (about 5 m/s) but not high enough to trigger the Nap state. I think it would be good to compare our wind measurements with some other weather station at JPL.
    • In addition, there must have been something wrong with the solar tracking, as the MXY were very low all morning.
    • The winds decreased enough in the afternoon to take the container out of Hold. The suntracker regained lock right away, and the sun beam is well centered on the input tube.
    • However, the two detectors are stuck in AC mode, so we still see low MXY, except at ZPD.
  • 2012-02-07
    • Very overcast: placed container in Hold to save pads of interferometer scanner. Taken out of Hold before end-of-day to record lamp runs.
    • Turned on the chiller to see if overheating explains the varying number of lamp runs.
    • Varying number of lamp runs actually caused by missing info files.

 Back to top

January 2012

  • 2012-01-30
    • ~9:40 LT, noted that the dome was only partially open.  Sent DomeOpen command and then sent dome to correct azimuth location.  Went with no problem.
  • 2012-01-29
    • Nightly "Cell" runs taken after 30-minute lamp settling delay. This time, no error messages in the citco2.log.
    • Made the following changes to IFS.tma:
      • Norton changed the cell run day selection to use local date rather than UT date. This should result in another set of "Cell" runs tomorrow. This week, we get two Mondays!
      • Lowered the lamp settling time from 30 minutes to just 1 minute. This should allow us to run the same software on site with no chiller and with a working lamp temperature limiter.
      • Enabled air temperature alarm since the sensor was added. We're just 2 lines away from CVS now: the IFS125 temperature emails are still commented out since the instrument temperature is not yet regulated (waiting on Minco heater installation).
  • 2012-01-25
    • TVac measurements started at sunrise. (FTS in the shade by 15:58 LT.)
  • 2012-01-23
    • Added a 30-minute delay between lamp turn-on and first scan of cell run. The delay worked for "Cellres02" on this Monday night, but the log contained messages like these through the cell scans:
      01:39:33: memo: TMA: DEBUG: Entering State IFS_Status_Failing
      01:39:33: memo: TMA: ERROR: IFS Read Status is responding slowly
  • 2012-01-10
    • Changes to "IFS.tma": With these changes, the JPL "IFS.tma" is now only 4 lines away from CVS: we still have the IFS125 and Air temperature emails commented out.
      • Fix the "SRC=0" command at end of cell runs.
      • Turn off the lamp in case of failure.
      • Re-enable failure on suntracker stale counter.
      • Re-enable error email if heater power is lost.
  • 2012-01-04
    • Connected dome power to the Herc. Did this after the sun set behind building 245, but before the official sunset, so the system went into play state. Dome opened correctly, then moved to AZ 238.8 degrees when commended by IFSloop to go to 243 degrees. We can probably reduce this undershoot by adjusting the potentiometer on the dome controller. Note that one must now issue "Power Dome On" command when starting the system from the all off state.
  • 2012-01-03
    • Swapped out dome controller board and replaced with the spare (14:56 LT); re-trained the dome
    • Connected the batteries to the new Herc power supply; Jess needs to file down the connectors to dome controller and that will also be connected to new power supply.

 Back to top

December 2011

  • 2011-12-30
    • Same dome problem as yesterday. Same solution.
  • 2011-12-29
    • Unrelated to the dome problem, we received an error email with message: "IFS125 failure. Indicated by failure to recover after software reset." The IFS125 stale counter was indeed increasing on the console. Re-starting IFSloop seemed to clear the problem.
    • Dome failed to close again at night. This time, the dome would not go to the home position, as it would not turn counter-clockwise. I ended up moving it clockwise in small increments until it could continue to home in the clockwise direction. Then I closed the shutter and re-started the software. As we suspected during our latest dome checks, we indeed have a faulty reverse direction relay on the dome controller circuit board.
  • 2011-12-27
    • Dome failed to close at night. It was left at the sunset azimuth. Sent "Dome Home" and "Dome CloseShutter" which seemed to work. Then did a "Savelog" and "Software Shutdown Quickly".
  • 2011-12-26
    • Solar azimuth and elevation calculated by Herc were 153 degrees and 27 degrees. Values calculated by the tracker were 140 degrees and 20 degrees. Time on Herc was 18:12:16 UTC, but on the tracker it showed 17:10:18 (falsely labeled as "local time"). Set the time on the tracker (which is a command well hidden in the hardware configuration menu). This left the tracker in "Position Sun" mode with non-zero values on the quadrant sensor. Then set the tracker in "Tracking Mode" which resulted in balanced quadrant sensor. Good tracking starts at 18:17:11 UTC (10:17:14 PST), sun acquired during slice b449099.0, with first good scan at slice b449103.0, a reverse run.
    • Solar tracker not in tracking mode and unable to find the sun from home.  Will try to get into JPL today, but not sure if I can make it.
    • Also, cameras both here and at Park Falls needed rebooting.  This issue is still a mystery.
  • 2011-12-16
    • Container in Hold for most of the day due to high winds. Weather improved enough in mid-afternoon to take data.
    • Installed new Herc main power supply which also includes the heater supply and the battery charger.
    • Dome was perfectly aligned with the sun around 14:45.
    • Cleared a spider web in the tracker column. Dust is present over the CaF2 window, blown by the wind.
  • 2011-12-11
    • The lamp is clearly not shutting off after lamp runs and Src_T=55 deg C at 9:55 p.m. LT (Status=Idle).  Typed IFSDirectSRC=0 and watched lamp temp. go down.
  • 2011-12-09
    • Returned to JPL to determine exactly when the palm tree was casting a shadow on the suntracker; I would say if we avoid taking measurements between 10:47 - 11:13 a.m., we should be fine.
    • Also found that the dome was casting a shadow on the suntracker, which by 13:30 LT was apparent inside the trailer.  Hence, I decide to re-train the dome again using the compass this time - aligned shutter opening toward N using the DomeGoTo command and IFSloop in Hold, exited IFSloop, and started "qtalk -m /dev/ser2".  Typed "<Ctrl>-A", then "e" to disable Local Echo, and finally "GTRN".  The dome stopped after a complete CW rotation upon the first attempt. (We saw this yesterday as well.)  The hand-held controller did not work in either direction, so I believe the dome was confused.  Repeated the "<Ctrl>-A" and the re-training seemed to work.  The dome no longer blocked the suntracker, but must check again in the morning.
  • 2011-12-08
    • Took lamp runs in order to compare spectra obtained with this SPARE InGaAs detector to those obtained with the original detector.  Intensity is down to about 1/3 of its value in July on both detectors; could this indicate that the alignment of the lamp has changed?
  • 2011-12-07
    • Cleaned the suntracker optics and CaF2 window; both the elevation mirror and window were very dirty.
    • Attempted to clean the 45 deg turning mirror sending the solar beam into the spectrometer.  Noted that the gold coated was flaking off, so stopped cleaning!
    • Adjusted the quadrant sensor screws (with the suntracker tracking and the cover of the source compartment balancing on the compartment) until the beam position was concentric with the plastic tube entrance.  
    • With the APT = 4000 and an index paper blocking the beam into the FTS chamber, the beam was adjusted (with the 2nd turning mirror, the OAP) on the entrance field stop.
    • The small flat mirror in the interferometer compartment was adjusted until the intensity was optimized on the InGaAs detector.  This was only a very small gain in detector output voltage.
    • Finally, the two mirrors before the two detectors were adjusted for optimal signal (with gains on both detectors; -3.776 V for InGaAs and -1.141V for Si).
    • Upon pumping down, we still needed to adjust the Offset Position of the suntracker slightly: -19.20 ; 124.05.
    • Then, tried to retrain the dome to eliminate the problem of shadows on the mirrors.  In the process, we uncovered a problem with the reverse (Azimuth) direction.  J-F deduced the problem to be either due a bad relay or solder connection and applied solder.  Worked upon turning on, but do not know if this fixed the problem.
    • Noticed that a palm tree casts a shadow on the suntracker around 11 AM LT.
  • 2011-12-05
    • Discovered Src_T = 58 C at around 13:00 LT; sent SRC=0 command and temp dropped.
  • 2011-12-04
    • Reinitialed FTS and commenced measurements.
  • 2011-12-01
    • Put the container in Hold due to the high winds.

 Back to top

November 2011

  • 2011-11-30
    • Dome azimuth followed the solar tracker this morning, but shuttter had not opened.  When noticed, dome sent Home, shutter opened, and dome returned to correct azimuth position.  Also needed to send sun tracker to Sun Position before it would track in Tracking mode.
  • 2011-11-29
    • FTS was reinitialized and measurements commenced.
  • 2011-11-20
    • FTS put in "Hold" due to construction near the trailer.
  • 2011-11-08
    • Discovered that the dual-acquisition cable was not connected to the InGaAs detector and hence the Si spectra has not been recorded properly since 11/02/11.
  • 2011-11-07
    • IFS125 failure. Indicated by failure to recover after software reset. Con1 showed failure, but all IFS Diagnostics showed OK.  Tried SoftwareShutdownInstantly, but now all IFS Diagnostics in Fail.  Tried soft reset of IFS (IFSresetSW, exited with SoftwareShutdownInstantly, and restarted IFSloop after a couple of minutes) just as I was doing when the short slices were being created and this worked.  Indeed the short slices were being created ... ?
  • 2011-11-02
    • Noted that dome is blocking the solar beam when it follows the tpg_azi.  Switching the flip state from left to right helped, but probably need to retrain the dome.
    • Data collection started at ~13:07 LT.
    • While the instrument was vented, looked at the beam on the entrance field stop.  It had drifted a bit since the previous alignment, so re-centered it.  Also, maximized the light into the InGaAs detector, first by lowering the detector in its mount and then tweeking the last optic before the detector.  Max V (on InGaAs) = 0.107 V; (on Si) = 0.060 V.
    • Jesse swapped the covers on the original and the spare (123428, SN:IRG 64, ECL02, RT-InGaAs, JPK) InGaAs detectors, so that we could return the faulty detector to Bruker.  The spare detector was reinstalled in the FTS.

 Back to top

October 2011

  • 2011-10-27
    • The FTS ran flawlessly without the InGaAs detector for a week, so decided to install the spare InGaAs detector (#64,dual acquisition, DC enabled, with resistors) to try and determine whether the detector is at fault or the InGaAs signal chain / cabling going to it are the cause of the problem. So, exited IFSloop, vented instrument, installed the spare detector in the original InGaAs detector position.  Then moved:
      IFS.tma to IFS.tma.Si
      IFS.tma.20111013 to IFS.tma
      IFSopts.pm to IFSopts.pm.Si
      IFSopts.pm.20111013 to IFSopts.pm
      then typed "make clean" <rtn>, "make" <rtn>, and "make distribution" <rtn> in /TM directory.  Finally, evacuated the instrument and restarted IFSloop.  Data collection started at 13:06 LT.  No signal alignment check was made and it is very possible that this is off.
  • 2011-10-23
    • No erroneous slices yet with the InGaAs detector removed.  The NIR source still does not shut off after the lamp runs though.  Found Src_T = 60 C at ~ 21:45 LT and sent SRC=0. Src_T started to decrease soon after that.
  • 2011-10-21
    • Baron suggested replacing the Si and now removing the InGaAs detector.  So, exited IFSloop, vented instrument, replaced Si detector in its original position and removed the InGaAs detector.  Then moved:
      IFS.tma to IFS.tma.InGaAs
      IFS.tma.Si to IFS.tma
      IFSopts.pm to IFSopts.pm.InGaAs
      IFSopts.pm.Si to IFSopts.pm
      then typed "make clean" <rtn>, "make" <rtn>, and "make distribution" <rtn>.  Finally, evacuated the instrument and restarted IFSloop.  Data collection started at 11:24 LT.
    • FTS started up and ran fine until 10:30 LT and then failed.
  • 2011-10-20
    • Was in error first thing in the morning and reset ~8:45 LT.  The "detectors" were in Fail again after running the overnight analysis, so I reset the instrument and restarted IFSloop before goign to bed.
  • 2011-10-19
    • Error remained until soft reset of IFS125 and restart of IFSloop at 11:06 LT. 
    • Was in error first thing in the morning and reset ~9:00 LT.
  • 2011-10-17
    • Was in error first thing in the morning; reset the instrument at 9:04 LT and it ran fine all day
  • 2011-10-16
    • Was in error first thing in the morning; reset the instrument at 7:45 LT and it ran fine until 17:55 LT (was not discovered until after sunset.)
  • 2011-10-15
    • Discovered the Detectors in FAIL twice during the day (~9:42 and 15:03 LT).  The instrument was in Sleep and was not reset.
  • 2011-10-14
    • There was a problem with the test software. It had not built completely due to a missing coma in theIFSopts.pm file.  JF fixed this and IFSloop was restarted.  The FTS ran fine the rest of the day.
  • 2011-10-13
    • At JPL to try Baron's test - first, exited IFSloop, vented instrument, and removed Si Detector.  Then moved:
      IFS.tma to IFS.tma.20111013
      IFS.tma.InGaAs to IFS.tma
      IFSopts.pm to IFSopts.pm.20111013
      IFSopts.pm.InGaAs to IFSopts
      then typed "make clean" <rtn>, "make" <rtn>, and "make distribution" <rtn>.  Finally, evacuated the instrument and restarted IFSloop.
    • When IFSloop restarted, it instantly began with the short scans.  However, upon reset the software and restarting IFSloop as we usually do, it started collecting solar scans as normal. 
    • Noted ~13:15 LT, that the dome had not rotating beyond home.  Attempted unsuccessfully to rotate it CW.  Eventually, after closing and reopening the shutter, I could rotate the dome.  It is rotating on its own again.
  • 2011-10-11
    • Software reset first thing in the morning to fix HTML error slices.
  • 2011-10-10
    • Reinitiated data collection today.
  • 2011-10-07
    • Planned to restart data collection, had trouble getting the suntracker to align today from home, so came to JPL.  Construction was halted due to the rain, but is in full swing today so will close things down again.
  • 2011-10-04
    • IFSloop halted (~22:30 LT, 2011-10-03) and hence dome is closed to prevent dust from the nearby construction project from getting onto the suntracker.
  • 2011-10-03
    • Noted at 8:00 LT that the software needed to be reset to fix HTML error slices.  Also found the Src_T= 59.0 C.  Apparently the lamp did not shut off after the lamp runs last night and also did not overheat and trip the shut-off.  Sent SRC=0  ~8:10  LT. Src_T started to decrease soon after that.  (Also came to the trailer to check that all was well.  Workers are building a power station into the hillside ~30 feet to the north-east of the trailer.  Lunch boxes and coolers are sitting on the rolling ladder; wooden crate for suntracker over turned and pushed into the fence.)
  • 2011-10-02
    • Restarted IFSloop at 16:41 LT.

 Back to top

September 2011

  • 2011-09-30
    • FTS put into hold and later IFSloop stopped for the weekend.
  • 2011-09-29
    • The suntracker did not appear well balanced across the four sensors this morning (as noted by the intensities).  Adjusted the time again; this seems to have helped a bit.
  • 2011-09-27
    • Software reset first thing in the morning to fix HTML error slices.
  • 2011-09-26
    • Software reset first thing in the morning to fix HTML error slices.
  • 2011-09-25
    • Src_T was at 52.0 C and constant. Sent SRC=0 at 19:45 LT. Src_T started to decrease soon after that.
  • 2011-09-23
    • HTML error slices started being produced at 16:08 LT, but went undetected until 2011-09-25 ~15:45 LT when software reset.
  • 2011-09-20
    • Software reset at first thing in the morning to fix HTML error slices.
  • 2011-09-19
    • Around 14:30 LT. discovered that the azimuth seemed stuck at 151 deg.  After closing the dome and reopening, was able to advance the azimuth.
  • 2011-09-18
    • Software reset at first thing in the morning to fix HTML error slices.
    • Checked on container after "nightime analysis" email. Src_T was at 52.0 C and constant. Sent SRC=0 at 20:03 LT. Src_T started to decrease soon after that.
  • 2011-09-17
    • Software reset at first thing in the morning to fix HTML error slices.
  • 2011-09-16
    • Software reset at first thing in the morning to fix HTML error slices.
  • 2011-09-15
    • Baron checked the cables to the detectors and re-seated the detectors while OPUS was running.  He did produce an error when jiggling the Si detector (rechecked on 9/21).  After re-seating again, the error could not be duplicated.  He replaced the step-up transformer and we restarted IFSloop.
  • 2011-09-11
    • Lamp appeared to be on, with temperature of 50 C. Sent SRC=0 command at 19:50 LT. "Src_T" started dropping right after that.  (IFSloop has initiated nightime analysis e-mail sent at 19:20 LT and FTS in sleep mode.)
  • 2011-09-07
    • Software reset at 16:01 LT to fix HTML error slices.
      First slice of the day taken at: 06:40
      HTML error slice starts at: 12:26
    • The suntracker seems to be misaligned; the quadrant intensities are too small and not even.  Resetting the time seemed to help.
  • 2011-09-06
    • ~9:30 a.m., discovered that dome had not opened this morning.  I opened it manually, set tracker to Sun Position and then Tracking Mode.  Hope all is now well.
  • 2011-09-04
    • Lamp appeared to be on, with temperature of 56 C. Sent SRC=0 command at 21:00 LT. "Src_T" started dropping right after that.
  • 2011-09-03
    • Had to do another software reset this morning; this is not looking good.
  • 2011-09-02
    • The FTS collected until 16:53 LT.  An e-mail came out at 17:00 LT that "IFS125 has been reset in software", but console 1 showed that the IFS Diagnotics' parameters were all in FAIL.  I did the software reset as described on 8/23 and the FTS is collecting again.
    • Decided to rethink the OPUS test and return to solar spectra collection in the meantime.  However, upon shutting down the laptop and typing "IFSloop" (in the Herc directory ../citco2/), the Herc froze and the cursor just blinked.  I had to do a hard power cycling of the Herc before I could proceed.  After restarting, all systems were in "FAIL" in the IFSloop Diagnostics, so I did a hardware reset.  The IFS then started producing the HTML error slices, leading to another hardware reset.  Finally (at 10:20 a.m.), all is well.

 Back to top

August 2011

  • 2011-08-30/31
    • Came in today (8/31) to find that OPUS had stopped collecting after the 12:42 a.m. measurement, which was 11 out of 24.  There was a small (OPUS) window open, stating Instrument Error and nothing more.  When I clicked OK, the message disappeared and no other errors were indicated.  Hence, I saved the Full Report from the webpage for Baron, unloaded the spectra shown, checked the signal, and started a new 24 spectra sequence.  OPUS immediately showed SAMPLE in green at the bottom, which is indicative of data processing.  Only one detector was recorded and I think this was the analysis of last data collected at or around 1:42 a.m.  (It was a flat line, which makes sense because there was no light.)  Then it started with the first measurement of the second sequence.  4 scans were to be collected.  After some time, I looked at the monitor and noticed it was written: 3 scans/1 "ignored".  (Ignored was not the word it used, but whatever it was indicated that there was a problem with 1 scan.)  Baron has been notified.
    • Shut off IFSloop to do repeated measurements running OPUS.  Will see if it is possible to simulate the HTML error slices.  Starting with a repeat cycle of 1 per hour for 24 hours.  I fear this is a matter of luck.
  • 2011-08-29
    • 8:25 LT noticed lamp was on (temp of 64 C) -> Hold -> SRC=0 -> re-started IFSloop at 8:32 LT
  • 2011-08-26
    • Software reset at 6:50 LT to fix HTML error slices.
  • 2011-08-24
    • Software reset at 6:50 LT to fix HTML error slices.
  • 2011-08-23
    • The system was producing HTML slices this morning. I re-started it at 8:30 LT. I did go to the container, mainly to check on the tracker alignment which was offset a bit towards the North-West. For the actual restart, I could have done it remotely:
        1) Under IFSloop, perform a soft reset of the IFS125.
        2) Do "software shutdown instantly"  (not "quickly").
        3) Wait for the soft reset to complete, e.g. a few minutes.
        4) Re-start IFSloop.
      No need to press the IFS125 reset button !! Note that, exactly as yesterday, the very first scan pair is in AC mode with the lower gain, but the second scan pair and all subsequent ones are in DC mode.
  • 2011-08-22
    • Re-started the IFS125 just before 2 PM LT.  The suntracker beam is no
      longer perfectly centered on the input tube: there is more light on the
      North side than on the South side.  This is also true with regards to
      the mark I made on the floor of the container.  I think this means an
      adjustment drifted on the path to the quadrant sensor.  If the quadrant
      sensor remained centered on the entrance field stop, we're okay, but
      we should probably check it.

      Now, for the really interesting part...  I checked the MNY/MXY value
      after the IFS125 reset, and those indicated that the pre-amp was stuck
      in AC mode.  Second reset, with same results.  Power cycle, idem.

      So I sat down and wrote an email to Norton for him to check on that.
      Just to be sure, I re-checked the MNY/MXY and they were okay at that
      time !!!

      Upon closer examination, the second run is in DC mode, as if the
      commands got to the instrument too late to take effect on the first
      scan.  So this has nothing to do with using OPUS to switch to DC:
      IFSloop does it on its own, but we have to wait for the second scan.

      This also means that I now have a way of getting out of the HTML
      slice problem remotely: just issue a software reset which we have
      seen places us in AC mode, then let IFSloop set the detector mode
      again (we may have to exit IFSloop for that to take place).

      First DC-mode slice for today was b129946.0 at 14:08 LT.
       
  • 2011-08-21
    • IFS125 is producing the HTML error slices. Placed the system in Hold.
  • 2011-08-18
    • Taped a piece of plastified paper to cover the dome sensing wheel and protect it from the sun.
    • Re-enabled dome movements in "IFS.tma". The dome is now following the solar position computed by the stub suntracker driver. WOOHOO!
    • Sun tracker went into tracking mode on its own, and is well centered. Values of MXY and MNY show DC mode and proper gains. Good solar scans started at 8:31 LT. The system should now do this on its own in the morning. Shall we call this "fully automated" ?
  • 2011-08-17
    • Reset IFS125.
    • Did a Savelog after many days of Hold. Had to subsequently slay memo and rdr, of course.
    • Re-started IFSloop to test stub suntracker driver.
    • Moved dome to az of 265. Tracker immediately found the sun, and is nicely centered! Slices indicate we're in DC mode and with proper gains. Good data started at 14:27 LT.
  • 2011-08-13
    • The instrument started collecting short (HTML) slices again this morning and was put into hold.
  • 2011-08-10
    • Found that the short slices were being collected first thing in the morning; tried restarting IFSloop, but did not work.  Without doing a hardware reset, switched to the Opus program on the laptop.  There is an error under "measurements" "optic" tabs.  The parameter Detector setting is red and the RT-Si DiodeDC+RT-InGaAs DC option does not show up in the list.  After a hardware reset, it appears fine.  Wrote to Baron with summary of the last ten days.
  • 2011-08-06
    • Logged in ~ 2p.m. to find that the short slices were being collected; this seemed to occur at 1:32 p.m.  Restarting IFSloop had no effect.  Went to JPL and did a hardward reset; as before, the slice size increased, but detectors in AC with wrong gain:
      OpusHdr b88840.0 | egrep "MXY|MNY"
        MXY1: 0.0705589801073074
        MNY1: -0.0795358046889305
        MXY2: 0.279942959547043
        MNY2: -0.402487397193909
      Switched to the laptop running Opus; loaded a solar scan experimental file, checked signal, and all is well (running DC, proper gains correct, and spectrum looked normal).  So, switched back to the Herc and started collecting as usual.
  • 2011-08-04
    • Logged in from home and got the dome in the right orientation this morning, but did not have time to check the slices before leaving home.  Now see that it is collecting short slices.  Reinitializing the software and restarting IFSloop had no effect.  Once in the trailer, manually reset the FTS; this corrected the problem.  Now see the larger slices with the appropriate magnitudes and signs.
  • 2011-08-02
    • FTS was producing short slices again this morning, which was fixed by doing a soft reset of the IFS125.  However, upon looking at the slices we found that the detectors were in AC mode and the gain was not set properly.  We attempted a hardware reset and power cycling, but to no avail.  Then, we switched FTS control from the Herc to the Opus software on the laptop.  This worked - saw dc signal with the correct intensity.  We switched back to IFSloop and the system still worked.
  • 2011-08-01
    • After a weekend power outage and resetting of computers, starting collecting data again ~4 p.m.

 Back to top

July 2011

  • 2011-07-29
    • Jean found a new Western Digital disk, identical to the one that was in the Herc and installed it; He reset the IFS125 which had been "Absent" all those days and the IFSloop started right up. He re-set the time on the tracker (it > was pretty close already). Re-init the tracker. Rotated the dome. Asked the tracker to position Sun, which it did, close enough to then go to tracking mode.  Good solar tracking was achieved at 17:22 LT.
       
  • 2011-07-20
    • Aligned the FTS under the suntracker
      - Looked at the beam on the entrance field stop and confimed that it was, indeed, not centered.
      - Removed (and cleaned ) the CaF2 window and tube holding the "3rd 45 deg suntracker mirror". 
      - Checked level of the suntracker base; no adjustment was needed.
      - Placed a leveling plate (with 3 point leveling), on the floor of the trailer and directly under the suntracker, to provide a level surface on which to set a planar mirror.  Taped a centering square with a ~1" hole (or incoming solar beam) and a larger circle marked by tape (for the reflected solar beam) on the inside of the jig plate which is mounted on the roof.  Set time in suntracker control screen. Then adjusted the Offset Adjust Angles until the reflected beam was centered on the tape circle.  Final position values: -18.65, 124.20.  Now know that the solar beam is vertical.
      - Replaced the CaF2 window and remounted the tube holding the "3rd 45 deg suntracker mirror" (also cleaned).
      - Pushed the FTS until it was directly below the incoming beam.
      - Did some slight adjusting of the  "3rd 45 deg suntracker mirror" to center the beam in the tube.
      - Started the sun tracker tracking, and adjusted the quadrant sensor screws until the beam position was concentric with the plastic tube entrance.  (This tube was also leveled.)  NOTE: it was necessary to balance the cover on the source compartment while adjusting the quandrant sensor screws.
      - The beam was well centered on the 1st turning mirror (large flat mirror) in the source chamber; the beam was centered on the 2nd turning mirror (OAP) in the source chamber with slight adjustments on the 1st mirror.
      - With the APT = 4000 and an index paper blocking the beam into the FTS chamber, the beam was adjusted (with the 2nd turning mirror, the OAP) on the entrance field stop.
    • Tracking and alignment still look good 2.5 hours later; will try turning on Cloud Detection.
    • Closed dome at 17:15 LT. Tracker was still working, with solar beam perfectly centered on input tube.
  • 2011-07-11
    • Norton compiled the suntracker stub code to compute the solar azimuth that will temporarily drive the dome.
    • Updated the QNX4 boot file, adding the -H option to Fsys to avoid heap exhaustion which I think is the source of the "Too many open files in system" error. This seems to be confirmed by the tracelog that were recorded in June.
    • Re-enabled the tracelogger. Looked fine for a while, but...
    • Notice read errors on drive hd1. Tried power cycling the Herc, but the drive does not spin properly and can no longer be mounted. Arrgh!
  • 2011-07-08
    • Updates to "ifsreduce", "/etc/hosts", "/etc/ssh/ssh_prng_cmds". See the respective CVS logs for details.
    • Checked into CVS files that were setup earlier: "/etc/ntp.conf", "/etc/profile", "/etc/config/bin/tcpip.7", and "/etc/default/login".

 Back to top

June 2011

  • 2011-06-21
    • Turned off IFSloop.
  • 2011-06-20
    - FTS not Absent tonight???  (Did someone restart IFSloop? Yeah, I did. Also make a minor change to ifsreduce to improve the presentation of the NTP sync check.)
    - Dome blocked the sun at some point during lunch/errands and therefore the suntracker oculd not track.  Re-positioned and started taking good data after 3 p.m. (?exact time?).
    - Spent a good hour trying to get the suntracker to track this morning.  (The problem is a slow connection from home coupled with the frustrating reload issue with the Java script.)  Also learned that with the CDT on, I cannot go directly to Tracking Mode.  I had to turn off the CDT, go to Position Sun, and then to the Tracking Mode.  Unfortunately, the Adjust Offset also  had to be adjusted.  This is difficult to do remotely, but I have not option today.  It seems to be tracking now finally at ~9:30 a.m.
    - Logged in at 2:35 AM (late morning here in Belgium :). Found Src_T to be 63.0 C. Used IFSdirect command menu to send "SRC=0". Source temperature now dropping.
  • 2011-06-19
    - Cloudy again.  IFSloop was in failure and needed restarting; not sure why.
    - Finally the sun came out.  I had a lot of trouble with the oco-fts running the suntracker Java script today.  My computer locked up a number of times and I had to do a Force Quit to exit Firefox and restart.  Eventually got to the suntracker.  Did not see the sun and had to Adjust the Offset.  Turned the CDT back on; started solar spectra ~3p.m.
  • 2011-06-18
    - Cloudy again in the morning, so did not attempt to login early.  When the sun appeared, however, I was unable to access the suntracker.  The Java script continually refreshed and I was unable to make any changes.  Also, discovered that if StatusShutdownInstantly is not run at night, the FTS will not pump down in the morning.
    - Went to JPL at noon; had to restart oco-fts in order to open Firefox and hence, the suntracker webpage.  Started collecting solar scans ~12:15 p.m.
  • 2011-06-17
    - System ready to go, but it is raining!  Also, Baron is coming today to swap out the outer encoder.
    - Baron swapped out the encoder; FTS running under IFSloop now (12:00 p.m.). 
    - Also turned Cloud Detector <on>.  Cloud Detector ON effects: 1) Auto Mode Switching from TTM to PREVIOUS MODE, if Total Intensity falls below MMT <min>, 2) Auto Mode Switching from PREVIOUS MODE to TTM, if Total Intensity rises above MMT <max>, and 3) One cloud event per second is counted, if Total Intensity falls below the Cloud Detector Threshold.
    - After lunch ran some tests with OPUS.  Started data collection with IFSloop at 3:30 p.m., noted at 4:30 p.m. that NIR source was left on.  Turned off with web browser, without turning off IFSloop (no problem.)  When scan finished, heard source compartment mirror move.
    - Baron is going to send a new step-up transformer to replace the current humming one.  Not sure if this is a problem or not.
  • 2011-06-16
    - There was no sun until about 2:30 p.m., so did not bother with the suntracker or dome until then.  Restarted IFSloop after the nighttime analysis.
  • 2011-06-15
    - The FTS had been in play since 8:11 a.m. without apparent error, but without much sun.  Advanced dome and went about installing the camera on the roof.  Started collecting images at 9:10 a.m.
    - Accidentally exited from IFSloop at 11:24 a.m. and had to restart.
    - Did StatusShutdownInstantly and restarted IFSloop after nighttime analysis was run.
  • 2011-06-14
    - As usual, FTS Absent, with all subsystems in Fail this morning.  Did StatusShutdownInstantly to re-start software.  Suntracker is tracking; dome positioned to 81 deg.
    - Arrived at the trailer ~11:30 a.m., instrument was in play.  When returned after lunch, 1:35 p.m., the suntracker had lost the sun.  Strange - the skies are clear and the dome was not blocking the light.  Sent to Sun Position and continued collection.
    - Returned to JPL @7:15 p.m. to set up camera and noted that 1) dome was still open and FTS seemed to be taking spectra, but 2) there was no data saved today!  This is perhaps due to the hercules hd being full (need to figure out howto check this).  Data is transferred to oco-fts and ccycle, so will delete files on the herc and try restarting IFSloop.  As usual, FTS comes up Absent and all subsystems are in failure.  Did StatusShutdownInstantly.  Have .jpg images now!!!
    - Also noted that the sprinker system below the trailer sprays the AC compressor, directly on the electrical lines.  Must do something about this.
    - Odd login error - only happened once.
    dwunch@10.10.0.4's password:
    cannot open $() input: Too many open files in system
    stty: Too many open files in system
    umask: Too many open files in system
  • 2011-06-13
    - Discovered that Firefox browser was open, although not visible and was unable to access suntracker.  Needed to restart oco-fts.
    - Advanced dome to 233 deg (well beyond the 121 deg of suntracker); collection began ~11:40 a.m.
    - 12:20 p.m. (LT), Pressure reading on barometer = 731.9 mm Hg @ T = 24.7 C (or  mm Hg @ 0 deg or  hPa); Setra  = 969.3 hPa.
  • 2011-06-11/12
    - Overcast both mornings.  Also found the FTS Absent, with all subsystems in Fail both mornings.  Probably no good data on 06/11, late afternoon data may be ok. on 06/12.
  • 2011-06-10
    - Cloudy skies again today, so no rush to check on the FTS.  At ~11:30 a.m. found it was Absent, with all subsystems in Fail.  Did a StatusShutdownQuickly, restarted IFSloop, and put into hold.
    - Started data collection ~2 p.m.
  • 2011-06-09
    - IFS started up fine this morning, but again it is completely overcast.
  • 2011-06-08
    - Completely overcast this morning.
    - At ~ 9:55 a.m., the instrument started creating short slices; did a software statusshutdownquickly, retreived the Full Report from the internet Browser page, and re-started IFSloop.  This did not resolve the problem, but given the clouds I am in no rush to get to JPL to do a reset.
    - Manually reset the FTS and began collecting spectra at 15:31 (although there is no sun!)
    - 5:15 p.m. (LT), Pressure reading on barometer = 731.2 mm Hg (727.74 mm Hg @ 0 deg or  970.24 hPa) Setra  = 968.7 hPa, T = 22.8 C.)
    - 10:40 p.m. FTS "Absent" and all subsystems in fail.  Did a Status Shutdown Instantly; now in "Idle" sleep.
  • 2011-06-07
    - FTS started up, suntracker tracking, dome at 77 deg; measurements started at 6:56 a.m. (LT)
    - Around noon, instrument started producing short slices; Stale counter is incrementing.  Not possible to reset from Caltech.
    - After three manual resets, IFS is operating again @4:16 p.m (LT). :0
    - 4:30 p.m. (LT), Pressure reading on barometer = 730.00 mm Hg (726.31 mm Hg @ 0 deg or  968.33 hPa; Setra  = 966.9 hPa, T = 24.8 C.
    - Discovered that the FTS was Absent after running the overnight analysis; Restarted IFSloop and seems to be o.k.
  • 2011-06-06
    - Cloudy this morning and instrument is creating short slices; Stale counter is incrementing.  Did a software reinit, then shutdown quickly, followed by IFSloop restart.  System went into Solar scans, but with same problem.  Must go to JPL for a manual reset.
    - Needed to reset the IFS125 twice before it worked properly.  Set suntracker to Position Sun and then Tracking mode; still thick clouds.  Adjusted dome position to 105 deg. at 9:55 a.m; still very overcast, but will try.
    - E-mailed Baron about Cloud Detection and intermittent instrument problem; he is looking into the cloud issue and asked for the Full Report to trouble shoot on the instrument problem.
    - Connected camera to serial port - com4 - of hercules and into Netgear.  Cannot see the image at http://10.10.0.5; must investigate further.
    - 1:10 p.m. (LT), Pressure reading on barometer = 733.70 mm Hg (729.99 mm Hg @ 0 deg or 973.24 hPa; Setra  = 971.5 hPa, T = 24.8 C.
  • 2011-06-05
    - Instrument again ready to go, but there is no sun yet (7:50 a.m. LT).
    - Patchy clouds, but suntracker is tracking; advanced dome to 103 deg. (9:47 a.m. LT).  Again to 120 deg at 11:16 a.m.  Again to 268 deg at 2:50 p.m.
    - Discovered that the IFS was Absent after running overnight analysis, so did a StatusShutdownInstantly and restarted IFSloop.
    - Also, Mother Nature is providing test data for the rain sensor - Rain sensor = 1; "Lightning" sensor = 90?
  • 2011-06-04
    - Instrument ready to go, but overcast skies (7:38 a.m. LT).
    - In "IFSctrl" changed webcam IP number from 192.168.1.107 to 10.10.0.5. Stopped IFSloop, built and installed new version, re-started IFSloop.
    - Re-centered suntracker on the sun at 10:50 LT. Moved dome to az 107 degrees.
    - Re-centered suntracker on the sun at 1:30 p.m LT.  Moved dome to Az. 178 deg.
    - At 3:30 p.m., moved dome to Az ~270 deg.
  • 2011-06-03
    - Set the suntracker to position sun, moved the dome to 88 degrees, set tracker in tracking mode. Solar acquisition started at 8:27 LT.
    - 10:00 a.m. (LT), Pressure reading on barometer = 730.20 mm Hg (726.86 mm Hg @ 0 deg or 969.07 hPa; Setra  = 967.3 hPa, T = 21.8 C.
    - At 13:00 LT noticed low MNY and MXY. Suntracker webpage showed spinning Java logo, then no connection error on retry. Went to container and pressed suntracker reset button. Then used webpage to go to position sun, then track mode. Solar data restarted at 13:15 LT.
    - Set IP address of camera (while at Caltech); will see if it is recognized on Mon.
    - Slew tracelogger and disabled it in /etc/config/sysinit.7 to conserve system resources. This seemed to help with the IFSloop restart at night: the instrument did not go "Absent".
    - Worked around "Non-recoverable failure in name resolution" in the nightly check of NTP sync. The trick is to parse /etc/hosts to get the Herc IP number and pass that as host to ntpq.
  • 2011-06-02
    - Dome turned to ~79 deg and suntracker put to Sun Position and then Tracking mode, shortly before 7 a.m.
    - Instrument was creating short slices that are actually HTML files saying "IFS125 HR SN_#092 EWS get file error" and "Can't get file /b26139.0". Stale counter was incrementing. Could also hear the "wailing" sound. Did a software shutdown quickly which took quite a while to wait for an idle scan. Then re-started IFSloop with a time warp. System went into Solar scans, but with same problem. Did a second software shutdown quickly with similar long wait for idle. Then pressed instrument reset button and waited for it to complete its initialization. Re-started IFSloop also with time warp. Now (8:27 LT) all okay.
    - 9:53 a.m. (LT) noted that tracker was not tracking.  Sent to Position Sun, then Tracking mode.  Also advanced dome to 113 deg.
    - Note: the Java applet no longer displays the values, including the quadrant sensor readings. But the MXY and MNY in data slices seem okay, so we must be tracking the sun.
    - Also: lots of high clouds.
    - 10:46 a.m. (LT) noted that tracker was not tracking again.  Sent to Position Sun, then Tracking mode.  Also advanced dome to 144 deg., but dome was not blocking the tracker.
    - 11:30 a.m. (LT), Press reading on barometer = 733.10 mm Hg (729.45 mm Hg @ 0 deg or 972.52 hPa); Setra  = 970.9 hPa, T = 24.3 C.
    - 13:20 LT instrument had low MXY and MNY. Sent tracker to position sun and track mode. Flip state says "Right". First good slice is b26786.0 at 13:27 which also happens to be the first slice of a fwd run.
    - Discovered JPL error from 8:20 p.m. and noted that IFS was Absent.  There is no means to reset remotely.  Ran a Savelog, StatusShutdownQuickly, StatusShutdownInstantly, and StatusReinit - nothing worked.
    - NOTE: the reason why the system did not come back up was the SaveLog. It simply needs to be restarted (software shutdown instantly ; IFSloop). I did so after your attempt.
  • 2011-06-01
    - Attempted to get suntracker tracking and dome moved into postion remotely.  No trouble accessing the hercules, but the webpage (via laptop to ccycle to scf-srv to oco-fts) connection was just too slow.  Was never able to see the quandrant sensors or switch in tracking mode.
    - Went to the container. The tracker was correctly in position sun. Sent the dome to 84 degrees azimuth and placed the tracker in tracking mode. Good solar data started at 8:13 LT.
    - 9:20 a.m. (LT), Press reading on barometer = 732.41 ( Local station pressure = 729.36 mm Hg @ 0 deg or 972.40 hPa;) Setra  = 970.6 hPa, T = 19.3.
    - Around 10:15 LT, noticed low values in data slices. The suntracker webpage first came up, but upon entering the control menu, the Java logo kept spinning. Attempted to stop the loading, but eventually got an "Unable to connect" error page from the browser. Lots of high clouds anyway, so no good data was lost.
    - Went to the container. Pressed the suntracker reset button, used the webpage to initialize, go to position sun, then to tracking mode. This changed the flip state from East to West (indicated as "Left" on the webpage), hence I was able to move the dome quite a bit, to azimuth of 227 degrees. Solar data recording started at 10:45 LT.

 Back to top

May 2011

  • 2011-05-31
    - Instrument reset was done and collection was started from within the trailer.
    - 12:00 p.m. (LT), Press reading on barometer = 733.11; Setra  = 970.6 hPa. (Did not record T.)
    - 1:19 p.m. (LT), moved dome to 303 deg.; sent tracker to Sun Position  and then back to Tracking mode.
    - Between 3:50 and 4:20 p.m., JF discovered that the instrument was in the dark:
    $ OpusHdr b23841.0 | egrep "MXY|MNY"
      MXY1: -0.000115804781671613
      MNY1: -0.000128336338093504
      MXY2: 0.000703093770425767
      MNY2: 0.000690987799316645
    
    and moved the dome to 278 deg:
    $ OpusHdr b23858.0 | egrep "MXY|MNY"
      MXY1: 0.213604494929314
      MNY1: 0.189191699028015
      MXY2: 0.528658747673035
      MNY2: 0.463931351900101
    
    Instrument left to shut down and restart in the morning by itself.
  • 2011-05-30
    - Started acquisition at solar elevation of 9.5 degrees (SolarRadiance was too low before that). Dome azimuth 78 degrees. Quadrant readings around 650 DN.
    - At 28 LT, noticed the IFS125 was creating single short slices (this may have been happening all along, I did not check the slices earlier). All subsystems indicated "Fail" and the stale counter was incrementing. Did Software Shutdown Instantly and restarted IFSloop. This re-established connection with the instrument, but the Detectors subsystem still showed Fail. A second re-start of the software showed the same result. This will likely require an instrument reset which currently cannot be done remotely. Put the software in Hold with the dome closed.
  • 2011-05-29
    - There is a 20 percent chance of showers this morning. The startup sequence took place without errors and the dome was open. Relative humidity is 79% and rain indicator says 0. Solar radiation varies greatly. Closed dome and put the software in Hold.
    - Did a SaveLog and a software re-start at night.
  • 2011-05-28
    - Started at the same elevation angle as yesterday: 11 degrees. Dome az of 76 degrees. Quadrant readings between 600 and 700 DN. Local time about 6:50.
    - Shortly after 10 AM LT, noticed low MXY and MNY values. Dome azimuth indicated that it was at home, with 149 degrees, and I don't know how that happened. Put the system in Hold and closed the dome.
    - Did a SaveLog and a software re-start after sunset so that the system would come up normally in the morning.
  • 2011-05-27
    - Started like yesterday, but a bit earlier: at solar elevation of 11 degrees. Dome azimuth of 77 degrees. Suntracker quadrant readings balanced in tracking mode and between 700 and 800 DN.
    - At 10:37 LT, dome azimuth display was 0.0 which is obviously wrong. Waited for a scan dead time, indicated by 2/0 in "Scan" field and by increasing stale counter, then did Dome Home, Dome Goto 110. The home position was at az 151 degrees with tracker quadrant sensor low, the dome did end up at 110 degrees as commanded with tracker quadrant sensor balanced and around 1200 DNs. Interferogram slices MXY and MNY were similar before and after this move, indicating that no data was lost.
    - At 12:47 LT, noticed that data slices MXY and MNY were low. Resisted the urge to walk up to the container, because I knew this was caused by the tracker needing a re-start around local noon, presumably to change its flip state. First sent the dome to azimuth 167 to better center it on the sun: no effect on the quadrant sensor readings which remained low. Then asked the tracker to go to Position Sun which caused its flip state to change from "right" (East) to "left" (West) and restored high quadrant sensor values, but unbalanced. Then switched to Tracking Mode which then balanced the quadrant sensor readings. This procedure took only a few minutes and restored the MXY and MNY is the data slices: DC term is around 0.21 for Si and 0.49 for InGaAs ; ZPD span is 0.13 to 0.29 for Si and 0.09 to 0.78 for InGaAs.
    - Forgot to keep track of the container... At 15:30 LT, reset dome and tracker. Azimuth 260 degrees.
    - Evening-time processing got stuck at "Waiting for data buffer" during restart of IFSloop. Saveset was produced correctly. Decided to reboot the Hercules which cleared the problem.
  • 2011-05-26
    - Started from home. IFS125 was already running okay. Commanded the tracker to Position Sun. Noted the azimuth angle reported by the tracker screen and sent the dome to Azimuth of 75 degrees. Then set the tracker mode to Track, which worked as witnessed by the equality of the quadrant readings (of about 1000 DN each).
    - The dome is acting funny: I first sent it to Azimuth 75 degrees, but the dome azimuth readout has been changing on its own: it's now showing 151 degrees but there are no move commands in the log. Maybe it's the sun shining on the sensing wheel ??
    - Restarted tracker at 09:35 LT at slice b18953.0.
    - Now at the container: restarted tracker at 13:10 LT which caused the tracker to change flip state. Sent dome to azimuth of 195 degrees to illuminate mirror now on the West side.
    - Puffy clouds, so solar intensity will be variable.
    - Sent dome to azimuth of 284 degrees at 13:40 LT
    - NOTE: all these dome az values are read from the IFSloop display. The commands were different, as the dome consistently undershoots.
  • 2011-05-25
    - Clear morning. The software was in play and the IFS125 was already taking scans with the correct slice progression and a stale counter of 0 during scans, no "wailing" either! Moved the dome to 93 degrees to allow the sun in. The suntracker had placed itself in a different mode, but all I had to do is command it to Position Sun which worked immediately, then Track Mode. Correct solar acquisition started around 8:35 LT. I could have done all this remotely: no need to press any reset buttons today!
    - Moved dome to azimuth 109 degrees at 10:18 LT.
    - At 12:48 LT, noticed the solar beam was mostly off the input tube, despite the tracker claiming to be in Tracking Mode. Sent the Position Sun command which eventually regained the sun, but only after forcing a change in the flip state. This put the elevation mirror towards the West which allowed me to rotate the dome by a large amount: sent the dome to 244 degrees in azimuth.
    - At 14:40 LT, moved dome to azimuth of 331 degrees which obscured a small part of the input tube while the IFS125 was inside a scan. Quickly backed off to 323 degrees. Looking inside the dome, that was still too close, so moved back to 301 degrees (I have the feeling the dome azimuth might be off: 333 degrees were fine yesterday).
  • 2011-05-24
    - Sky cleared up mid-morning. Went up to the container: the system was in Play, indicating that the morning pump-down went well. However, the IFS125 was in the single slice mode (these are tiny erroneous files, BTW) and it was making the "wailing" sound.
    - Tried to bring the suntracker up with its web interface, but that resulted in many beeps and it ignored all my commands. Pushed the reset button of the suntracker, and that allowed me to control it: did an initialize, followed by a position sun which brought it close enough to switch to tracking mode. Positioned the dome way ahead of the sun: this is now possible because we're close to the summer solstice (we use mainly the top of the dome).
    - Tried to simply re-start IFSloop, but that was not sufficient (I did not expect it would be, but the idea is to try the least damaging thing first). Exited IFSloop, used the reset button of the IFS125, restarted IFSloop, and the system is now taking solar scans with the correct number of slices. Data taking started at about 11:09 LT.
    - Moved dome to azimuth 333 degrees at 13:50 LT.
  • 2011-05-23
    - Evacuated instrument down to 0.60 hPa as reported by the IFS125 itself. Pump pressure reading was 0.56 hPa at that time. After closing IFS125 valve, pump pressure reading reached 0.12 hPa and kept that value even after turning the pump off thanks to the valve at pump inlet. At 9:26 LT, IFS125 pressure was 0.66 hPa with instrument temperatures of 21.8 C and 22.0 C.
    - IFSloop had been running in Hold through the weekend without problems in the dialog with the IFS125. Did a Savelog and a Shutdown Quickly. Upon auto-restart, the IFS125 was marked Absent. Did a Shutdown Instantly. Transferred all recorded data to nicolet: this is best done outside of IFSloop to avoid having too many files opened on the QNX4 system. Ran "dircksum" on nicolet and verified that files were correctly copied.
    - Re-started IFSloop at 10:27 LT, this time the connection to the IFS125 came up correctly. Instrument pressure is now 0.90 hPa with temperatures of 22.2 C and 22.0 C. Placed system in Hold: it is cloudy today.
    - Tried to install the optical fiber between the Herc and the Zeno, but it did not work. Will investigate later. We're back on the cable.
    - At 18:04 LT the IFS125 pressure is 1.54 hPa with temperatures of 23.1 C and 24.0 C. Pump pressure is 0.37 hPa.
    - Took system out of Hold via software ReInit to allow it to record nightly cell runs. However this placed the instrument in the mode in which it produces only one slice per scan. Then exited IFSloop and re-started from scratch. This also did not work: still one slice per scan. Restarted IFSloop, closed the dome, placed the system back in Hold.
    - Went up to the container, stopped IFSloop, reset the IFS125 with the reset button, restarted IFSloop, and it all worked. The system recorded the automated lamp run and those 16 spectra look nice.
  • 2011-05-20
    - Came in to find the IFS125 marked as Absent, so attempted to reinitialze the IFSloop software.  Noted the faint hooting sound again when collecting solar spectra.  In addition, the # of scans (2/0 etc.) was not updating on the IFS125 section of the Herc con1 page and the stale counter went to about 78, the FTS beeped, and all was reset. The Wr and Rd updated at this point.  Did a StatusShutdownQuickly and then left for Gretchen's defense celebration.
    - Returned and noted that the indicator lights on the front of the power supply were not blinking as I would expect - FWD and TKD were blinking continuously and that strange hooting sound was pretty regular.  Decided to vent the instrument and verify that it was indeed scanning while running OPUS.The instrument did not find the detectors and beamsplitter initially and I needed to cycle the power to the FTS and restart OPUS.  The indicator lights began blinking properly; the hooting remained, but was much less regular.  Evacuated the instrument and restarted IFSloop.
    - Returned in the evening to shut down the sun tracker software.  IFSloop was in failure.
  • 2011-05-16
    - After the risk of the JPL brush fire spreading to the hillside 10 ft. from the trailer subsided, the Zeno weather station was configured to match the data string format that IFSloop expects.  The config files and wiring are different in every trailer and the JPL config file still needs further modification.  Changes included changing 1) the Wind Gust parameter to an interger and 2) the Solar Rad. to a floating decimal and switching the order of the 1) Bar. Press and Solar Rad. and the 2) V. Batt. and Firmware Met Expert parameters.
    - The IFS125 is sort-of-working.  While collecting solar spectra, the encoder/motor was making a faint hooting sound and the instrument did go into error.  We've also had a case by which the instrument would not recognize some of its elements (detectors and beamsplitter), and we had to power cycle it to get it to work again.  IFSloop controls the instrument and displays nearly all of the parameters correctly, with the exception of the laser amplitudes and offsets.  The web interface might have changed a bit for those, but they are not important.
    - The suntracker is tracks well.  There is one issue with the user interface: we have to stop the web page transfer and refresh every time we send a command, otherwise the browser is stuck in an update loop.  Also, there is still no news about the source code of the Java applet.
    - Network firewall is setup: we have DNS, NTP, SMTP, and SSH working.
  • 2011-05-13
    - Again, Baron could not find the problem with the FTS.  It went into fail when first turned it on, but after a hard power off/on it began to work.  He unplugged and removed all the cables from the main (top) power supply and checked for loose wires in the "mercury reader encoder" before putting the encoder back in.  There have been no errors since - vented or evacuated, lamp or solar, touching the FTS or not ...  Baron wrote on the report that if the problem occurs again, the customer wants a new encoder.
    - Added 220 ohm resistor to rotating part of dome: the controller now reports that DSR is closed and we can rotate the dome by command from the Herc.
    - Trained the dome by pointing it manually to North then issuing the GTRN command.
    - Set correct IP address for the IFS125 in the real-time software, then verified that the instrument status properly updates on the Herc screen (with the exception of the laser offsets and amplitudes).
    - Recorded first cell run under control of IFSloop!
  • 2011-05-12
    - Connected the pump (via a "power box" and fiber optic) and MKS pressure transducer to the Herc and pump status and pressure values were displayed with IFSloop.
    - Data stream frequency from Zeno was changed from 60 to 1 sec, but values were not displayed in IFSloop.  Instead, Herc beeped every sec.  Still needs work.
    - Dome was connected to the Herc (via COM2) and status was displayed in IFSloop.  Initially, the shutter could be opened and closed, but the azimuth rotation would not work.  From the information string output by the dome, Jean-Francois discovered that the DSR (Dome Support Ring) status is "open", i.e., it thinks there is an open door at the base of the dome.  There should be a 220 ohm resistor across the copper contacts to indicated that we don't have a door, but instead JF found a 22 kohm resistor.  Will switch tomorrow to see if it works.
    - Baron is coming tomorrow, so hopefully we can connect the FTS to the Herc then.
  • 2011-05-11
    - Herc 4 now in trailer; boots and runs IFSloop.  Within IFSloop - seems to apply voltages to IFS125, solar tracker, dome, and pump upon command and reset the hardware.  Connected the Zeno output through a serial port on the Herc (COM3) and watched data streaming:
     
ID # Date & Time

W Speed

(m/s)

W Dir

(deg)

Std Dev

W Dir

W Gust

(m/s)

Air T

(Deg C)

RH% Bar Pres (mb)

Solar Rad

Wm-2 per microA

Precip/ Snow Leaf Wetness

Firmware Met Expert

V. Batt (Volts)
4251

11/05/12

02:02:55

0.0 0 41.0 0.0 18.5 61 968.7 27 0 90 180000 13.7

 

  • 2011-05-09
    - FTS went into error mode, while in idle (~12:40 p.m.)  Noted that the encoder LED's were red again.  Tried doing a reset, reinitialized instrument via the webpage, and tried a hard power off/on, but the scanner would not start up again.  Finally, vented the instrument to look inside and confirmed that the scanner was indeed not moving.  Wrote to Baron and tried one last reset, which worked - at least temporarily.
  • 2011-05-05
    - Put Cell H (#4) in spectrometer.
    - Replaced the Oriel red filter PN #59520 (Schott or Hoya colored glass, cut-on 630 nm), with a Newport RG645 (Schott, Hoya colored glass or equivalent, cut-on 645 nm)
    - Baron did a site visit to determine the problem with the encoder error; error was reproduced, encoder was removed, examined, cleaned with Dust-Off, and adjusted.  Problem did not reappear with lamp or solar runs!  Baron also pointed out that we should not run at Phase Resolution of 0.3 cm-1 because this results in a Phase interferogram point of 189570 which is beyond the Peak Position of 145055.  A Phase Resolution of 0.4 cm-1 works (142180).
    - Solar tracker offsets were reset as -17.8 = Azi_offset and 123.45 = Ele_offset
    - Several things to note in the Si spectra - the larger wavenumber side drops too quickly, as compared to Park Falls/Lamont/etc., probably as a result of the beamsplitter coating.  There is leakage on the smaller wavenumber side due to the dichroic coating.  It was thought that the dichroic was put in backward by Bruker, but turning it made no effect.
  • 2011-05-03 
    Ran the ghost test with filter (70-3F-FA) using the XPM files dictated by Messershmidt et al.  (The laser board arrived very well optimized and a half turn adjustment on the pot was all that was necessary.)  The double-frequency signal (max ~ 11845 cm-1) measured at 7.5 kHz is smaller by more than an order of magitude than that seen in Lamont.  The GPR's are shown in the table below:
Scanner Frequency Number of Scans Ghost Amplitude Parent Amplitude GPR Parent Wavenumber Ghost Wavenumber
7.5 kHz 113 0.000004974 0.2040894 2.44x10-5 5915.898 9877.322
10 kHz 150 0.000007178 0.2040120 3.52x10-5 5915.898 9877.322
20 kHz 300 0.000012622 0.2038570 6.19x10-5 5915.898 9877.322
40 kHz 600 0.000030687 0.2033147 1.51x10-4 5915.898 9877.322


Tested an interferometer beamsplitter (A79) for Stan Sanders; he feared that it had some curvature.


Brian Drouin filled spare HCl cell; pressure = 3.80 +/- 0.05 Torr (3.8 torr = 506.6 Pascal).  Will run hi-res. lamp run after Baron's visit.
 

  • 2011-05-02
    Rotated the dichroic, placed ghost filter in the detector compartment, and started the ghost tests.  After running the 7.5 kHz, 1.0 cm-1 res test, decided to first do a 0.02 cm-1 lamp run and hence removed the ghost filter.  An error occurred near the end of the run and the scanner stopped.  Restarted a number of times, but to no avail.  The error did not occur at low resolution, nor during Front or Back Short Adjust modes.  Sent the Full Report to Baron, who said he will be in S. CA later this week and will stop by.  Then, completed low res. ghost tests.

  Back to top

April 2011

  • 2011-04-29  Put red filter in mount and measured "first" solar spectrum.
  • 2011-04-20  Base replaced on the solar turning mirror; now just need some sun.

 Back to top

March 2011

  • 2011-03-31  First solar spectra measured!
  • 2011-03-30  First interferogram measured!

    Laser type                    front        live max      live min      back
    Laser A  amplitude (V)    7.014            8.5         8.1         6.185
    Laser A offset (V)            -2.744         -2.7       -2.8         -2.805
    Laser B ampl (V)            5.003            6.3         5.9         4.397
    Laser B offset                4.186            4.2         0.2         4.169
     
  • 2011-03-28  Installation of the new FTS system to begin!
  • 2011-03-09  Trailer in its new home at JPL!

 Back to top

February 2011

  • 2011-02-09  Solar tracker assembly, weather station, and RoboDome now mounted on trailer

December 2010

  • 2010-12-10  Container is on the ground at Caltech!

October 2010

  • 2010-10-27  Electrical wiring and insulation in place
  • 2010-10-21  First inspection of trailer modifications

September 2010

  • 2010-09-15  Fabrication of a new TCCON container has begun!

Back to top

Tag page

Files 1

FileSizeDateAttached by 
barometer_vs_Serta ifs8.xlsx
With last of JPL measurements
19.32 kB09:00, 24 Oct 2013coleenActions