TCCON > Sites > East Trout Lake > Instrument History

Instrument History

July, 2016

Back to top

August, 2016

  • August 3, 2016 - Had to reboot webcam to see images...
  • August 5, 2016 - Rebooted webcam again...
  • August 12, 2016 - Rebooted webcam.

Back to top

October, 2016

  • October 3, 2016 - IFSloop initiated!
    • Time was set incorrectly on the suntracker - timezone was UTC-4. Reset to UTC.
    • Too much light on the InGaAs detector - asked Stephen to put tape across the input tube. Values were:
      • 16:20:15 MXY1: 1.014 MNY1: 0.257
        16:20:15 MXY1: 0.884 MNY1: 0.883
        16:20:15 MXY1: 0.885 MNY1: 0.883
        16:20:15 MXY1: 0.886 MNY1: 0.883
        16:20:15 MXY1: 0.886 MNY1: 0.885
        16:20:15 MXY1: 0.886 MNY1: 0.884
        16:21:33 MXY1: 0.885 MNY1: 0.883
    • Now are:
      • 17:27:28 MXY1: 0.559 MNY1: 0.097
        17:27:28 MXY1: 0.345 MNY1: 0.340
        17:27:28 MXY1: 0.341 MNY1: 0.338
        17:27:28 MXY1: 0.343 MNY1: 0.339
        17:27:28 MXY1: 0.345 MNY1: 0.341
    • Don't understand why they're so high. Gregor's spectra looked fine using REF gain...
    • image2.jpeg
    • Norhof seems to be working ~correctly after moving the temperature sensor up *above* the level of the fill tube. InSb detector is cold, but not connected to the herc, as we suspect there may be a problem with ser4 in the herc.
  • October 4, 2016
    • Realized that the gains were too high because the resistors weren't changed. So we need to use REF gain. Norton changed the gains to 3 3 in IFS.tma, but we haven't seen the sun yet to confirm. Tape has been removed. (Edit: The resistors were changed, but we were still seeing too much light.)
    • Asked Stephen to change the InSb fill repeat time to 24 hours, which he did. He couldn't change the length of the fill tube to 1.5 m because it was already set to 1 m and wouldn't take decimal points. This is odd, because I thought Joseph and Orfeo set it to 2 m, and I had set it to 1.25 m.
    • Pumping down the pump is getting harder and harder. It took 5 tries in the afternoon to get the pump down below 0.3 hPa.
  • October 5, 2016 - SNOW!
    • 8:39 am EST - Pumping down manually before wakeup sequence.
    • Interestingly, the outside temperature is -0.5 C, but the suntracker temperatures are all above 5C (5.2-6.3C). No heaters are on. I presume that the heater connected to the suntracker enclosure is on, however, but it's hard to tell from the webcam.
    • Worked with Stephen on the Norhof system. We changed the following:
      • We set the fill tube length to 3 m (up from 1 m). That changed the pump freeze alarm set point to 62 mb (up from 45 mb).
      • We set the fill line height to 60 cm (up from 50 cm). That increased the rising height calibration from 20 mB to 24 mB.
      • We realized that we needed to unplug the pump and plug it back in for these changes to take effect.
      • Once this was done, the pump filled the dewar for 30 (or so) minutes without tripping a freeze alarm, and finished pumping when the temperature sensor was cold. Within 5 minutes, however, the temperature in the InSb detector was up at -138 C, but has held steady at -136C since then. Hopefully tomorrow's InSb fill will be quicker, as it should still be cold 24 hours later.
    • The solar tracker does not seem to move its elevation mirror after local noon. An initialization seems to "fix" this problem, and it doesn't appear to be a problem when in TTM.
      • Perhaps coincidentally, Gregor had complained that the GPS attached to the tracker (that has been disabled in software) was failing every day at local noon.
    • Pump down continues to be painfully slow, with very high ultimate pressures (~0.7hPa)
    • Norhof LN2 35L tank at 25 cm.
  • October 6, 2016
    • Power failure overnight.
    • Asked Stephen to take the Norhof pump out of the LN2 tank, as there was a power failure for ~6 hours.
    • Stephen is busy cutting up trees knocked over by the heavy, wet snow.
    • Stephen checked the UPS connections:
      • The UPS is a EATON PowerWare 9130. It is divided into two "segments". Segment one has a the router (that I added) and a power-bar. the power-bar goes to the de-ionised water container (I believe this is the IFS and chiller), the norhof, and a device called "A547-N electronic unit", presumably the suntracker electronics.

      • The Norhof after being placed back in LN2 seemed to fill fine after a short freeze alarm Stephen felt was tripped by starting the fill too quickly after placing the warm pump back into LN2.

  • October 7, 2016

    • Cold inside the lab (~10C). Maybe another power failure?

    • Norhof LN2 35L tank at 22 cm.

  • October 11, 2016

    • The solar tracker continues to stop moving its elevation mirror in TPS mode after hitting local noon. Gregor has had us change the parameter: MOD_AMPLITUDE_ELE from 1400 to 2400, and when that didn't work, we changed it to MOD_AMPLITUDE_ELE=3000, to no avail. This was done using links (links I edited the hardware configuration STRACK.INI file, saved the new config, and then reset the tracker. Then in IFSloop, issued a Sun Tracker Initialization command, which is a nice new feature that sends the tracker through its INIT mode, TPS, then TTM.

    • The Norhof doser has been filling on schedule (every 24 hours) since October 6. Today, Stephen filled the Norhof tank (it was at 18 cm before filling).

    • We're still trying to work out the formula for ensuring that both the InGaAs and InSb detectors are set to use REF gain, which Gregor found was the appropriate setting to ensure that the detectors do not saturate (they both reached ~10,000 counts, far from the 32,768 count saturation limit).

    • The initial pumpdown (of just the pump itself) continues to degrade. The pump is now barely reaching 0.8 hPa. The IFS itself creeps up slowly throughout the day, but generally does not rise much above 2 hPa after 24 hours.

    • We will likely repurpose the Zeno_T temperature sensor to measure the temperature in the pump room of the lab. It seems to get pretty warm in there. And given that it stays cool in the main room, this might not be a bad idea in general. Stephen will need to cut the cable and splice in an extension to reach the back room, however.

    • The herc has been up for 7 days, although there was a weird 13-hour glitch last night that we don't understand.

    • Jean-Francois added a timezone to the ETL herc, and so the savesets will now be named for the correct date (previously they have been named for the next day (due to the UTC date changeover)).

    • At some point, we'll get up the nerve to plug the Norhof doser back into the herc. But that day is not today.

  • October 12, 2016

    • The InGaAs is the detector still saturating, not InSb! We have now switched to use a pinhole of 0.8mm to prevent saturation.

    • Modified IPP on the herc to account for the M16 XSM parameter, following this bug report:

    • Pump down is still slow.

    • Rebooted the herc to implement the timezone change and reinstate /dev/ser4

    • Skies are nice and clear (note that MXY1 is InSb; MXY2 is InGaAs):

      • 15:36:50 MXY1: 0.259 MNY1: 0.022 MXY2: 0.790 MNY2: 0.110
        15:36:50 MXY1: 0.140 MNY1: 0.139 MXY2: 0.480 MNY2: 0.479
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.481 MNY2: 0.480
        15:38:08 MXY1: 0.141 MNY1: 0.140 MXY2: 0.481 MNY2: 0.480
        15:38:08 MXY1: 0.261 MNY1: 0.022 MXY2: 0.791 MNY2: 0.111

    • Getting close to saturating (now at ~11 am CST):

      • 17:09:02 MXY1: 0.166 MNY1: 0.166 MXY2: 0.519 MNY2: 0.518
        17:09:02 MXY1: 0.166 MNY1: 0.166 MXY2: 0.519 MNY2: 0.518
        17:09:02 MXY1: 0.167 MNY1: 0.166 MXY2: 0.519 MNY2: 0.519
        17:09:02 MXY1: 0.167 MNY1: 0.166 MXY2: 0.520 MNY2: 0.518
        17:09:02 MXY1: 0.310 MNY1: 0.026 MXY2: 0.854 MNY2: 0.121

        17:10:24 MXY1: 0.310 MNY1: 0.026 MXY2: 0.854 MNY2: 0.120
        17:10:24 MXY1: 0.167 MNY1: 0.166 MXY2: 0.520 MNY2: 0.519
        17:10:24 MXY1: 0.167 MNY1: 0.167 MXY2: 0.520 MNY2: 0.519

    • Herc crashed!! Something about the combination of the Norhof driver and the /dev/ser4 may be causing the crash

    • Upon bootup, Norton disabled the Norhof driver. We'll see if the herc remains stable.

    • Stephen extended the Zeno_T temperature sensor to reach the back room where the LN2 generator and other pumps live. This will allow us to monitor the temperature back there and ensure that it doesn't get too hot.

  • October 13, 2016

    • The InSb was not behaving - it seemed to be unable to communicate with the IFS.

      • IFS: 15:49:26 [ERROR] 15:49:24:213:f:Cmd Interpreter:DTC=0x4041:Device not connected

    • It was confirmed cold by Stephen. Resetting the IFS in software did not resolve the problem, so Stephen pressed the reset button on the IFS. Unfortunately, there was no sunlight after the reset to check to see whether the InSb detector began recording sensible signals.

      • 14:32:00 MXY1: -0.02 MNY1: -0.02 MXY2: 0.458 MNY2: 0.457
        14:32:00 MXY1: -0.02 MNY1: -0.02 MXY2: 0.752 MNY2: 0.106

        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.752 MNY2: 0.105
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.458
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.458
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.458
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.459
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.460 MNY2: 0.459

  • October 14, 2016

    • Orfeo asked Stephen to run a few tests on the pump to see whether we can get it working properly. It is now unable to pump below ~0.9 hPa. Here are the steps. Stephen reached #6 today, and will check in again tomorrow. The pump, however, after ~12 hours, has still not reached below 0.65 hPa

    • Disconnect pump from HERC

    • Power off pump

    • Open inlet to pump by removing “T” junction connected to pump
    • Power on pump for 3 minutes and let pump draw on the lab air
    • Power off pump
    • Re-install “T” junction to close inlet
    • Operate pump for 24 hours
    • Air flush pump
      • Stop vacuum pump
      • Remove plug from air flush port
      • Lightly attach Teflon air muffler (hand tight) to air flush port
      • Remove “T” junction from inlet and replace with blank cap found in our drawer system. I believe it’s in the bottom drawer in a bag with other vacuum parts.
      • Power on vacuum pump
      • Wait 5 seconds Power off vacuum pump
      • Remove Teflon air muffler and re-install plug slightly more than hand tight.
      • Remove blank from inlet port and re-install “T” junction.
      • Reconnect pump to HERC (Debra will recommend IFS pump down)
      • Monitor pressure over 24-48 hrs
    • As it stands, the pump is disconnected from the herc switch box and the IFS. Do NOT attempt to pump down the IFS! IFSloop has had the pumpdown and pump_P check disabled.

  • October 15, 2016
    • Ultimate pressure, after about 20 hours is *up* to 0.78 hPa!
  • October 18, 2016
    • Stephen tried pumping down the pump (without the tube attached to the IFS), and it reached ~0.6 hPa, then rose up above 100 hPa.
    • Stephen is packing up the pump and we're shipping it to get serviced. Running at ~4 hPa in the IFS now.
    • The InSb seemed to be producing sensible values today when the sun peaked out from behind the clouds.
  • October 21, 2016
    • Pump should be on its way to be serviced.
  • October 22, 2016
    • Suntracker lockup detected. Logged in and issued the following commands (while watching the webcam to ensure that the tracker stopped moving between commands):
      • sun tracker init
      • sun tracker track to position of sun
      • sun tracker track by diode
  • October 25, 2016
    • Norton implemented an automatic reinitialization sequence on the tracker when it senses it's locked up.
  • October 27, 2016
    • Rebooted the webcam, since it wasn't displaying images since 10 pm local last night.
    • 12cm remaining in 35L Norhof tank
    • Stephen starting up the LN2 generator.
  • October 29, 2016
    • Stephen filled the Norhof tank to 35L. It was at 10 cm.

Back to top

November, 2016

  • November 1, 2016
    • IFS_P now at 14.70 hPa
  • November 2, 2016
    • Sun!
      • 15:47:55 MXY1: 0.129 MNY1: 0.129 MXY2: 0.469 MNY2: 0.468
        15:47:55 MXY1: 0.130 MNY1: 0.129 MXY2: 0.469 MNY2: 0.469
        15:47:55 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.469
        15:49:13 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.469
        15:49:13 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.470
        15:49:13 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.470
        15:49:13 MXY1: 0.131 MNY1: 0.130 MXY2: 0.471 MNY2: 0.470
        15:49:13 MXY1: 0.131 MNY1: 0.130 MXY2: 0.471 MNY2: 0.470
        15:49:13 MXY1: 0.241 MNY1: 0.022 MXY2: 0.775 MNY2: 0.094
      • 8:13:33 MXY1: 0.159 MNY1: 0.144 MXY2: 0.509 MNY2: 0.467
        18:13:33 MXY1: 0.159 MNY1: 0.158 MXY2: 0.508 MNY2: 0.505
        18:13:33 MXY1: 0.293 MNY1: 0.025 MXY2: 0.833 MNY2: 0.102

        18:14:55 MXY1: 0.285 MNY1: 0.025 MXY2: 0.810 MNY2: 0.099
        18:14:55 MXY1: 0.160 MNY1: 0.159 MXY2: 0.509 MNY2: 0.507
        18:14:55 MXY1: 0.159 MNY1: 0.153 MXY2: 0.507 MNY2: 0.492
        18:14:55 MXY1: 0.157 MNY1: 0.075 MXY2: 0.503 MNY2: 0.275
        18:14:55 MXY1: 0.158 MNY1: 0.074 MXY2: 0.505 MNY2: 0.271
      • 18:57:39 MXY1: 0.159 MNY1: 0.158 MXY2: 0.509 MNY2: 0.509
        18:57:39 MXY1: 0.158 MNY1: 0.158 MXY2: 0.509 MNY2: 0.509
        18:57:39 MXY1: 0.158 MNY1: 0.152 MXY2: 0.509 MNY2: 0.491
        18:58:57 MXY1: 0.158 MNY1: 0.150 MXY2: 0.508 MNY2: 0.486
        18:58:57 MXY1: 0.158 MNY1: 0.156 MXY2: 0.509 MNY2: 0.500
        18:58:57 MXY1: 0.158 MNY1: 0.153 MXY2: 0.508 MNY2: 0.492
        18:58:57 MXY1: 0.158 MNY1: 0.150 MXY2: 0.509 MNY2: 0.489
        18:58:57 MXY1: 0.158 MNY1: 0.155 MXY2: 0.509 MNY2: 0.500
        18:58:57 MXY1: 0.293 MNY1: 0.025 MXY2: 0.838 MNY2: 0.103
    • Norhof tank at 26.7 cm; InSb temp just before a fill: -138C.
    • Stephen away until Nov 7.
  • November 7, 2016
    • 18:24:06 MXY1: 0.147 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.147 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.147 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.148 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.272 MNY1: 0.024 MXY2: 0.814 MNY2: 0.099

      18:25:28 MXY1: 0.273 MNY1: 0.024 MXY2: 0.814 MNY2: 0.098
      18:25:28 MXY1: 0.148 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:25:28 MXY1: 0.148 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
    • IFS_P at 19.30 hPa
    • 19:30:55 MXY1: 0.144 MNY1: 0.144 MXY2: 0.490 MNY2: 0.489
      19:30:55 MXY1: 0.145 MNY1: 0.144 MXY2: 0.490 MNY2: 0.489
      19:30:55 MXY1: 0.144 MNY1: 0.144 MXY2: 0.489 MNY2: 0.489
      19:30:55 MXY1: 0.144 MNY1: 0.144 MXY2: 0.489 MNY2: 0.488
      19:30:55 MXY1: 0.267 MNY1: 0.023 MXY2: 0.805 MNY2: 0.099

      19:32:17 MXY1: 0.267 MNY1: 0.023 MXY2: 0.805 MNY2: 0.098
      19:32:17 MXY1: 0.144 MNY1: 0.144 MXY2: 0.489 MNY2: 0.489
      19:32:17 MXY1: 0.145 MNY1: 0.144 MXY2: 0.490 MNY2: 0.489
  • November 10, 2016
    • InSb at -138C; Norhof tank at 18cm.
    • Stephen moved the webcam to the east corner of the building to avoid direct sunlight.
    • Stephen started up the Cryomech system to generate a new tank of LN2.
  • November 20, 2016
    • Stephen noticed that the enclosure was open while it was snowing. Put container in hold. Will leave it in hold overnight, as sleet is in the forecast for tomorrow.
  • November 21, 2016
    • Took system out of hold at 4pm CST.

Back to top

December 2016

  • December 1, 2016
    • Rebooted webcam (no images)
    • 9.9 cm Norhof
    • 32 hPa IFS P
  • December 8, 2016
    • Scroll pump should be shipped back to Prince Albert - delivery around Tuesday, December 13.
    • -25.7C outside today! Temperatures look okay; ST temps are ~-15C.
    • IFS_P = 38 hPa.
  • December 12, 2016
    • Jean-Francois changed the ALT field in the header from 495 to 502m, but accidentally added a decimal place, which made the altitudes default to -858993459 on Dec. 12. This is easily fixed in create_sunrun.f:
      • $ hg diff create_sunrun.f 
        diff -r 4837d78ca9d8 src/create_sunrun/create_sunrun.f
        --- a/src/create_sunrun/create_sunrun.f Fri Oct 31 07:24:29 2014 -0700
        +++ b/src/create_sunrun/create_sunrun.f Tue Feb 14 09:24:40 2017 -0500
        @@ -206,7 +206,14 @@
                        tcorr = 0.0d0
        +          if(instr.eq.2 .and. logfile(1:2).eq.'et') then ! ETL
        +            if(obalt.le.0) then
        +              write(*,*)'Warning: replacing obalt=',obalt
        +              obalt=0.502
        +              write(*,*)'with obalt=',obalt
        +            endif
        +          endif
                   call write_sunrun(lunt,col1,specname,object,tcorr,oblat,
              &    oblon,obalt,tins,pins,hins,tout,ptrue,hout,sia,fvsi,wspd,wdir,
  • December 14, 2016
    • Scroll pump has arrived and is back in ETL.
    • Pumping down for ~1 hour to flush. Ultimate pressure initially was only ~0.18 hPa, but subsequent pumpdown reached 0.08 hPa quickly.
    • Will pump down IFS (currently 43 hPa) this evening if the hour-long flush goes well.
  • December 15, 2016
    • Pump seems to be working; IFS now evacuated.
  • December 21, 2016
    • Pump was flushed; Stephen found more dust inside, but it was different this time:
      • "The grey dust was back En force. I cleaned everything with methanol and kimwipes. This dust is different though. Instead of grey-green it's more salt and pepper like, and it isn't oily feeling like the other dust. everything is hooked up, so you should be able to observe pressures."
    • This seems to suggest to me that there may have been desiccant inside the pump before it was evacuated.
  • December 26, 2016
    • In /etc/ntp.conf, added four servers from the group. If these work better than the ones from the, we'll delete the US servers later.
    • They seem to be a bit better.
    • Here's Dec 25:
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        *biisoni.miuku.n   2 u 1024  377  718.730    4.771  17.143  2 u 1024  377  5608.63  -2468.0 218.962     2 u 1024  377  846.973  -83.177 1236.84
        +mirror    3 u 1024  377  699.252   20.113  27.063
    • Here's Dec 26:
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        *biisoni.miuku.n   2 u 1024  377  719.863    3.209  20.762  2 u 1024  377  4133.16  -1722.5 1266.50       2 u 1024  377  709.081    9.737  41.939
        +mirror    3 u 1024  377  691.964   10.869  37.316
    • Here's Dec 27 (when the change took place):
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        -time.srv.ualber    2 u 1024  377  869.895  -56.449 278.075  2 u 1024  377  681.190    7.279  62.100      2 u 1024  377  661.542    1.434   8.995
         mirror3.rafal.c     2 u 1024  377  1001.13  -172.40 2280.68  2 u 1024  377  730.552   14.582  33.815
        +mirror    3 u 1024  377  670.350   -0.833  22.328   2 u 1024  377  688.732   -2.811 952.879
        *   2 u 1024  377  672.552   -1.718   0.977
    • Here's Dec 28:
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        +time.srv.ualber     2 u 1024  377  691.243   -6.208  12.278  2 u 1024  377  671.941    8.350  62.157
        *      2 u 1024  377  638.989    0.058   9.930
         mirror3.rafal.c     2 u 1024  377  951.716  -154.82 2346.32  2 u 1024  377  726.713   15.988  17.780
         mirror    3 u 1024  377  792.983  -23.498 1031.49   2 u 1024  377  4195.89  -1721.0 760.918   2 u 1024  377  720.833   31.859   2.832

Back to top

January 2017

  • January 2, 2017
    • The required resets on the tracker are becoming more and more frequent. Perhaps the cold temperatures are adding even more friction to the tracker, or the tracker azimuth drive is failing...
  • January 4, 2017
    • Norton implemented the last of Gregor's suggestions for the tracker (PFACT_ELE=35000)... it has not locked up since! Attached is the full report just before changing PFACT_ELE=35000.
  • January 12, 2017
    • Beautiful sunny day, but the enclosure didn't open because the rain gauge says it's raining, and the pyranometer says it's cloudy!
    • Stephen blew off the snow and ice, and I've contacted Coastal Environmental to see what the temperature range is on the gauge.
    • Stephen is running another pump flush - found lots of grey dust again when opening the pump.
    • Precip sensor finally read 0 around 11 pm EST
    • Lab temperatures got very hot in the evening (~38C in the pump room; 32C in the main room). Stephen went over to check it out - the main air conditioner was not blowing cold air. Stephen and I made the decision to turn off the cryomech system until we know how to fix it. Stephen turned on the auxiliary air conditioner, but we didn't think that could keep the lab cool with the LN2 generator on. The Norhof system has ~18cm LN2.
      • Stephen noticed that the breaker for the AC in the lab had tripped. He turned it back on and the system is back up and running. We decided to leave the cryomech off until tomorrow to ensure that the AC remains on.
  • January 13, 2017
    • Sunny day again, but the tracker has frozen up; won't initialize. Several resets and re-inits have not helped. The tracker temps are now really cold (-18C, whereas previously they were only -10C). Will try again in an hour or so. Indoor temps are now cool (15-18C).
    • Nothing has worked to get the tracker to initialize. Stephen tried warming up the tracker with a heat gun, and I tried changing PFACT_ELE=40000, but neither worked after several resets and reinits. The error we saw was an elevation motor jam error. Emailed Gregor for advice.
  • January 16, 2017
    • Stephen and Orfeo did some sleuthing today. Here's what they found.
      • To begin with, the elevation mirror was stuck against the limit switch.
        The arrow keys in the webpage moved the tracker's azimuth drive, but not the elevation drive.
        After switching ele/azi cables, the elevation and azimuth mirrors moved and stopped before reaching their limit switches.
        After switching the cables back, Stephen disconnected the limit switch. At this point, the elevation mirror turned several times in full circles without stopping (several rotations despite trying to hit the arrow keys or disabling the motors, but I think the system had AUTO_INIT=1 turned back on at this point). The limit switch was reinstalled and again the elevation mirror stopped against the switch and won't budget.
        The voltages read +19.999V on the elevation mirror and +20V on the azimuth mirror.
        Stephen is working on installing TeamViewer on his laptop.
      • Tracker still not working. Cloudy day.
  • January 19, 2017
    • Gregor and Stephen working on diagnosing the tracker - either an error in the elevation motor itself or in the controller
    • Stephen installed the TimeMachine GPS time server (with the antenna on the railing on the roof deck on the west side, near the pressure sensor funnel) and Norton added it to the ntp config list - reduced the delay:
    • # ntpq -p
           remote           refid      st t when poll reach   delay   offset  jitter
      ==============================================================================       .GPS.            1 u   29   64    1    0.977   -7.407   0.977  2 u   28   64    1  846.691   30.268   0.977  2 u   28   64    1  686.028  -18.119   0.977
       sanction.trebor     2 u   28   64    1  700.930  -21.796   0.977
       mirror3.rafal.c     2 u   27   64    1  682.719   -7.840   0.977 .CDMA.           1 u   27   64    1  999.740  -26.191   0.977
       clocka.ntpjs.or    2 u   27   64    1  704.757  -25.094   0.977
       host2.kingrst.c    3 u   27   64    1  794.335  -41.438   0.977   2 u   26   64    1  800.474   19.541   0.977
    • Gregor has determine the error is not in the motors themselves, but that the ELE amplifier output is unable to control either ELE or AZI drive. We need to fix the controller. We have an extra one in the lab at U of T that we will need to configure to put into the field.
  • January 27, 2017
    • Stephen and I installed the new tracker contoller. Gregor had put a copy of the controller firmware on Stephen's laptop, and I was able to upload it back to the tracker (after changing the IP address, Gateway and Subnet mask).
    • The batteries on both the original enclosure electronics box and the newly shipped one are dead. It's not clear whether either will recharge enough to be useful. Stephen will let the new batteries charge up overnight and check on them tomorrow.
    • Took a few solar scans manually today. Signal levels look good.
  • January 31, 2017
    • Stephen put the recharged batteries back into the new controller. The enclosure now seems to open and close as it should.
    • Signal levels are a bit high now on this bright and sunny day:
      • 18:48:28 MXY1: 0.395 MNY1: 0.035 MXY2: 0.950 MNY2: 0.109
        18:48:28 MXY1: 0.212 MNY1: 0.212 MXY2: 0.574 MNY2: 0.573
        18:48:28 MXY1: 0.212 MNY1: 0.212 MXY2: 0.574 MNY2: 0.573
        18:48:28 MXY1: 0.212 MNY1: 0.211 MXY2: 0.574 MNY2: 0.571
        18:48:28 MXY1: 0.212 MNY1: 0.212 MXY2: 0.574 MNY2: 0.573

Back to top

February 2017

  • February 2, 2017 - rebooted webcam
  • February 7, 2017
    • Swapped out the old batteries from the enclosure electronics box for the newly purchased ones. New batteries seem to be working. They read 29.5V when plugged in (in series); 25.7V when disconnected.
      • sw status hold
      • enclosure relays off
      • sw status shutdown quickly
      • <<swap batteries>>
      • IFSloop start
      • tested the enclosure: (enclosure open.... enclosure close)
      • sw status reinit
    • Swapped disks:
      • Shut down IFSloop (sw status shutdown quickly)
      • Rebooted the herc (sudo shutdown)
      • Stephen went into BIOS upon reboot (DEL)
      • Turned the key to unlock the enclosure
      • Removed upper disk, wrapped in antistatic bag and placed in pelican case
      • Put the new removable disk into the enclosure
      • Turned the key to lock the enclosure
      • Exited out of BIOS (without saving)
      • Restarted IFSloop
    • I also formatted the new drive by typing "removable initialize" which seemed to work well. Note that all three disks associated with this herc should now be correctly formatted and this step can be skipped from now on.
  • February 23, 2017
    • Pump failed to reach any sort of pressure. Stephen opened it to start an air flush and noticed lots of black soot, and smelled burning (like burnt brakepads). Pump should be kept off. VPC inc has been contacted, since it has only be 2 months since the major service. Stephen has cleaned out as much of the black soot as he can from the tubing connecting the pump to the IFS.
    • Stephen also ran Paul Garner's instructions on testing the precipitation sensor (rain gauge), which seems to be reading 1 even when it's completely dry. The results of the tests indicated that the sensor is faulty and Coastal Environmental is sending a new sensor.
      • The rain gauge appeared dry and the leaf wetness sensor read 0. The outside temperature was -9C and it was partly cloudy.
        The rain gauge read 1.
        Stephen disconnected the green wire from channel 14 and the yellow wire from channel 12.
        The rain gauge read 0.
        Stephen reconnected the yellow wire to channel 12.
        The rain gauge read 1.
        Stephen reconnected the green wire to channel 14.
        The rain gauge read 1.
  • February 27, 2017
    • IFS_P = 3.60 hPa

Back to top

March 2017

  • March 2, 2017 - IFS_P = 5.33 hPa
  • March 7, 2017 - IFS_P = 7.95 hPa
    • The tracker does not seem to be pointing toward the sun at 1:45 pm EST, it's too far East. Skies are cloudy, elevation seems to be tracking the model. (Did we blow an azimuth amplifier?!?)
    • Snowing and windy, so put container in hold.
    • Did SW Status Sun Tracker Initialize and that seemed to move the azimuth closer to what I'd expect... for now. It's quite snowy and -15.7C. Perhaps there was snow built up on the enclosure base plate?
  • March 9, 2017 - IFS_P - 8.33? hPa
    • Tracker did not turn in the morning. Norton tried initializing it (sw status sun tracker initialize), but to no avail. By 2 pm EST, it was able to initialize and point toward the sun.
  • March 10, 2017
    • Tracker won't initialize. Stephen watched and it turned in azimuth, but the elevation mirror didn't come off the limit switch.
    • Emailed Gregor.
  • March 13, 2017
    • Tried Gregor's suggestions - it still seems as though the amplifier for the elevation motor has blown (again). The old controller didn't manage to make it out of Canada to Germany, so it's still not in Gregor's possession to fix. Here's what we tested today (and sent to Gregor):
    • Stephen went out to the lab and reported that for all your tests, the 
      +5V/+/-12V lights remained on and steady. After running your tests, we 
      tried initializing the tracker, and again it rotated in azimuth but not 
      in elevation. It looks (to us) like the same amplifier problem as last 
      time. Do you think we need to have the amplifier in this controller 
      modified as well? Is this something our electronics shop could do here 
      in Toronto or is it best done at Bruker? Here are the details:
      To begin with:
         +5V and ±12V lights were on and steady
      Stephen turned off the electronics box, disconnected the AZI and ELE 
      cables and rebooted the electronics
          +5V and ±12V lights were on and steady
          Stephen was able to connect to the tracker with the webpage interface.
      Stephen then turned off the electronics box and reconnected the AZI cable
          +5V and ±12V lights were on and steady
          Stephen was able to connect to the tracker via the webpage.
      Stephen then turned off the electronics box, disconnected the AZI cable 
      and connected the ELE cable
          +5V and ±12V lights were on and steady
          Stephen was able to connect via the webpage.
      Stephen then turned off the electronics box and reconnected both the AZI 
      and ELE cables
          +5V and ±12V lights were on and steady
          We were able to connect via the webpage.
  • March 27, 2017
    • IFS_P is now at 15.70 hPa
    • The third replacement tracker controller is not identical to the previous two controllers (it has an AC plug whereas the others had DC plugs). Confirming with Gregor that those differences do not impact the firmware and interface with the hardware before installation.
  • March 29, 2017
    • IFS_P now around 17 hPa
    • We got the third replacement tracker up and running. Here were the steps:
      • Without connecting the motor cables, but connecting the power and ethernet directly to Stephen's laptop:
        • Logging in to the laptop via TeamViewer
        • Set Stephen's laptop's ethernet IP address to (anything on the 10.10.0.X except 1 and 2)
        • Point browser to
        • Change the tracker timezone to UTC and the time roughly correct
        • Set the IP address to, subnet mask to, gateway to (router)
        • Reset the instrument
      • Ensure that you can reconnect via the webpage pointed to
      • On Stephen's computer is a program called fconf.exe. It is located in Desktop/EC and UofT/Fconf_for_Opus_Release/fconf.exe
      • Run fconf.exe, selecting the "Restore" option
      • Select the run3 version of the firmware and strack.ini files by selecting Desktop/EC and UofT/fconf/run3 when prompted
      • Run fconf.exe
      • Restart the electronics and ensure that the parameters have changed to roughly match the full report appended below.
      • We then opened the enclosure and shut down IFSloop (sw status shutdown quickly).
      • We then connected the controller through the router to the laptop and ensured that still reached the controller
      • Finally, we turned off the controller, connected the elevation and azimuth motor cables, and turned on the tracker
      • With the webpage, we initialized the tracker and when finished, put the tracker into position sun and tracking mode
      • Closed the webpage.
      • We restarted IFSloop (IFSloop start), and the herc seemed to be able to communicate with the tracker
  • March 30, 2017
    • Stephen installed the new precipitation sensor from Coastal Environmental around 5 pm CST.

Back to top

April, 2017

  • April 12, 2017
    • IFS_P = 24.4 hPa
  • April 18, 2017
    • IFS_P = 29? hPa
  • April 27, 2017
    • IFS_P = 29.30 hPa
    • Swapped disks

Back to top

May, 2017

  • May 2, 2017
    • Vacuum Products Canada, Inc. admitted to us that they installed the tip seal "the wrong way causing an irreversible damage to the pump. We have taken the pump back and inspected it but it is clear that it is unfortunately not repairable – or at least not at a good price."
    • They will be replacing the pump with a new one. However, it will be a privately labeled Anest Iwata ISP500: Edwards GVP30. It will come with a 12 month warranty.
  • May 19-20, 2017
    • InSb was not cold due to a power failure that caused the Norhof system to stop filling.
  • May 26, 2017
    • "New" pump (with original motor) was installed and seems to be working fine. Reinstating morning pumpdowns.
    • No temperature sensor cable installed.

Back to top

June, 2017

  • June 8, 2017 - Greg Neufeld's first trip to ETL as our tech.
    • Greg reported that the lab was messy. There were tools out and pieces of hardware on the IFS.
    • There is some dirt/pine needles on the CaF2 window in the ceiling. This may need cleaning soon.
    • We swapped disks, following the procedure from Feb 7, 2017.
    • We used the USB-serial cable already connected to the Norhof system to plug into the herc. It seemed to work. We'll wait and see if the herc crashes...
      • I'm also a bit concerned about whether the autofill will interfere with the herc's fill commands. I'm not sure when the autofill happens, and I also think the time drifts every day. But I do think that the timer for the autofill will be delayed every time the herc fills the InSb.
    • The norhof system has only 3 cm left! We'll need to fill today, if possible.
    • Greg is running through the purge stage of starting up the Cryomech LN2 generator.
  • June 9, 2017
    • Greg filled the Norhof tank about halfway (9cm), which was just about all the generator had made. He'll come back on Monday to fill it the rest of the way.
    • The O2 sensor is going off frequently. I wonder if it needs to be calibrated again.
  • June 12, 2017
    • Greg filled the Norhof - it's now full (~20cm).
  • June 14, 2017
    • LN2 did not fill - data are all bad on InSb
  • June 15, 2017
    • Greg went out to the lab and the Norhof was beeping. Starting pumping with the button; seemed to work okay. However, the temperature would reach ~-190C, and then bounce back up pretty quickly to ~-140C. This caused the Norhof to claim that the InSb was "warm and dry" (i.e., the green light), and the PStat was still active (although not pumping).
    •  After the first time it did this, I had Greg push the button again - that put the system into "Sleep" mode. Then I asked him to push the button again, and it started pumping, but with the same result. It reached ~-190C, but only stayed there for a few seconds, increasing reasonably quickly to ~-140C.
    • Either the InSb jacket is starting to lose pressure, or perhaps the dewar is so long that the temperature sensor isn't close enough to the LN2.
    • 3:40 pm EDT: InSb is cold (for now). ~20 minutes later, and the temperature is still ~-139C.
  • June ~25, 2017
    • Greg noticed that the solenoid on the LN2 generator wasn't functioning correctly (it was turning on and off too rapidly). New one purchased and sent to PA; should arrive July 6, 2017.

Back to top

July 2017

  • July 5, 2017
    • Planned OCO-2 target-mode maneuver today...
    • Power outage all morning (reported by SaskPower at 8:37 am CST). ETR 11:45 am CST.
    • Power restored around 11:30 am CST, but suntracker cannot initialize.
    • Norton did the following to get the tracker back:
      • 'SW Status Reinit' followed by a 'SW Status Time Warp' to open the enclosure.
      • Sun Tracker Connect - to get it out of stale
      • Sun Tracker Init - to retry the init
      • Sun Tracker Track to Programmed Sun Position
      • Sun Tracker Track by Diode
    • Then, I noticed that the InSb was saturating, so I followed J-F's instructions:
      • I went to "Hold" and issued the these commands:
        • IFS Direct DTC=0x4041
        • IFS Direct PGN=3
        • Followed by re-init and time warp.
      • That seemed to work. 2 hour spare before the OCO-2 target.
  • July 8, 2017
    • Went into ~InSb_Chilled in the evening. Perhaps because the Norhof was filled a few cm by Greg.
  • July 11, 2017
    • New solenoid installed in Cryomech, perhaps broken, perhaps not?
  • July 12, 2017
    • Rebooted webcam, as it hadn't updated since yesterday evening.
  • July 20, 2017
    • Swapped disks following the Feb 7, 2017 instructions.
    • Looked at O2 calibration procedure and the location of the sensor.
      • The O2 calibration procedure is as follows:
        • Enter into the “ Main Menu ” by pressing the ENTER button while on the " Home " screen.
        • Navigate the menu until you reach the menu heading “CALIBRATE”.
        • Under the “CALIBRATE” heading there is a list of enabled sensors.
        • Navigate to the appropriate sensor name ("LOCAL") and press the ENTER button to edit the parameter.
        • Adjust the value for 20.9% and press the ENTER button to store this parameter.
        • Press the ENTER button a second time to confirm the Calibration.
        • Press the ESCAPE button to return to the main menu.
      • Greg thought that perhaps moving the O2 sensor below the air conditioner would get it out of the blast of the AC. He thought that over the door would still get too much direct air from the AC.
  • July 28, 2017
    • Lab is getting very hot. The pump room is at 42C and the IFS is at ~35C. The laser isn't able to lock ("not ready"). System in hold.

Back to top

August 2017

  • August 2, 2017
    • InSb didn't fill; ran on InGaAs only most of the day.
  •  August 15, 2017
    • Norhof LN2 doser pump has been removed from Dewar as onsite maintenance is on hiatus as Greg Neufield is sick with pneumonia in hospital. Darrel Ernst (Environment Canada) removed the pump for us as checked on the Cryomech LN2 generator which is turned off and empty. 
  • August 16, 2017
    • IFS has switched to InGaAs measurements only as InSb is no longer cooled. 
  • August 23, 2017
    • IFS cannot communicate with herc. In hold until Greg can reboot the IFS.
  • August 24, 2017
    • Restarted the IFS by switching off the bottom switch, followed by the top switch. Greg noticed that the MIR and NIR lights were on before shutting off. This made me think to look at the chiller's liquid level, which Greg thought was a bit low. He added ~2 cups of de-ionized water (up to the max line).
    • After filling the chiller, Greg switched the top switch, followed by the bottom switch back on. It took several minutes to clear all the errors. The laser ready light was not on, but eventually did lock, and the system is now back in play (at ~1:25 pm EDT).
    • Greg will now start running the LN2 generator.
    • IFS_P around 2.5 hPa.
  • August 26, 2017
    • InSb measurements restarted mid-day.
  • August 28, 2017
    • Swapping disks using the procedure outlined on February 7, 2017.
      • The disk in the top slot of the enclosure is shipped back to Toronto
      • The disk in the bottom slot of the enclosure is moved to the top slot
      • The disk that was in the box is now in the bottom slot
      • Unfortunately, the monitor in the lab wasn't working, so instead of rebooting the computer, we shut it down, swapped disks, and then rebooted it.
    • Greg will be out of town from August 30 - September 8.

Back to top

September, 2017

  • September 18, 2017
    • There was a power failure overnight. It's quite cold outside early in the morning (-3 C).
    • The tracker did not reconnect after the power failure. Did the followin:
      • Sun Tracker Connect - to get it out of stale
      • Sun Tracker Init - to retry the init
      • Sun Tracker Track to Programmed Sun Position
      • Sun Tracker Track by Diode
    • This seemed to get us back into Solar mode.
    • Pumpdown is very slow.

Back to top

Tag page

Files 1

FileSizeDateAttached by 
Sun tracker full report just before changing PFACT=35000
81.46 kB08:38, 11 Jan 2017dwunchActions