TCCON > Sites > East Trout Lake > Instrument History

Instrument History

July, 2016

Back to top

August, 2016

  • August 3, 2016 - Had to reboot webcam to see images...
  • August 5, 2016 - Rebooted webcam again...
  • August 12, 2016 - Rebooted webcam.

Back to top

October, 2016

  • October 3, 2016 - IFSloop initiated!
    • Time was set incorrectly on the suntracker - timezone was UTC-4. Reset to UTC.
    • Too much light on the InGaAs detector - asked Stephen to put tape across the input tube. Values were:
      • 16:20:15 MXY1: 1.014 MNY1: 0.257
        16:20:15 MXY1: 0.884 MNY1: 0.883
        16:20:15 MXY1: 0.885 MNY1: 0.883
        16:20:15 MXY1: 0.886 MNY1: 0.883
        16:20:15 MXY1: 0.886 MNY1: 0.885
        16:20:15 MXY1: 0.886 MNY1: 0.884
        16:21:33 MXY1: 0.885 MNY1: 0.883
    • Now are:
      • 17:27:28 MXY1: 0.559 MNY1: 0.097
        17:27:28 MXY1: 0.345 MNY1: 0.340
        17:27:28 MXY1: 0.341 MNY1: 0.338
        17:27:28 MXY1: 0.343 MNY1: 0.339
        17:27:28 MXY1: 0.345 MNY1: 0.341
    • Don't understand why they're so high. Gregor's spectra looked fine using REF gain...
    • image2.jpeg
    • Norhof seems to be working ~correctly after moving the temperature sensor up *above* the level of the fill tube. InSb detector is cold, but not connected to the herc, as we suspect there may be a problem with ser4 in the herc.
  • October 4, 2016
    • Realized that the gains were too high because the resistors weren't changed. So we need to use REF gain. Norton changed the gains to 3 3 in IFS.tma, but we haven't seen the sun yet to confirm. Tape has been removed. (Edit: The resistors were changed, but we were still seeing too much light.)
    • Asked Stephen to change the InSb fill repeat time to 24 hours, which he did. He couldn't change the length of the fill tube to 1.5 m because it was already set to 1 m and wouldn't take decimal points. This is odd, because I thought Joseph and Orfeo set it to 2 m, and I had set it to 1.25 m.
    • Pumping down the pump is getting harder and harder. It took 5 tries in the afternoon to get the pump down below 0.3 hPa.
  • October 5, 2016 - SNOW!
    • 8:39 am EST - Pumping down manually before wakeup sequence.
    • Interestingly, the outside temperature is -0.5 C, but the suntracker temperatures are all above 5C (5.2-6.3C). No heaters are on. I presume that the heater connected to the suntracker enclosure is on, however, but it's hard to tell from the webcam.
    • Worked with Stephen on the Norhof system. We changed the following:
      • We set the fill tube length to 3 m (up from 1 m). That changed the pump freeze alarm set point to 62 mb (up from 45 mb).
      • We set the fill line height to 60 cm (up from 50 cm). That increased the rising height calibration from 20 mB to 24 mB.
      • We realized that we needed to unplug the pump and plug it back in for these changes to take effect.
      • Once this was done, the pump filled the dewar for 30 (or so) minutes without tripping a freeze alarm, and finished pumping when the temperature sensor was cold. Within 5 minutes, however, the temperature in the InSb detector was up at -138 C, but has held steady at -136C since then. Hopefully tomorrow's InSb fill will be quicker, as it should still be cold 24 hours later.
    • The solar tracker does not seem to move its elevation mirror after local noon. An initialization seems to "fix" this problem, and it doesn't appear to be a problem when in TTM.
      • Perhaps coincidentally, Gregor had complained that the GPS attached to the tracker (that has been disabled in software) was failing every day at local noon.
    • Pump down continues to be painfully slow, with very high ultimate pressures (~0.7hPa)
    • Norhof LN2 35L tank at 25 cm.
  • October 6, 2016
    • Power failure overnight.
    • Asked Stephen to take the Norhof pump out of the LN2 tank, as there was a power failure for ~6 hours.
    • Stephen is busy cutting up trees knocked over by the heavy, wet snow.
    • Stephen checked the UPS connections:
      • The UPS is a EATON PowerWare 9130. It is divided into two "segments". Segment one has a the router (that I added) and a power-bar. the power-bar goes to the de-ionised water container (I believe this is the IFS and chiller), the norhof, and a device called "A547-N electronic unit", presumably the suntracker electronics.

      • The Norhof after being placed back in LN2 seemed to fill fine after a short freeze alarm Stephen felt was tripped by starting the fill too quickly after placing the warm pump back into LN2.

  • October 7, 2016

    • Cold inside the lab (~10C). Maybe another power failure?

    • Norhof LN2 35L tank at 22 cm.

  • October 11, 2016

    • The solar tracker continues to stop moving its elevation mirror in TPS mode after hitting local noon. Gregor has had us change the parameter: MOD_AMPLITUDE_ELE from 1400 to 2400, and when that didn't work, we changed it to MOD_AMPLITUDE_ELE=3000, to no avail. This was done using links (links I edited the hardware configuration STRACK.INI file, saved the new config, and then reset the tracker. Then in IFSloop, issued a Sun Tracker Initialization command, which is a nice new feature that sends the tracker through its INIT mode, TPS, then TTM.

    • The Norhof doser has been filling on schedule (every 24 hours) since October 6. Today, Stephen filled the Norhof tank (it was at 18 cm before filling).

    • We're still trying to work out the formula for ensuring that both the InGaAs and InSb detectors are set to use REF gain, which Gregor found was the appropriate setting to ensure that the detectors do not saturate (they both reached ~10,000 counts, far from the 32,768 count saturation limit).

    • The initial pumpdown (of just the pump itself) continues to degrade. The pump is now barely reaching 0.8 hPa. The IFS itself creeps up slowly throughout the day, but generally does not rise much above 2 hPa after 24 hours.

    • We will likely repurpose the Zeno_T temperature sensor to measure the temperature in the pump room of the lab. It seems to get pretty warm in there. And given that it stays cool in the main room, this might not be a bad idea in general. Stephen will need to cut the cable and splice in an extension to reach the back room, however.

    • The herc has been up for 7 days, although there was a weird 13-hour glitch last night that we don't understand.

    • Jean-Francois added a timezone to the ETL herc, and so the savesets will now be named for the correct date (previously they have been named for the next day (due to the UTC date changeover)).

    • At some point, we'll get up the nerve to plug the Norhof doser back into the herc. But that day is not today.

  • October 12, 2016

    • The InGaAs is the detector still saturating, not InSb! We have now switched to use a pinhole of 0.8mm to prevent saturation.

    • Modified IPP on the herc to account for the M16 XSM parameter, following this bug report:

    • Pump down is still slow.

    • Rebooted the herc to implement the timezone change and reinstate /dev/ser4

    • Skies are nice and clear (note that MXY1 is InSb; MXY2 is InGaAs):

      • 15:36:50 MXY1: 0.259 MNY1: 0.022 MXY2: 0.790 MNY2: 0.110
        15:36:50 MXY1: 0.140 MNY1: 0.139 MXY2: 0.480 MNY2: 0.479
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:36:50 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.480 MNY2: 0.480
        15:38:08 MXY1: 0.140 MNY1: 0.140 MXY2: 0.481 MNY2: 0.480
        15:38:08 MXY1: 0.141 MNY1: 0.140 MXY2: 0.481 MNY2: 0.480
        15:38:08 MXY1: 0.261 MNY1: 0.022 MXY2: 0.791 MNY2: 0.111

    • Getting close to saturating (now at ~11 am CST):

      • 17:09:02 MXY1: 0.166 MNY1: 0.166 MXY2: 0.519 MNY2: 0.518
        17:09:02 MXY1: 0.166 MNY1: 0.166 MXY2: 0.519 MNY2: 0.518
        17:09:02 MXY1: 0.167 MNY1: 0.166 MXY2: 0.519 MNY2: 0.519
        17:09:02 MXY1: 0.167 MNY1: 0.166 MXY2: 0.520 MNY2: 0.518
        17:09:02 MXY1: 0.310 MNY1: 0.026 MXY2: 0.854 MNY2: 0.121

        17:10:24 MXY1: 0.310 MNY1: 0.026 MXY2: 0.854 MNY2: 0.120
        17:10:24 MXY1: 0.167 MNY1: 0.166 MXY2: 0.520 MNY2: 0.519
        17:10:24 MXY1: 0.167 MNY1: 0.167 MXY2: 0.520 MNY2: 0.519

    • Herc crashed!! Something about the combination of the Norhof driver and the /dev/ser4 may be causing the crash

    • Upon bootup, Norton disabled the Norhof driver. We'll see if the herc remains stable.

    • Stephen extended the Zeno_T temperature sensor to reach the back room where the LN2 generator and other pumps live. This will allow us to monitor the temperature back there and ensure that it doesn't get too hot.

  • October 13, 2016

    • The InSb was not behaving - it seemed to be unable to communicate with the IFS.

      • IFS: 15:49:26 [ERROR] 15:49:24:213:f:Cmd Interpreter:DTC=0x4041:Device not connected

    • It was confirmed cold by Stephen. Resetting the IFS in software did not resolve the problem, so Stephen pressed the reset button on the IFS. Unfortunately, there was no sunlight after the reset to check to see whether the InSb detector began recording sensible signals.

      • 14:32:00 MXY1: -0.02 MNY1: -0.02 MXY2: 0.458 MNY2: 0.457
        14:32:00 MXY1: -0.02 MNY1: -0.02 MXY2: 0.752 MNY2: 0.106

        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.752 MNY2: 0.105
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.458
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.458
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.458
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.459 MNY2: 0.459
        14:33:22 MXY1: -0.02 MNY1: -0.02 MXY2: 0.460 MNY2: 0.459

  • October 14, 2016

    • Orfeo asked Stephen to run a few tests on the pump to see whether we can get it working properly. It is now unable to pump below ~0.9 hPa. Here are the steps. Stephen reached #6 today, and will check in again tomorrow. The pump, however, after ~12 hours, has still not reached below 0.65 hPa

    • Disconnect pump from HERC

    • Power off pump

    • Open inlet to pump by removing “T” junction connected to pump
    • Power on pump for 3 minutes and let pump draw on the lab air
    • Power off pump
    • Re-install “T” junction to close inlet
    • Operate pump for 24 hours
    • Air flush pump
      • Stop vacuum pump
      • Remove plug from air flush port
      • Lightly attach Teflon air muffler (hand tight) to air flush port
      • Remove “T” junction from inlet and replace with blank cap found in our drawer system. I believe it’s in the bottom drawer in a bag with other vacuum parts.
      • Power on vacuum pump
      • Wait 5 seconds Power off vacuum pump
      • Remove Teflon air muffler and re-install plug slightly more than hand tight.
      • Remove blank from inlet port and re-install “T” junction.
      • Reconnect pump to HERC (Debra will recommend IFS pump down)
      • Monitor pressure over 24-48 hrs
    • As it stands, the pump is disconnected from the herc switch box and the IFS. Do NOT attempt to pump down the IFS! IFSloop has had the pumpdown and pump_P check disabled.

  • October 15, 2016
    • Ultimate pressure, after about 20 hours is *up* to 0.78 hPa!
  • October 18, 2016
    • Stephen tried pumping down the pump (without the tube attached to the IFS), and it reached ~0.6 hPa, then rose up above 100 hPa.
    • Stephen is packing up the pump and we're shipping it to get serviced. Running at ~4 hPa in the IFS now.
    • The InSb seemed to be producing sensible values today when the sun peaked out from behind the clouds.
  • October 21, 2016
    • Pump should be on its way to be serviced.
  • October 22, 2016
    • Suntracker lockup detected. Logged in and issued the following commands (while watching the webcam to ensure that the tracker stopped moving between commands):
      • sun tracker init
      • sun tracker track to position of sun
      • sun tracker track by diode
  • October 25, 2016
    • Norton implemented an automatic reinitialization sequence on the tracker when it senses it's locked up.
  • October 27, 2016
    • Rebooted the webcam, since it wasn't displaying images since 10 pm local last night.
    • 12cm remaining in 35L Norhof tank
    • Stephen starting up the LN2 generator.
  • October 29, 2016
    • Stephen filled the Norhof tank to 35L. It was at 10 cm.

Back to top

November, 2016

  • November 1, 2016
    • IFS_P now at 14.70 hPa
  • November 2, 2016
    • Sun!
      • 15:47:55 MXY1: 0.129 MNY1: 0.129 MXY2: 0.469 MNY2: 0.468
        15:47:55 MXY1: 0.130 MNY1: 0.129 MXY2: 0.469 MNY2: 0.469
        15:47:55 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.469
        15:49:13 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.469
        15:49:13 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.470
        15:49:13 MXY1: 0.130 MNY1: 0.130 MXY2: 0.470 MNY2: 0.470
        15:49:13 MXY1: 0.131 MNY1: 0.130 MXY2: 0.471 MNY2: 0.470
        15:49:13 MXY1: 0.131 MNY1: 0.130 MXY2: 0.471 MNY2: 0.470
        15:49:13 MXY1: 0.241 MNY1: 0.022 MXY2: 0.775 MNY2: 0.094
      • 8:13:33 MXY1: 0.159 MNY1: 0.144 MXY2: 0.509 MNY2: 0.467
        18:13:33 MXY1: 0.159 MNY1: 0.158 MXY2: 0.508 MNY2: 0.505
        18:13:33 MXY1: 0.293 MNY1: 0.025 MXY2: 0.833 MNY2: 0.102

        18:14:55 MXY1: 0.285 MNY1: 0.025 MXY2: 0.810 MNY2: 0.099
        18:14:55 MXY1: 0.160 MNY1: 0.159 MXY2: 0.509 MNY2: 0.507
        18:14:55 MXY1: 0.159 MNY1: 0.153 MXY2: 0.507 MNY2: 0.492
        18:14:55 MXY1: 0.157 MNY1: 0.075 MXY2: 0.503 MNY2: 0.275
        18:14:55 MXY1: 0.158 MNY1: 0.074 MXY2: 0.505 MNY2: 0.271
      • 18:57:39 MXY1: 0.159 MNY1: 0.158 MXY2: 0.509 MNY2: 0.509
        18:57:39 MXY1: 0.158 MNY1: 0.158 MXY2: 0.509 MNY2: 0.509
        18:57:39 MXY1: 0.158 MNY1: 0.152 MXY2: 0.509 MNY2: 0.491
        18:58:57 MXY1: 0.158 MNY1: 0.150 MXY2: 0.508 MNY2: 0.486
        18:58:57 MXY1: 0.158 MNY1: 0.156 MXY2: 0.509 MNY2: 0.500
        18:58:57 MXY1: 0.158 MNY1: 0.153 MXY2: 0.508 MNY2: 0.492
        18:58:57 MXY1: 0.158 MNY1: 0.150 MXY2: 0.509 MNY2: 0.489
        18:58:57 MXY1: 0.158 MNY1: 0.155 MXY2: 0.509 MNY2: 0.500
        18:58:57 MXY1: 0.293 MNY1: 0.025 MXY2: 0.838 MNY2: 0.103
    • Norhof tank at 26.7 cm; InSb temp just before a fill: -138C.
    • Stephen away until Nov 7.
  • November 7, 2016
    • 18:24:06 MXY1: 0.147 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.147 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.147 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.148 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:24:06 MXY1: 0.272 MNY1: 0.024 MXY2: 0.814 MNY2: 0.099

      18:25:28 MXY1: 0.273 MNY1: 0.024 MXY2: 0.814 MNY2: 0.098
      18:25:28 MXY1: 0.148 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
      18:25:28 MXY1: 0.148 MNY1: 0.147 MXY2: 0.495 MNY2: 0.494
    • IFS_P at 19.30 hPa
    • 19:30:55 MXY1: 0.144 MNY1: 0.144 MXY2: 0.490 MNY2: 0.489
      19:30:55 MXY1: 0.145 MNY1: 0.144 MXY2: 0.490 MNY2: 0.489
      19:30:55 MXY1: 0.144 MNY1: 0.144 MXY2: 0.489 MNY2: 0.489
      19:30:55 MXY1: 0.144 MNY1: 0.144 MXY2: 0.489 MNY2: 0.488
      19:30:55 MXY1: 0.267 MNY1: 0.023 MXY2: 0.805 MNY2: 0.099

      19:32:17 MXY1: 0.267 MNY1: 0.023 MXY2: 0.805 MNY2: 0.098
      19:32:17 MXY1: 0.144 MNY1: 0.144 MXY2: 0.489 MNY2: 0.489
      19:32:17 MXY1: 0.145 MNY1: 0.144 MXY2: 0.490 MNY2: 0.489
  • November 10, 2016
    • InSb at -138C; Norhof tank at 18cm.
    • Stephen moved the webcam to the east corner of the building to avoid direct sunlight.
    • Stephen started up the Cryomech system to generate a new tank of LN2.
  • November 20, 2016
    • Stephen noticed that the enclosure was open while it was snowing. Put container in hold. Will leave it in hold overnight, as sleet is in the forecast for tomorrow.
  • November 21, 2016
    • Took system out of hold at 4pm CST.

Back to top

December 2016

  • December 1, 2016
    • Rebooted webcam (no images)
    • 9.9 cm Norhof
    • 32 hPa IFS P
  • December 8, 2016
    • Scroll pump should be shipped back to Prince Albert - delivery around Tuesday, December 13.
    • -25.7C outside today! Temperatures look okay; ST temps are ~-15C.
    • IFS_P = 38 hPa.
  • December 12, 2016
    • Jean-Francois changed the ALT field in the header from 495 to 502m, but accidentally added a decimal place, which made the altitudes default to -858993459 on Dec. 12. This is easily fixed in create_sunrun.f:
      • $ hg diff create_sunrun.f 
        diff -r 4837d78ca9d8 src/create_sunrun/create_sunrun.f
        --- a/src/create_sunrun/create_sunrun.f Fri Oct 31 07:24:29 2014 -0700
        +++ b/src/create_sunrun/create_sunrun.f Tue Feb 14 09:24:40 2017 -0500
        @@ -206,7 +206,14 @@
                        tcorr = 0.0d0
        +          if(instr.eq.2 .and. logfile(1:2).eq.'et') then ! ETL
        +            if(obalt.le.0) then
        +              write(*,*)'Warning: replacing obalt=',obalt
        +              obalt=0.502
        +              write(*,*)'with obalt=',obalt
        +            endif
        +          endif
                   call write_sunrun(lunt,col1,specname,object,tcorr,oblat,
              &    oblon,obalt,tins,pins,hins,tout,ptrue,hout,sia,fvsi,wspd,wdir,
  • December 14, 2016
    • Scroll pump has arrived and is back in ETL.
    • Pumping down for ~1 hour to flush. Ultimate pressure initially was only ~0.18 hPa, but subsequent pumpdown reached 0.08 hPa quickly.
    • Will pump down IFS (currently 43 hPa) this evening if the hour-long flush goes well.
  • December 15, 2016
    • Pump seems to be working; IFS now evacuated.
  • December 21, 2016
    • Pump was flushed; Stephen found more dust inside, but it was different this time:
      • "The grey dust was back En force. I cleaned everything with methanol and kimwipes. This dust is different though. Instead of grey-green it's more salt and pepper like, and it isn't oily feeling like the other dust. everything is hooked up, so you should be able to observe pressures."
    • This seems to suggest to me that there may have been desiccant inside the pump before it was evacuated.
  • December 26, 2016
    • In /etc/ntp.conf, added four servers from the group. If these work better than the ones from the, we'll delete the US servers later.
    • They seem to be a bit better.
    • Here's Dec 25:
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        *biisoni.miuku.n   2 u 1024  377  718.730    4.771  17.143  2 u 1024  377  5608.63  -2468.0 218.962     2 u 1024  377  846.973  -83.177 1236.84
        +mirror    3 u 1024  377  699.252   20.113  27.063
    • Here's Dec 26:
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        *biisoni.miuku.n   2 u 1024  377  719.863    3.209  20.762  2 u 1024  377  4133.16  -1722.5 1266.50       2 u 1024  377  709.081    9.737  41.939
        +mirror    3 u 1024  377  691.964   10.869  37.316
    • Here's Dec 27 (when the change took place):
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        -time.srv.ualber    2 u 1024  377  869.895  -56.449 278.075  2 u 1024  377  681.190    7.279  62.100      2 u 1024  377  661.542    1.434   8.995
         mirror3.rafal.c     2 u 1024  377  1001.13  -172.40 2280.68  2 u 1024  377  730.552   14.582  33.815
        +mirror    3 u 1024  377  670.350   -0.833  22.328   2 u 1024  377  688.732   -2.811 952.879
        *   2 u 1024  377  672.552   -1.718   0.977
    • Here's Dec 28:
      • Status of NTP sync:
             remote           refid      st t poll reach   delay   offset  jitter
        +time.srv.ualber     2 u 1024  377  691.243   -6.208  12.278  2 u 1024  377  671.941    8.350  62.157
        *      2 u 1024  377  638.989    0.058   9.930
         mirror3.rafal.c     2 u 1024  377  951.716  -154.82 2346.32  2 u 1024  377  726.713   15.988  17.780
         mirror    3 u 1024  377  792.983  -23.498 1031.49   2 u 1024  377  4195.89  -1721.0 760.918   2 u 1024  377  720.833   31.859   2.832

Back to top

January 2017

  • January 2, 2017
    • The required resets on the tracker are becoming more and more frequent. Perhaps the cold temperatures are adding even more friction to the tracker, or the tracker azimuth drive is failing...
  • January 4, 2017
    • Norton implemented the last of Gregor's suggestions for the tracker (PFACT_ELE=35000)... it has not locked up since! Attached is the full report just before changing PFACT_ELE=35000.
  • January 12, 2017
    • Beautiful sunny day, but the enclosure didn't open because the rain gauge says it's raining, and the pyranometer says it's cloudy!
    • Stephen blew off the snow and ice, and I've contacted Coastal Environmental to see what the temperature range is on the gauge.
    • Stephen is running another pump flush - found lots of grey dust again when opening the pump.
    • Precip sensor finally read 0 around 11 pm EST
    • Lab temperatures got very hot in the evening (~38C in the pump room; 32C in the main room). Stephen went over to check it out - the main air conditioner was not blowing cold air. Stephen and I made the decision to turn off the cryomech system until we know how to fix it. Stephen turned on the auxiliary air conditioner, but we didn't think that could keep the lab cool with the LN2 generator on. The Norhof system has ~18cm LN2.
      • Stephen noticed that the breaker for the AC in the lab had tripped. He turned it back on and the system is back up and running. We decided to leave the cryomech off until tomorrow to ensure that the AC remains on.
  • January 13, 2017
    • Sunny day again, but the tracker has frozen up; won't initialize. Several resets and re-inits have not helped. The tracker temps are now really cold (-18C, whereas previously they were only -10C). Will try again in an hour or so. Indoor temps are now cool (15-18C).
    • Nothing has worked to get the tracker to initialize. Stephen tried warming up the tracker with a heat gun, and I tried changing PFACT_ELE=40000, but neither worked after several resets and reinits. The error we saw was an elevation motor jam error. Emailed Gregor for advice.
  • January 16, 2017
    • Stephen and Orfeo did some sleuthing today. Here's what they found.
      • To begin with, the elevation mirror was stuck against the limit switch.
        The arrow keys in the webpage moved the tracker's azimuth drive, but not the elevation drive.
        After switching ele/azi cables, the elevation and azimuth mirrors moved and stopped before reaching their limit switches.
        After switching the cables back, Stephen disconnected the limit switch. At this point, the elevation mirror turned several times in full circles without stopping (several rotations despite trying to hit the arrow keys or disabling the motors, but I think the system had AUTO_INIT=1 turned back on at this point). The limit switch was reinstalled and again the elevation mirror stopped against the switch and won't budget.
        The voltages read +19.999V on the elevation mirror and +20V on the azimuth mirror.
        Stephen is working on installing TeamViewer on his laptop.
      • Tracker still not working. Cloudy day.
  • January 19, 2017
    • Gregor and Stephen working on diagnosing the tracker - either an error in the elevation motor itself or in the controller
    • Stephen installed the TimeMachine GPS time server (with the antenna on the railing on the roof deck on the west side, near the pressure sensor funnel) and Norton added it to the ntp config list - reduced the delay:
    • # ntpq -p
           remote           refid      st t when poll reach   delay   offset  jitter
      ==============================================================================       .GPS.            1 u   29   64    1    0.977   -7.407   0.977  2 u   28   64    1  846.691   30.268   0.977  2 u   28   64    1  686.028  -18.119   0.977
       sanction.trebor     2 u   28   64    1  700.930  -21.796   0.977
       mirror3.rafal.c     2 u   27   64    1  682.719   -7.840   0.977 .CDMA.           1 u   27   64    1  999.740  -26.191   0.977
       clocka.ntpjs.or    2 u   27   64    1  704.757  -25.094   0.977
       host2.kingrst.c    3 u   27   64    1  794.335  -41.438   0.977   2 u   26   64    1  800.474   19.541   0.977
    • Gregor has determine the error is not in the motors themselves, but that the ELE amplifier output is unable to control either ELE or AZI drive. We need to fix the controller. We have an extra one in the lab at U of T that we will need to configure to put into the field.
  • January 27, 2017
    • Stephen and I installed the new tracker contoller. Gregor had put a copy of the controller firmware on Stephen's laptop, and I was able to upload it back to the tracker (after changing the IP address, Gateway and Subnet mask).
    • The batteries on both the original enclosure electronics box and the newly shipped one are dead. It's not clear whether either will recharge enough to be useful. Stephen will let the new batteries charge up overnight and check on them tomorrow.
    • Took a few solar scans manually today. Signal levels look good.
  • January 31, 2017
    • Stephen put the recharged batteries back into the new controller. The enclosure now seems to open and close as it should.
    • Signal levels are a bit high now on this bright and sunny day:
      • 18:48:28 MXY1: 0.395 MNY1: 0.035 MXY2: 0.950 MNY2: 0.109
        18:48:28 MXY1: 0.212 MNY1: 0.212 MXY2: 0.574 MNY2: 0.573
        18:48:28 MXY1: 0.212 MNY1: 0.212 MXY2: 0.574 MNY2: 0.573
        18:48:28 MXY1: 0.212 MNY1: 0.211 MXY2: 0.574 MNY2: 0.571
        18:48:28 MXY1: 0.212 MNY1: 0.212 MXY2: 0.574 MNY2: 0.573

Back to top

February 2017

  • February 2, 2017 - rebooted webcam
  • February 7, 2017
    • Swapped out the old batteries from the enclosure electronics box for the newly purchased ones. New batteries seem to be working. They read 29.5V when plugged in (in series); 25.7V when disconnected.
      • sw status hold
      • enclosure relays off
      • sw status shutdown quickly
      • <<swap batteries>>
      • IFSloop start
      • tested the enclosure: (enclosure open.... enclosure close)
      • sw status reinit
    • Swapped disks:
      • Shut down IFSloop (sw status shutdown quickly)
      • Rebooted the herc (sudo shutdown)
      • Stephen went into BIOS upon reboot (DEL)
      • Turned the key to unlock the enclosure
      • Removed upper disk, wrapped in antistatic bag and placed in pelican case
      • Put the new removable disk into the enclosure
      • Turned the key to lock the enclosure
      • Exited out of BIOS (without saving)
      • Restarted IFSloop
    • I also formatted the new drive by typing "removable initialize" which seemed to work well. Note that all three disks associated with this herc should now be correctly formatted and this step can be skipped from now on.
  • February 23, 2017
    • Pump failed to reach any sort of pressure. Stephen opened it to start an air flush and noticed lots of black soot, and smelled burning (like burnt brakepads). Pump should be kept off. VPC inc has been contacted, since it has only be 2 months since the major service. Stephen has cleaned out as much of the black soot as he can from the tubing connecting the pump to the IFS.
    • Stephen also ran Paul Garner's instructions on testing the precipitation sensor (rain gauge), which seems to be reading 1 even when it's completely dry. The results of the tests indicated that the sensor is faulty and Coastal Environmental is sending a new sensor.
      • The rain gauge appeared dry and the leaf wetness sensor read 0. The outside temperature was -9C and it was partly cloudy.
        The rain gauge read 1.
        Stephen disconnected the green wire from channel 14 and the yellow wire from channel 12.
        The rain gauge read 0.
        Stephen reconnected the yellow wire to channel 12.
        The rain gauge read 1.
        Stephen reconnected the green wire to channel 14.
        The rain gauge read 1.
  • February 27, 2017
    • IFS_P = 3.60 hPa

Back to top

March 2017

  • March 2, 2017 - IFS_P = 5.33 hPa
  • March 7, 2017 - IFS_P = 7.95 hPa
    • The tracker does not seem to be pointing toward the sun at 1:45 pm EST, it's too far East. Skies are cloudy, elevation seems to be tracking the model. (Did we blow an azimuth amplifier?!?)
    • Snowing and windy, so put container in hold.
    • Did SW Status Sun Tracker Initialize and that seemed to move the azimuth closer to what I'd expect... for now. It's quite snowy and -15.7C. Perhaps there was snow built up on the enclosure base plate?
  • March 9, 2017 - IFS_P - 8.33? hPa
    • Tracker did not turn in the morning. Norton tried initializing it (sw status sun tracker initialize), but to no avail. By 2 pm EST, it was able to initialize and point toward the sun.
  • March 10, 2017
    • Tracker won't initialize. Stephen watched and it turned in azimuth, but the elevation mirror didn't come off the limit switch.
    • Emailed Gregor.
  • March 13, 2017
    • Tried Gregor's suggestions - it still seems as though the amplifier for the elevation motor has blown (again). The old controller didn't manage to make it out of Canada to Germany, so it's still not in Gregor's possession to fix. Here's what we tested today (and sent to Gregor):
    • Stephen went out to the lab and reported that for all your tests, the 
      +5V/+/-12V lights remained on and steady. After running your tests, we 
      tried initializing the tracker, and again it rotated in azimuth but not 
      in elevation. It looks (to us) like the same amplifier problem as last 
      time. Do you think we need to have the amplifier in this controller 
      modified as well? Is this something our electronics shop could do here 
      in Toronto or is it best done at Bruker? Here are the details:
      To begin with:
         +5V and ±12V lights were on and steady
      Stephen turned off the electronics box, disconnected the AZI and ELE 
      cables and rebooted the electronics
          +5V and ±12V lights were on and steady
          Stephen was able to connect to the tracker with the webpage interface.
      Stephen then turned off the electronics box and reconnected the AZI cable
          +5V and ±12V lights were on and steady
          Stephen was able to connect to the tracker via the webpage.
      Stephen then turned off the electronics box, disconnected the AZI cable 
      and connected the ELE cable
          +5V and ±12V lights were on and steady
          Stephen was able to connect via the webpage.
      Stephen then turned off the electronics box and reconnected both the AZI 
      and ELE cables
          +5V and ±12V lights were on and steady
          We were able to connect via the webpage.
  • March 27, 2017
    • IFS_P is now at 15.70 hPa
    • The third replacement tracker controller is not identical to the previous two controllers (it has an AC plug whereas the others had DC plugs). Confirming with Gregor that those differences do not impact the firmware and interface with the hardware before installation.
  • March 29, 2017
    • IFS_P now around 17 hPa
    • We got the third replacement tracker up and running. Here were the steps:
      • Without connecting the motor cables, but connecting the power and ethernet directly to Stephen's laptop:
        • Logging in to the laptop via TeamViewer
        • Set Stephen's laptop's ethernet IP address to (anything on the 10.10.0.X except 1 and 2)
        • Point browser to
        • Change the tracker timezone to UTC and the time roughly correct
        • Set the IP address to, subnet mask to, gateway to (router)
        • Reset the instrument
      • Ensure that you can reconnect via the webpage pointed to
      • On Stephen's computer is a program called fconf.exe. It is located in Desktop/EC and UofT/Fconf_for_Opus_Release/fconf.exe
      • Run fconf.exe, selecting the "Restore" option
      • Select the run3 version of the firmware and strack.ini files by selecting Desktop/EC and UofT/fconf/run3 when prompted
      • Run fconf.exe
      • Restart the electronics and ensure that the parameters have changed to roughly match the full report appended below.
      • We then opened the enclosure and shut down IFSloop (sw status shutdown quickly).
      • We then connected the controller through the router to the laptop and ensured that still reached the controller
      • Finally, we turned off the controller, connected the elevation and azimuth motor cables, and turned on the tracker
      • With the webpage, we initialized the tracker and when finished, put the tracker into position sun and tracking mode
      • Closed the webpage.
      • We restarted IFSloop (IFSloop start), and the herc seemed to be able to communicate with the tracker
  • March 30, 2017
    • Stephen installed the new precipitation sensor from Coastal Environmental around 5 pm CST.

Back to top

April, 2017

  • April 12, 2017
    • IFS_P = 24.4 hPa
  • April 18, 2017
    • IFS_P = 29? hPa
  • April 27, 2017
    • IFS_P = 29.30 hPa
    • Swapped disks

Back to top

May, 2017

  • May 2, 2017
    • Vacuum Products Canada, Inc. admitted to us that they installed the tip seal "the wrong way causing an irreversible damage to the pump. We have taken the pump back and inspected it but it is clear that it is unfortunately not repairable – or at least not at a good price."
    • They will be replacing the pump with a new one. However, it will be a privately labeled Anest Iwata ISP500: Edwards GVP30. It will come with a 12 month warranty.
  • May 19-20, 2017
    • InSb was not cold due to a power failure that caused the Norhof system to stop filling.
  • May 26, 2017
    • "New" pump (with original motor) was installed and seems to be working fine. Reinstating morning pumpdowns.
    • No temperature sensor cable installed.

Back to top

June, 2017

  • June 8, 2017 - Greg Neufeld's first trip to ETL as our tech.
    • Greg reported that the lab was messy. There were tools out and pieces of hardware on the IFS.
    • There is some dirt/pine needles on the CaF2 window in the ceiling. This may need cleaning soon.
    • We swapped disks, following the procedure from Feb 7, 2017.
    • We used the USB-serial cable already connected to the Norhof system to plug into the herc. It seemed to work. We'll wait and see if the herc crashes...
      • I'm also a bit concerned about whether the autofill will interfere with the herc's fill commands. I'm not sure when the autofill happens, and I also think the time drifts every day. But I do think that the timer for the autofill will be delayed every time the herc fills the InSb.
    • The norhof system has only 3 cm left! We'll need to fill today, if possible.
    • Greg is running through the purge stage of starting up the Cryomech LN2 generator.
  • June 9, 2017
    • Greg filled the Norhof tank about halfway (9cm), which was just about all the generator had made. He'll come back on Monday to fill it the rest of the way.
    • The O2 sensor is going off frequently. I wonder if it needs to be calibrated again.
  • June 12, 2017
    • Greg filled the Norhof - it's now full (~20cm).
  • June 14, 2017
    • LN2 did not fill - data are all bad on InSb
  • June 15, 2017
    • Greg went out to the lab and the Norhof was beeping. Starting pumping with the button; seemed to work okay. However, the temperature would reach ~-190C, and then bounce back up pretty quickly to ~-140C. This caused the Norhof to claim that the InSb was "warm and dry" (i.e., the green light), and the PStat was still active (although not pumping).
    •  After the first time it did this, I had Greg push the button again - that put the system into "Sleep" mode. Then I asked him to push the button again, and it started pumping, but with the same result. It reached ~-190C, but only stayed there for a few seconds, increasing reasonably quickly to ~-140C.
    • Either the InSb jacket is starting to lose pressure, or perhaps the dewar is so long that the temperature sensor isn't close enough to the LN2.
    • 3:40 pm EDT: InSb is cold (for now). ~20 minutes later, and the temperature is still ~-139C.
  • June ~25, 2017
    • Greg noticed that the solenoid on the LN2 generator wasn't functioning correctly (it was turning on and off too rapidly). New one purchased and sent to PA; should arrive July 6, 2017.

Back to top

July 2017

  • July 5, 2017
    • Planned OCO-2 target-mode maneuver today...
    • Power outage all morning (reported by SaskPower at 8:37 am CST). ETR 11:45 am CST.
    • Power restored around 11:30 am CST, but suntracker cannot initialize.
    • Norton did the following to get the tracker back:
      • 'SW Status Reinit' followed by a 'SW Status Time Warp' to open the enclosure.
      • Sun Tracker Connect - to get it out of stale
      • Sun Tracker Init - to retry the init
      • Sun Tracker Track to Programmed Sun Position
      • Sun Tracker Track by Diode
    • Then, I noticed that the InSb was saturating, so I followed J-F's instructions:
      • I went to "Hold" and issued the these commands:
        • IFS Direct DTC=0x4041
        • IFS Direct PGN=3
        • Followed by re-init and time warp.
      • That seemed to work. 2 hour spare before the OCO-2 target.
  • July 8, 2017
    • Went into ~InSb_Chilled in the evening. Perhaps because the Norhof was filled a few cm by Greg.
  • July 11, 2017
    • New solenoid installed in Cryomech, perhaps broken, perhaps not?
  • July 12, 2017
    • Rebooted webcam, as it hadn't updated since yesterday evening.
  • July 20, 2017
    • Swapped disks following the Feb 7, 2017 instructions.
    • Looked at O2 calibration procedure and the location of the sensor.
      • The O2 calibration procedure is as follows:
        • Enter into the “ Main Menu ” by pressing the ENTER button while on the " Home " screen.
        • Navigate the menu until you reach the menu heading “CALIBRATE”.
        • Under the “CALIBRATE” heading there is a list of enabled sensors.
        • Navigate to the appropriate sensor name ("LOCAL") and press the ENTER button to edit the parameter.
        • Adjust the value for 20.9% and press the ENTER button to store this parameter.
        • Press the ENTER button a second time to confirm the Calibration.
        • Press the ESCAPE button to return to the main menu.
      • Greg thought that perhaps moving the O2 sensor below the air conditioner would get it out of the blast of the AC. He thought that over the door would still get too much direct air from the AC.
  • July 28, 2017
    • Lab is getting very hot. The pump room is at 42C and the IFS is at ~35C. The laser isn't able to lock ("not ready"). System in hold.

Back to top

August 2017

  • August 2, 2017
    • InSb didn't fill; ran on InGaAs only most of the day.
  •  August 15, 2017
    • Norhof LN2 doser pump has been removed from Dewar as onsite maintenance is on hiatus as Greg Neufield is sick with pneumonia in hospital. Darrel Ernst (Environment Canada) removed the pump for us as checked on the Cryomech LN2 generator which is turned off and empty. 
  • August 16, 2017
    • IFS has switched to InGaAs measurements only as InSb is no longer cooled. 
  • August 23, 2017
    • IFS cannot communicate with herc. In hold until Greg can reboot the IFS.
  • August 24, 2017
    • Restarted the IFS by switching off the bottom switch, followed by the top switch. Greg noticed that the MIR and NIR lights were on before shutting off. This made me think to look at the chiller's liquid level, which Greg thought was a bit low. He added ~2 cups of de-ionized water (up to the max line).
    • After filling the chiller, Greg switched the top switch, followed by the bottom switch back on. It took several minutes to clear all the errors. The laser ready light was not on, but eventually did lock, and the system is now back in play (at ~1:25 pm EDT).
    • Greg will now start running the LN2 generator.
    • IFS_P around 2.5 hPa.
  • August 26, 2017
    • InSb measurements restarted mid-day.
  • August 28, 2017
    • Swapping disks using the procedure outlined on February 7, 2017.
      • The disk in the top slot of the enclosure is shipped back to Toronto
      • The disk in the bottom slot of the enclosure is moved to the top slot
      • The disk that was in the box is now in the bottom slot
      • Unfortunately, the monitor in the lab wasn't working, so instead of rebooting the computer, we shut it down, swapped disks, and then rebooted it.
    • Greg will be out of town from August 30 - September 8.

Back to top

September, 2017

  • September 18, 2017
    • There was a power failure overnight. It's quite cold outside early in the morning (-3 C).
    • The tracker did not reconnect after the power failure. Did the followin:
      • Sun Tracker Connect - to get it out of stale
      • Sun Tracker Init - to retry the init
      • Sun Tracker Track to Programmed Sun Position
      • Sun Tracker Track by Diode
    • This seemed to get us back into Solar mode.
    • Pumpdown is very slow.
  • September 25, 2017
    • InSb went into "not filled" state. A manual fill did the trick:
      • sw status InSb Fill
    • Pump down is now marginal.
    • Later in the day, InSb went into "not filled" state again.This may imply that the InSb detector jacket is no longer evacuated...
  • September 26, 2017
    • InSb went into "not filled" state after 4 hours of being filled. Suspect detector jacket evacuation.
    • Pump still slow. At ~8:40 pm CST, Greg looked at the tubing from the scroll pump to the instrument. Several connections were very loose (requiring ~4 turns of the wingnuts). He also looked in the first junction after the "T" and noticed a small amount of soot. We'll see if this improves pumpdown tomorrow.
      • Pump down was improved, but not fixed entirely. During the site visit, we'll have to check all junctions carefully.
  • September 29, 2017
    • InSb went into "not chilled" state around 6pm EDT.
  • September 30, 2017
    • InSb went into "not chilled" state around 12pm EDT; refilled InSb an hour or so later. Pumping took a long time. Suspect the "puck" is not secured to the InSb detector.

Back to top

October, 2017

  • October 1, 2017
    • InSb warmed up around 1:40 pm EDT
  • October 2, 2017
    • Webcam stopped working after 4:51 am on October 2. Requred two reboots to get it back online. Rebooted at ~1:30 pm EDT.
  • October 5-11: Site Visit
  • October 12, 2017
    • Greg visited the site and tried starting up the LN2 again. It is tripping the voltage monitoring relay (ABB CM-EFS.2). He says the outer white box lights up the green and orange lights but then the green turns off and is replaced by the red light (after the delay setting of the ABB relay). The ABB settings are 27 (>U), 18 (<U), 11 (Tv) and 300 (Range). I've called Brian Stoddard (Cryomech ext 227) to confirm what we should do.
    • This turned out to be fine?
  • October 16, 2017
    • LN2 pump was in active mode and couldn't fill using the herc. (Presumably Greg did something different. The norhof tank is now full.)
    • To take the pump out of "active" mode, send the following command on the herc: LN2 Pump Off
    • Then, the LN2 was able to fill using: Sw Status InSb Fill
  • October 27, 2017
    • Disk swapped. New monitor works!

Back to top

November, 2017

  • November 10, 2017
    • Webcam stopped connecting. Rebooted at 5:15 pm EST.
  • November 24, 2017
    • Webcam rebooted at 4:55 EST.
    • New pump delivered to ETL, but Greg will wait until next week to install.

Back to top

December, 2017

  • December 10, 2017
    • Greg filled the LN2 yesterday and I think he didn't put it back to standby mode, so it didn't fill with startup.
    • Sent LN2 Pump Off command before startup. Probably doesn't help, since it was already in sleep mode.
  • December 18-21, 2017: Site Visit to realign the spectrometer and install the new TriScroll pump
  • December 22, 2017
    • Rebooted the netcam.
    • Went into InSb not cold state around 11:10 am CST. Not sure why; the InSb was still at -122C at 7:35 pm CST...

Back to top

January, 2018

  • January 4, 2018
    • Webcam not connecting since Jan 3, 2018 around 20:00. Cannot get through using the Live Image page.
    • Seems to have resolved itself a few hours later.
  • January 8, 2018
    • Changed webcam time server from to (the TimeMachine inside the lab) on Jan 8, 2018 at 10:37 am EST. This seems to have fixed the ~10 minute time error.
  • January 12, 2018
    • Heat appears to be off in the lab: pump temperature is ~-17C, air temperature is ~-5C. It's -31C outside. Put system in hold to avoid running the pumps while the heat is off. Put in a call to Greg.
  • January 30, 2018
    • Tracker is stopped turning in azimuth after 12:00 local time. Has not turned since.
    • Looks like lots of ice on the enclosure.

Back to top

February, 2018

  • February 2, 2018
    • Greg and I swapped disks
      • sw status shutdown quickly
      • sudo shutdown
      • Greg went into bios but was unsure of how to get out, so we just turned off the computer and unplugged the batteries
      • swapped disks
      • plugged in the batteries and turned on the switch
      • IFSloop start
    • Greg knocked ice blocks off of the tracker, and the tracker was able to initialize!
    • InSb didn't fill:
      • To take the pump out of "active" mode, send the following command on the herc: LN2 Pump Off
      • Then, the LN2 was able to fill using: Sw Status InSb Fill
    • This did not stay cold: InSb gets cold and then immediately warms up...
  • February 3, 2018
    • LN2 still not staying cold. Not sure what's going on. Greg is out of town for ~10 days so we may get no InSb measurements until he returns.
  • February 12, 2018
    • Greg found the problem with the Norhof puck: the fill tube had fallen off and cannot be found. Greg found some other tubing in the lab that seemed to have no trouble getting cold with LN2, and used that instead. The InSb is now filling.
    • Greg also used his air compressor to start making new LN2. Orfeo is in contact with Cryomech to find a long-term solution to this problem.

Back to top

March, 2018

  • March 2, 2018
    • Finally reached the lowest limit of the Norhof system (-8.4cm) - No InSb measurements until we can generate some new LN2
    • This is interesting, however, because it means we can get another ~week of LN2 out of the Norhof than we previously thought. We use about 1 cm/day.
  • March 8, 2018
    • Kris filled the Norhof system after sunset: now at ~17 cm!
  • March 19, 2018
    • Kris swapped disks.
    • Kris' relative noticed that the capacitor in the Cryomech air compressor is likely a problem. There is an $18.50 replacement in Prince Albert (NGM 61B7D120850NCAC). We'll give it a try...

Back to top

April, 2018

  • April 3, 2018
    • Webcam not reachable since ~2200h April 2.
  • April 4, 2018
    • Webcam magically came back to life between April 3 at 11 pm EDT and April 4 at 6 am EDT.

Back to top

May, 2018

  • May ~9, 2018
    • "Phenocam" installed at ETL facing trees to the north of the lab.
  • May 14, 2018
    • Ran out of LN2; running on InGaAs only.
    • Replacement connector for the Cryomech ordered and shipped to PA; He fill kit also shipped.
  • May 22, 2018
    • Looked at O2 calibration procedure and the location of the sensor.
    • The O2 calibration procedure is as follows:
      • Take off 6 screws on front panel to access the buttons
      • Enter into the “ Main Menu ” by pressing the ENTER button while on the " Home " screen.
      • Navigate the menu until you reach the menu heading “CALIBRATE”.
      • Under the “CALIBRATE” heading there is a list of enabled sensors.
      • Navigate to the appropriate sensor name ("LOCAL") and press the ENTER button to edit the parameter.
      • Adjust the value for 20.9% and press the ENTER button to store this parameter.
      • Press the ENTER button a second time to confirm the Calibration.
      • Press the ESCAPE button to return to the main menu.
    • Disk swap following Feb 7, 2017 instructions
      • Yet again, could not enter bios upon reboot. Here are the current instructions for swapping disks:
        • Shut down IFSloop (sw status shutdown quickly)
        • Shut down the herc (sudo shutdown -b)
        • When shut down, turn off switch on herc power supply, then disconnect the 3-pin mil spec circular connector on the back to the batteries
        • Turn the key to unlock the enclosure
        • Remove upper disk, wrap in antistatic bag and place in pelican case
        • Put the new removable disk into the enclosure
        • Turn the key to lock the enclosure
        • Reconnect the 3-pin battery connector to the back of the herc power supply; flick on the power supply switch
        • After the herc boots up, restart IFSloop (remotely): IFSloop start
        • Check to ensure that the new removable disk has old data on it and that the sizes look approximately right on the IFSloop front screen
  • May 25, 2018
    • Looks like Kris added a bit of LN2 to the Norhof tank (maybe ~3 cm after cooling the dewar).
    • The Cryomech is still not running properly. Kris had to change out the capacitor again and the pressure inside the dewar reads 0.
  • May 28, 2018
    • Ran out of LN2 again.
    • Deciding whether to ship the Cryomech motor to Quincy in Saskatoon or if more tests can be done in-place.
  • May 30, 2018
    • We recorded our 1,000,000th slice!

Back to top

June, 2018

  • June 8, 2018
    • Unplanned power outage. Suntracker not connecting upon power restore.
    • Kris went to the lab. The reset button on the tracker was not enough to reset properly. This worked:
      • sw status shutdown quickly
      • shut off the power button on the tracker electronics, wait 15 seconds, flip it back on
      • IFSloop start
      • sw status suntracker init

Back to top

July, 2018

  • July 4, 2018
    • Planned power outage from 4:30 am - 8:30 am CST. Unfortunately, the power outage started right during the startup sequence. Can no longer access the instrument to check on the pumpdown status.
    • Power came back up at 9:30 am EDT (7:30 am CST). Hopefully this is not just a temporary power return. The system restarted, pump P was ~0.65 hPa. The tracker was stale, but this time, this sequence was able to restart the tracker:
      • sw status shutdown quickly
      • IFSloop start
      • sw status hold
      • sw status suntracker initialize
      • sw status reinit
  • July 7, 2018
    • Unplanned outage from 1 am CST (3 am EDT) to 10 am CST (noon EDT). Power was returned, but nothing is accessible (herc, webcam, etc.)
      • When power was restored to the building, the UPS powering the router and laser (and perhaps chiller) did not restore power to the devices. Kris had to hold down the power button for 1+ seconds before power was restored.
      • After shutting down ifsloop and restarting it, I could reinitialize the tracker, but the laser has not yet locked.
      • I've left the system in hold
      • Oddly, in IFSloop, the Failure is of the Scanner, and not the HeNe (which I think is just a 2-year timer, anyway). But the Laser Amplitude values are 0 V, which leads me to believe that the laser shutter is closed, because it has not locked. We'll see if I'm right...
      • The laser locked by 10:50 EDT, and I've taken the system out of hold. I don't remember the previous values, but the laser Amplitudes are now A: 11V; B: 8V; Offsets A: -1; B: -3.
  • July 20, 2018
    • Power outage not listed on saskpower's outage map:
    • I did not get to a computer in time to put the system into Hold. It was scanning when the power failed.
    • It's only 6:45 am CST, so I will call in 1.5 hours to see whether the power is out at the resort.
    • Power restored at 10:20 am EDT. Used sequence from July 4th power failure to get system back into Play.
  • July 22, 2018
    • Norton implemented the new interspersed low resolution scans at ETL. We now measure:
      • Solar (NSS=2)
      • SolarLR (NSS=2)
      • SolarLR (NSS=2)
      • Solar (NSS=2)
      • etc.
  • July 26, 2018
    • Power outage. Used sequence from July 4 successfully. Power was restored at 10:20 am EDT, but I didn't restart the software until 1:10 pm EDT. No data loss, since it's cloudy and the system went straight into Nap.

Back to top

August, 2018

  • August 1, 2018
    • Swapped disks using the Feb 7, 2017 method. It actually went into BIOS this time!
    • Noticed that the HeNe laser is now in fail. I presume (hope) that means it has just passed its "nominal" lifetime!
  • August 2, 2018
    • Brief power failure overnight. Everything came back up except the tracker was stale. Followed the July 4, 2018 instructions for bringing the tracker back online.
    • Lots of O2 errors, but the Zeno_T (which is actually the back room temperature where the O2 sensor is) is reading ~16C, so it's quite cool.
  • August 5, 2018
    • Lengthy unplanned overnight power failure. Followed the July 4, 2018 instructions to get tracker back online around 2:45 pm EDT. Lost a few hours of sunny data.
  • August 9 - 19, 2018 - Site Visit
  • August 18, 2018 - "Phenocam" disk and batteries swapped.
  • August 14, 2018
    • Planned power outage from 5 am - 9 am CST. Power restored at 11 am CST. Restarted IFSloop following the July 4, 2018 instructions. No data loss: cloudy.
    • Laser reset:
      • Total run time | 1 year, 335 days, 15 hours, 6 mn  | since Mon, 21 Feb 2011 17:32:20 | [ RESET ] |
    • Screen Shot 2018-08-14 at 1.42.16 PM.png
    • Laser Reset at 1:45:45 pm EDT
    • Joseph was unable to fix the chiller - it is off and disconnected.
  •  August 15, 2018
    • Rebooted webcam at 12:50 pm EDT.
    • Raging fires in BC have blanketed ETL in smoke. Here's the MODIS image:


  • August 18, 2018
    • LN2 procured and Norhof filled.
    • OCO-2 target: eastTroutLakeCa 2018-08-18 14:00:27 CST (2018-08-18 20:00:27 UTC)

  • August 31, 2018

    • Norhoff tank refilled sometime around the automated fill time, as the fill failed to complete. InSb stayed cold until ~3:00 pm CST. The first attempt to do a manual sw status InSb fill timed out, but the second attempt was successful.

Back to top

September, 2018

  • September 20, 2018
    • Rebooted webcam
  • September 26, 2018
    • Disk swap: Kris tried backspace and then delete, and that seemed to work to get into bios! Quit without saving.
    • Kris picked up the LN2 from Prince Albert and refilled the Norhoff

Back to top

October, 2018

  • October 4, 2018
    • Rebooted webcam
  • October 11, 2018
    • Jean-Francois noticed that the APT and AP2 settings for the SolarLR scans were set to 1000, instead of 800. This was getting perilously close to saturation (almost.
    • At ~3 pm EDT, added APT=800 and AP2=800 to, and recompiled the software.
      • Before change:
        •   17:46:40 Srvr: TMA: IFS Solar
            17:49:36 Srvr: TMA: IFS SolarLR
            17:50:18 Srvr: TMA: IFS SolarLR
            17:50:40 Srvr: TMA: IFS Solar
            --- LATEST MIN/MAX ---
            17:48:15 MXY1: 0.126 MNY1: 0.126 MXY2: 0.335 MNY2: 0.335
            17:48:15 MXY1: 0.126 MNY1: 0.126 MXY2: 0.335 MNY2: 0.334
            17:48:15 MXY1: 0.232 MNY1: 0.021 MXY2: 0.557 MNY2: 0.063
            17:49:58 MXY1: 0.365 MNY1: 0.032 MXY2: 0.889 MNY2: 0.102
            17:50:20 MXY1: 0.364 MNY1: 0.031 MXY2: 0.887 MNY2: 0.102
            17:50:57 MXY1: 0.232 MNY1: 0.021 MXY2: 0.557 MNY2: 0.062
            17:50:57 MXY1: 0.126 MNY1: 0.126 MXY2: 0.335 MNY2: 0.334
            17:50:57 MXY1: 0.126 MNY1: 0.126 MXY2: 0.335 MNY2: 0.335
      • After change, the ZPD values are much more consistent:
        • 18:59:27 Srvr: TMA: IFS Solar
          19:02:27 Srvr: TMA: IFS SolarLR
          19:02:52 Srvr: TMA: IFS SolarLR
          19:03:17 Srvr: TMA: IFS Solar
          --- LATEST MIN/MAX ---
          19:01:07 MXY1: 0.232 MNY1: 0.021 MXY2: 0.558 MNY2: 0.064
          19:02:32 MXY1: 0.233 MNY1: 0.021 MXY2: 0.558 MNY2: 0.063
          19:02:57 MXY1: 0.232 MNY1: 0.021 MXY2: 0.557 MNY2: 0.063
          19:03:19 MXY1: 0.232 MNY1: 0.021 MXY2: 0.557 MNY2: 0.063
  • Oct 23, 2018 - Kris dropped off the 3x20L dewers to be filled with LN2

Back to top

November, 2018

  • November 5, 2018 - Ice buildup on the enclosure was causing the tracker to reinitialize. I put the system in hold, but J-F reminded me that that wouldn't stop the azimuth drive from moving.
    • Kris has been contacted to clear off the ice. He's returning from Saskatoon tonight.
  • November 6, 2018 - Rebooted webcam.
    • Ice buildup remains. System still in hold, but it's completely overcast.
    • Tuesday night - noticed that Kris had cleared off the ice; put the container back in Play
  • November 7, 2018 - InSb fill not working - the detector didn't get cold on the first try, though the pressure in the Norhof seems okay (~75mb). Perhaps the lid popped up? Trying a second 'sw status InSb fill'.
    • Second fill worked (mostly). It timed out but the temperature is ~-175C, so good enough.
  • November 14, 2018 - Swapped disks using Feb 7 instructions. Backspace/DEL got Kris into BIOS! Escape to exit without saving.
    • Kris filled the Norhoff. Plan to get more LN2 on Dec 4.

Back to top

December, 2018

  • December 3, 2018
    • Another CAN error at bedtime. Ran SW Status Bedtime to run the lamp scans.
  • December 4, 2018
    • Filled 2x20L dewars with LN2 in Prince Albert (PA).
  • December 7, 2018
    • First sunny day in weeks, but the SolarRad was smaller than 0.8*model, so I changed location.h to make the scale factor 0.7*model. This worked, but the Rain/Wetness sensors are triggered, presumably because they have snow on them. Taking a few manual scans.
  • December 8, 2018
    • More CAN errors. The email says:
      • IFS Reports a CAN Error on AP2|APT|SRC|RES|VEL|PHR
      • In the logs:
        • MOT24=670:Device reported an error during command execution
    • MOT24 is the motor that controls the spherical mirror as it moves in and out of the solar beam
      • MOT24=0 commands the motor to move completely into the beam (i.e., for lamp scans)
      • MOT24=670 is completely out of the beam
      • MOT24=100 is moving ~90 degrees or ~15 mm.
    • This may be why we're not getting light during our lamp scans
  • December 9, 2018
    • CAN errors continue
  • December 10, 2018
    • To investigate the CAN errors, which seem to be for MOT24=670, venting the instrument so Kris can take a look following these instructions to vent):
      • sw status hold
      • sw status clear to vent
      • IFS direct VAC=2
      • when vented completely
      • IFS direct VAC=0
    • Kris took off the foam, disconnected the heaters, and removed the source compartment lid.
    • I was unable to send the MOT24=0, etc. commands through IFSloop using IFS direct MOT24=0. That resulted in an error. I tried accessing the IFS webpage through Firefox on my computer, but that didn't work, either. I then used Links on the webpage to access, and I was able to get through, but direct commands MOT24=0 and MOT24=100 weren't resulting in any changes to the motor. So instead, I used IFS direct SRC=104 to change to the NIR source, which also moves the motor. Kris and I could hear a grinding noise and some loud other noises. There are nice videos of the mirror motion here, here, and here.
    • Changing back to SRC=201 (sun) resulted in less grinding. We did this several times, and Kris filmed the movement of the motor. He could not find anything wrong with the belt - it was in place and intact, he noticed no missing gears. We confirmed that when the spherical mirror was in front of the OAP, it did not deflect the lamp light into the pinhole - it was off by ~1/8". Kris adjusted the mirror to centre it on the pinhole, but I'm sure that won't help the situation, except perhaps for tonight's cell measurements.
    • I will send an email to Gregor once I have the pictures from Kris, but it looks like we'll need a new motor.
    • When Kris was finished putting the lid/heaters/foam back on, I pumped down the IFS using sw status pump down
    • I expect some slow vacuum messages in the coming days, but hopefully the vacuum will hold!
  • December 24, 2018
    • CAN errors have reappeared. Lost bedtime scans.
  • December 25, 2018
    • Multiple CAN errors upon scanning. Started scanning eventually.

Back to top

January, 2019

  • January 9, 2019
    • Webcam rebooted at 1 pm EST.
  • January 10, 2019
    • Lots of CAN errors this morning, and none of the interferograms obtained could be processed by the overnight analysis:
      • Error: raw/flight/190110.1/scan/b1263589.0 has bad magic
        Error: raw/flight/190110.1/scan/b1263590.0 has bad magic
        Error: raw/flight/190110.1/scan/b1263591.0 has bad magic
        Error: raw/flight/190110.1/scan/b1263592.0 has bad magic
  • January 11, 2019
    • On J-F's recommendation, I sent a reset command: IFS reset SW at ~10:22 EST. Unfortunately, I did this just seconds before startup was about to begin. Hopefully that doesn't mess up the startup sequence.
    • Rebooted webcam at ~10:20 EST.

Back to top

February, 2019

  • February 6, 2019
    • O2 sensor error. I believe this is due to cool temperatures inside the lab (~18C). Outside, it's -35C.
  • February 7, 2019
    • OCO-2 target today:eastTroutLakeCa 2019-02-07 13:30:09 CST (2019-02-07 19:30:09 UTC)
  • February 11, 2019
    • Swapped disks using Feb 7 instructions. Backspace/DEL got Kris into BIOS for the third time in a row! Escape to exit without saving.
    • Very low on LN2; should get more tomorrow.
  • February 12, 2019
    • Finally ran out of LN2. Kris went down to PA today to pick up 60L, but the last time we filled the tanks was Dec 4, 2018!
    • Based on my records, we have filled the tanks with 4 week lag, 6 week lag, and 10 week lag. Only with the 10 week lag did we lose (1) day of InSb measurements. I'm guessing in the summer, the LN2 might boil off faster, so I think 8 weeks is probably the right pace to refill the tanks.
    • Kris filled the Norhof and I filled the InSb at ~9:30 pm EST to give it a head start for tomorrow morning's fill
  • February 21, 2019
    • Started getting CAN errors again about "AP2|APT|SRC|RES|VEL|PHR". I'm still assuming that it's the SRC motor.

Back to top

March, 2019

  • March 15, 2019
    • A few CAN errors upon starting to scan.

Back to top

Tag page

Files 4

FileSizeDateAttached by
A movie of the motion of the MOT24 motor when setting SPT=104 (NIR)
1029.19 kB17:26, 10 Dec 2018dwunchActions
A movie of the motion of the MOT24 motor when setting SPT=201 (sun)
1356.16 kB17:26, 10 Dec 2018dwunchActions
The motion of the MOT24 motor
1366.89 kB17:26, 10 Dec 2018dwunchActions
Sun tracker full report just before changing PFACT=35000
81.46 kB08:38, 11 Jan 2017dwunchActions