A week-long watch at the Taiga Experiment site

[Cross-posting to news:sci.physics.particle, news:comp.unix.misc
and news:rec.travel.asia for reasons. I’m going to be checking
all of them from time to time over the next few weeks, so feel
free to post your followups to any of them as appropriate.]

So, my employer decided to send me, as part of a team of two,
on a week-long watch over the Taiga Experiment [1, 2]. The
easier one, as during this part of the Lunar cycle the air
Cherenkov light sensors (that make up both Tunka-133 and the
newer Tunka-HiSCORE instrument, as well as IACT) are effectively
useless and require virtually no attention on the part of
the team on duty, leaving us with only Grande and Rex to keep
an eye on.

We took a plane to Irkutsk and, after a brief stay at a hotel,
were picked up from there and traveled by car to the site –
roughly at the midpoint between the Shuluta ulus and the
Tory village proper, in a bend of the Irkut River. [3]

We’ve arrived there January 26th, and our shift began after the
previous team left the site on 28th.

Initially, we took residence at the guest house. A particular
problem arose due to the unfortunate reliance of the building’s
heating system on electric power (for the heater’s water pump.)
Of which there was an outage just the night prior – combined
with the temperature reportedly reaching −47 °C. (See [4].)

The end result was that even after heating the rooms with all
the means available it was still a rather chilly night. Not to
mention the inconveniences of the frozen plumbing, which made me
rely on the sesame seed bars I had mind to buy several back home
for an occasion like this. (I generally prefer to eat from a
freshly washed bowl and with a freshly washed fork, yes.)

[1] http://taiga-experiment.info/
(May as well be called “Taiga Observatory,� I suppose, as there’re
hardly any conditions that can feasibly be controlled.)
[2] http://en.wikipedia.org/wiki/Tunka_experiment
[3] http://openstreetmap.org/#map=13/51.80/103.07
[4]
http://earth.nullschool.net/#2018/01...c=103.07,51.80

Speaking of which, if you’re going to stay there, you should
consider bringing more or less everything you need with you,
like: one or more sets of clothes, food, slippers, a dishcloth,
a soap bar, a pen or pencil and a notebook, skis for faster
travel across the sensors array (and to the village when
necessary) during the winter, etc. You may find some of that
left by one of the previous teams, but you may as well not.
(There’s a bicycle on site, though, which can be useful should
your visit happen to be in summer.)

And keep in mind that the closest convenience store is about an
hour walk away.

Apart from the seed bars, I had a couple cans of corn, as well
as a decent amount of canned white kidney beans (in tomato
juice), a few packs of dried sesame seeds, some dried lemons and
apricots, green tea, and turmeric powder in my luggage. Also,
I took time to buy more canned beans, three 400 g jars of mashed
marrows, an 1.5 l bottle of natural mineral water (or, if you so
prefer, 1% carbonic acid solution), two 800 g packs of
buckwheat, some bananas, and a loaf of white bread. (I was
unable to find any decent rye bread in the store we’ve visited
on our way to the site, alas.)

My idea was to use the microwave oven at the site to prepare
buckwheat, but the one at the lab’s kitchen looked rather
“used,� so to say, and I only partially succeeded in cleaning it
up to the standards I’ve grown used to. I guess I’m going to
leave it all to the next team.

Thankfully, in the two village’s convenience stores I’ve found
more canned beans and corn, as well as mineral water, canned
kelp, and oranges. (Per my experience, citrus fruits help
prevent cold, as does the mineral water I usually buy.)

The watch itself was rather uneventful so far, mostly consisting
of checking that the data collection program stops correctly at
about 23:55 UTC (07:55 local time), moving the data (over
160 GiB at a time typically) to the archive, then starting the
‘resave’ process to find and store simultaneous events. (About
5 GiB.) Then, starting at about 00:02 UTC, we’re reseting the
instruments’ hardware (the so-called VMEs), start the daily
sensors’ check and, after its completion, start the data
gathering software anew.

(It looks like this part can benefit from some automation, IMO.)

Also, we keep an eye on the real-time “health report� (as
gathered via a dedicated XBee radio network) from the
Grande stations and the Tunka-133 clusters they’re connected
through, mainly to take an action should the equipment begin to
overheat or overcool.

On 30th, about 02:23 UTC, the routine was, however, broken by an
hour and a half long power outage, after which my colleague had
to walk 400 m to the data center to ensure proper startup of
some of the boxes there, while I brought up the boxes at the
lab, which are used mostly as SSH terminals. (The only system
to survive the outage on UPS power was ‘meteo’ – the Raspberry
Pi board used to record data from the lab’s personal weather
station. I’ve reconnected to the GNU Screen session there as
soon as I’ve figured out how to $ ssh to it.)

The instruments’ hardware cooled down well beyond its specified
range (which is, AIUI, 15 ÷ 35 °C), so the automated heaters
were already all on by the time I’ve started the radio
monitoring software.

The problem is that, however, the heaters are quite low-power
and it takes a while for them to bring the temperature back to
15 °C from below 5 °C. The trick we were instructed beforehand
to use in this case is to configure the hardware to start the
payload equipment when the temperature is over 1 °C instead –
and let it heat itself. (That is: the measurement hardware
produces more heat during its normal operation than its
dedicated heaters. Am I the only one to find it somewhat
surprising?)

At the same time, three of the Grande stations seem to produce
way too much heat for the automated coolers to handle: the
reported temperature is about 50 °C higher there than that of
the outside air. That’s hardly an issue when the latter is
below −15 °C, but what’s going to happen when it goes above
−10 °C [5]?

[5]
http://earth.nullschool.net/#2018/02...c=103.07,51.80

About the only issue we’ve encountered was that the system that
the sensors are synchronized with failed to obtain current time
from the GPS receiver it’s (supposed to be) connected to, so we
ended up synchronizing it (AIUI) with the host the data
collection process runs on – which resulted in about 16 s
inaccuracy if the clock on the only system that we have access
to here and that uses NTP (and seemingly successfully, per
$ ntp -pn) is to be believed. We were instructed to ignore
this difference.

(As to why not /every/ box here has NTP configured I cannot
fathom. Especially given that most seem to have Ubuntu or some
other Debian derivative installed.)

Another curiosity is that virtually all the instruments-related
software used here seem to require ‘root’ privileges to run.
I have a feeling that that could’ve been avoided with some
additional Udev rules.

That’s about all for now, although I hope to post more on this
later, perhaps when I’ll get back home. The car is expected to
bring us to Irkutsk the tomorrow (2018-02-03) evening, after
which I’m going to take a train back home.

--
FSF associate member #7257 http://am-1.org/~ivan/