Cloudnet data submission tool

Installation

Linux/macOS

If you have python and pip installed (python >= 3.8), run the command:

pip install cloudnet-submit

After that, you can use cloudnet-submit command to run the program.

Windows

If you are using Windows Subsystem for Linux, then the aforementioned Linux installation should work.

If you are using windows command prompt and have python and pip installed, you can install cloudnet-submit with pip:

pip install cloudnet-submit

Test if the cloudnet-submit command works:

cloudnet-submit --version

If not, you can use an alternative way to run the program:

python -m cloudnet_submit --version

Getting started

Configuration file

Generate a configuration file:

cloudnet-submit --generate-config

This will generate a file named cloudnet-config.toml in your working directory.

Update your submission credentials in the user_account section, and update site, instrument, instrument_pid and path_fmt fields to match your instrument/model setup. Remove unnecessary instrument and model sections.

Possible values for site are listed in https://cloudnet.fmi.fi/api/sites. You can only upload to sites your user account has permissions for.

Possible values for instrument are listed in https://docs.cloudnet.fmi.fi/api/upload-file-types.html.

You can find values for instrument_pid from the list of instruments. If your instrument does not have PID yet, please fill the form first.

# cloudnet-config.toml
[user_account]
username       = "alice"
password       = "alicesSecretPassword"

[[instrument]]
site           = "hyytiala"
instrument     = "rpg-fmcw-94"
instrument_pid = "https://hdl.handle.net/21.12132/3.191564170f8a4686"
path_fmt       = "/data/hyytiala/rpg-fmcw-94/%Y/%m/%y%m%d_*_P10_ZEN.LV1"

[[instrument]]
# you can have additional sections for the same instrument
# e.g. for different path
site           = "hyytiala"
instrument     = "rpg-fmcw-94"
instrument_pid = "https://hdl.handle.net/21.12132/3.191564170f8a4686"
path_fmt       = "/home/alice/hyytiala/rpg-fmcw-94/%y%m%d_*_P09_ZEN.LV0"

[[instrument]]
site           = "granada"
instrument     = "chm15k"
instrument_pid = "https://hdl.handle.net/21.12132/3.77a75f3b32294855"
path_fmt       = "/data/granada/chm/%Y-%m/%Y%m%d_Granada_CHM170119_*.nc"

[[instrument]]
site           = "kenttarova"
instrument     = "halo-doppler-lidar"
instrument_pid = "https://hdl.handle.net/21.12132/3.a93d1483f10742ff"
path_fmt       = "/data/kenttarova/halo/system_parameters_*_%Y%m.txt"
periodicity    = "monthly"

[[model]]
site           = "hyytiala"
model          = "ecmwf"
path_fmt       = "/data/hyytiala/ecmwf/%Y/%Y%m%d_hyytiala_ecmwf.nc"

# You can use proxies (optional)
[network.proxies]
http  = "http://10.10.1.10:3128"
https = "http://10.10.1.10:1080"
# Alternatively, You can define proxies as environment variables
# HTTP_PROXY and HTTPS_PROXY
# see: https://requests.readthedocs.io/en/latest/user/advanced/#proxies

cloudnet-submit will look for files specified in the path_fmt field for a given measurement date.

Use the following format codes:

Directive	Meaning	Example
%Y	Year with century	0001, ..., 2023, ..., 9999
%y	Year without century (zero-padded)	00, ..., 23, ..., 99
%m	Month (zero-padded)	01, 02, ..., 12
%d	Day (zero-padded)	01, 02, ..., 31

You can also use wildcard character * in path_fmt field.

By default, cloudnet-submit expects the cloudnet-config.toml file to be in your working directory. You can also use --config to specify another location for the config file:

cloudnet-submit --config /path/to/your/config.toml

Periodicity (advanced)

In some rare cases, the file does not contain data from a single day, but rather from an entire month. For instance, HALO Photonics Doppler lidars generate system_parameters.txt files on a monthly basis. In such cases, it is possible to modify the periodicity setting.

The allowed values for periodicity are daily (the default) or monthly. This determines whether the file contains data from a single day or an entire month. If a monthly file changes, e.g. new data are appended to the file each day of the month, the file will be resubmitted and replaces the previously submitted data.

Note that it is allowed that the file contains data from shorter period that one full day (e.g. hourly).

Usage

By default, cloudnet-submit submits data from the past three days.

Use --dry-run to list files that would be submitted:

cloudnet-submit --dry-run

Submit data to the Cloudnet data portal:

cloudnet-submit

You can also set the number of days to be submitted (including today):

cloudnet-submit --last-ndays 5

You can also specify a date you want to submit:

cloudnet-submit --date 2022-06-21

Or a list of dates:

cloudnet-submit --date 2022-06-21 2022-05-01

Or a range of dates:

cloudnet-submit --from-date 2022-05-01 --to-date 2022-06-24

See all the options:

cloudnet-submit --help

Feedback and contact

• Bugs, feature requests, documentation: Create an issue on Github
• Or just send us mail :)