Only partially working out of the box on my ubuntu laptop

[goatchurchprime]

Cloned the repo, disabled any services attaching to the ports 1880, 1883 and 3000, then:

Do this with conda activated

pip install docker-compose
docker-compose up -d
big install
docker-compose logs -f

MQTT works (with these commands in different windows):

mosquitto_sub -h localhost -v -t "#"
mosquitto_pub -h localhost -t "hi" -m "there"

Node-RED now appears to work:

http://localhost:1880
You need to use "mosquitto" as the broker, not localhost.

The influxDB node is missing, so I can't make database data.

Jupyter also didn't come live for a while on http://localhost:8888/lab with password mingstack. Then does after a few minutes. It starts out with no syntax highlighting and nothing will run. Now it does.

Grafana comes in on http://localhost:3000 default user/pass=admin/admin and then you are forced to change it.

I was hoping to run the following to connect+download from the grafana database using the following python code:

import influxdb
client = influxdb.DataFrameClient("192.168.43.1", 8086, "root", "root", "balena-sense")
k = client.query("select * from fridgepower order by time desc limit 10")

---

Some needs

• Where is the /data file connected to externally that the jupyter lab could connect/load/save from?
• install influxdb into python
• Inclusion of node-red serial
• Inclusion of pyserial in jupyter
• Inclusion of jupyter-micropython-kernel
• Provision the system with an initial flow (eg doing random numbers) saving to an initial database with an initial dashboard.

[MatthewCroughan]

This was due to Julian being on an older version of docker (docker 17.03), not much could really be done. It might be worth putting some prerequisite dependencies in the Readme, such as what docker version should be in use and a link to how to install Docker.

[ajlennon]

@goatchurchprime can you retest this with whatever Docker version @MatthewCroughan says you need and revise whatever outstanding issues there are?

[MatthewCroughan]

@ajlennon we already went through this in telegram, I had him uninstall all old versions of Docker then run https://get.docker.com/.

Should mark this issue as resolved once we document this in the README.

[ajlennon]

There are a bunch of other issues up there about things that need to be installed by default (like the InfluxDB nodes)

[goatchurchprime]

Things are getting close for an amazing package that knocks people out. I'm looking for an instant installation of the full stack pre-loaded with all the tools on the go. There may be multiple installs of the MING stack on personal computers, that can push data around via mosquitto brokers, until it's understood that one should do it with one lab install (properly backed up) that everyone connects to.

Having been around to some researchers, we need things the serial-ports to work, because that's how lab-equipment is often operated. Also, there are some systems that only have windows interfaces (through the same serial-COM ports) that dump their data into a file. Node-red should be able to tail those files and adjust its control behavour. The jupyter-influxDB integration makes it possible to pick up and analyse the data from the experiments when they are run.

Chris says that wsl2 means we should be able to get docker in linux on windows almost natively.

---

The notes were made after we'd sorted out the docker version:

• Docker version 19.03.5, build 633a0ea838

Docker-compose only runs when I have conda activated, then:

• docker-compose version 1.25.4, build unknown

Overcoming missing components (that should be loaded by default)

Node-red:
Installing node-red-contrib-influxdb using manage pallette

This should have a couple of example flows to kick it off, like a repeating inject node that passes the following javascript payload into the influxdb database:

• msg.payload = [ { "xxx":Math.random()*2+1}, {"yyytagtype":"zzztag"} ];

I am using database=aaadatabase, and measurement=bbbmeasurement to make it clear that those names are user-defined and not mistaken for keywords.

Influxdb:
This should boot up with one default database in it. The http://localhost:8086 link, which is supposed to be a dashboard for it, gets 404 page not found. I don't have the command line login to work it. See Jupyter section for how I added a database.

Jupyter notebook:

• The influxdb connection

[install the module]
import subprocess
print(subprocess.check_output(["pip3",  "install", "influxdb"]).decode())
[now restart kernel]

import influxdb
client = influxdb.DataFrameClient("influxdb")
client.create_database("aaadatabase")

[wait for the nodered to add a bunch of random values]
c = influxdb.DataFrameClient("influxdb", database="aaadatabase")
x = c.query("select * from bbbmeasurement order by time")
z = x["bbbmeasurement"]
display(z)

from matplotlib import pyplot as plt
plt.plot(z.xxx)

• The serial port connection:

import subprocess
print(subprocess.check_output(["pip3",  "install", "pyserial"]).decode())

import serial, serial.tools.list_ports
lp = list(serial.tools.list_ports.grep(""))
lp.sort(key=lambda X: (X.hwid == "n/a", X.device))  # n/a could be good evidence that the port is non-existent
ports = [x.device  for x in lp]
print(ports)

• Micropython

If we can get serial connection, we might as well install the micropython kernel. There might be some other kernels to add in, eg Julia, which is a very popular language among academics.

Other node-RED components

Install node-red-node-serialport (this won't work till we solve it for the Jupyter container).

Install node-red-dashboard (hooray! this works).

We should have a default node-flow showing off the dashboard with a couple of dials on it because getting it going

[ajlennon]

Yeah agreed. There's a bunch of stuff that should be installed out of the box on node-red

[MatthewCroughan]

@ajlennon I disagree, it should be an optional or build argument, it should be a feature that can be activated at build time, but why would we install things by default for the user rather than letting them import their own stuff?

[MatthewCroughan]

We could also make an additional project that uses this project as a submodule, MING then becomes a base layer that is pulled in, MING with absolutely no addons. We then build features on top.

[MatthewCroughan]

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.

The instructions you have provided are more than enough to get on with this. Thanks.

[MatthewCroughan]

Influxdb:
This should boot up with one default database in it. The http://localhost:8086 link, which is supposed to be a dashboard for it, gets 404 page not found. I don't have the command line login to work it. See Jupyter section for how I added a database.

The default database name is ming_default which is specified in the docker-compose file:
https://github.com/DynamicDevices/ming/blob/4c77867e654a5bb35ee6d450e5d0c6d5ca3fa8b8/docker-compose.yml#L19

[ajlennon]

@ajlennon I disagree, it should be an optional or build argument, it should be a feature that can be activated at build time, but why would we install things by default for the user rather than letting them import their own stuff?

The value of Docker / MING is that users can get going immediately without faffing about or having to understand what to configure or turn on or which buttons to push.

I see no downside in activities that people are likely to want to undertake (e.g. using serial ports) having support installed and ready to go...

[ajlennon]

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.

The instructions you have provided are more than enough to get on with this. Thanks.

Not at all keen on forking MING. There's no downside to having all this installed and ready to go in the standard build

[goatchurchprime]

In Node-RED set the Influx node to: influxdb:8086, database: ming_default

In Grafana we need a standardized password. And then to be provisioned with a link to the ming_default database. Then it needs a panel view into the

Use as a function to seed the database with:
Math.sin(Math.PI2(((new Date()).getTime()%60000)/60000) + 1 + Math.random()*0.2;

May seed with two functions to show how values can interact in the same table.

Where are these /data directories? Can they get shared with the outer file system?

Jupyter-lab has a Terminal window, so we can install anything we need there, though it doesn't do the control-Up history commands.

Also can't run apt-get install, because I want Git to see if I can get micropython.

Version of JupyterLab may be out of date. My personal one says this on startup:

Build Recommended
JupyterLab build is suggested:

jupyter-threejs needs to be included in build
jupyterlab-datawidgets needs to be included in build

This could mean that the ipyvolume examples might work:
https://ipyvolume.readthedocs.io/en/latest/

I wonder what happens if I do this docker-compose up a second time:
It just says it's all up to date.

May need the cheatsheet for closing down and opening up and updating the system.

Also, I'll need scripts to help backup the different components, etc. The kind of guidance and workflows.

---

other keep safe commands:

Clone https://github.com/DynamicDevices/ming

In the directory, do:

docker-compose up -d

This shows what's up and running:

docker ps

To list the images:

docker images

To log what's going on, do:

docker-compose logs -f

To kill all docker containers and images:

echo "Nuking docker containers, images, and volumes." && docker stop $(docker ps -aq) || true && docker rm $(docker ps -aq) || true && docker rmi -f $(docker images -aq) || true && docker volume rm $(docker volume ls -q) || true

[ajlennon]

Also can't run apt-get install, because I want Git to see if I can get micropython.

Why not? Have you tried apt-get update first?

[ajlennon]

@goatchurchprime the hostname to use for servers is the name of the container (as you say above) rather than localhost. This is correct behaviour and should probably be written down somewhere