Use IPC for local calib connections #390

davidrohr · 2022-05-11T16:55:15Z

@chiarazampolli : Haven't tested this yet, but this is what I had in mind for the proxy channels:
So for testing locally, but in 2 separate shells, i.e. without CALIB_LOCAL_INTEGRATED_AGGREGATOR, one has to set both CALIB_PROXIES and CALIB_LOCAL_AGGREGATOR. Then it will use SHM transport with a socket file based on the channel name.
If CALIB_PROXIES is set, but CALIB_LOCAL_AGGREGATOR is not, i.e. when we run online on the EPN, we use zeromq transport and leave the address empty, to be filled by FairMQ automatically, in the same way we are doing for the calib workflows at P2 right now already.

chiarazampolli · 2022-05-11T20:59:11Z

Ciao @davidrohr ,

Seems fine to me, thanks!

EPN:

To run locally:

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_LOCAL_INTEGRATED_AGGREGATOR IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

adds the calib processes to the workflow:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TRD,ITS-TPC-TRD,TPC-TOF,ITS-TPC-TOF,TPC-TRD-TOF,ITS-TPC-TRD-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0,TRD --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-tpc-scdcalib-interpolation-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0  --disable-root-input --pipeline tpc-track-interpolation:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-calibration-mean-vertex-calibration-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 | \
o2-calibration-tof-calib-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 | \
o2-calibration-tof-calib-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 | \
o2-calibration-tof-diagnostic-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --tf-per-slot 26400 | \
o2-calibration-trd-vdrift-exb --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0

Aggregator on the same node, transport with a socket file based on the channel name.

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_PROXIES,CALIB_LOCAL_AGGREGATOR IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

produces:
CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1"

and then in the workflow:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TRD,ITS-TPC-TRD,TPC-TOF,ITS-TPC-TOF,TPC-TRD-TOF,ITS-TPC-TRD-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0,TRD --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-tpc-scdcalib-interpolation-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0  --disable-root-input --pipeline tpc-track-interpolation:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-dpl-output-proxy --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1" | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0

Aggregator on a different node, with ZMQ:

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_PROXIES IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

Produces:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0

and

PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=zeromq,rateLogging=1"

in the workflow:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TRD,ITS-TPC-TRD,TPC-TOF,ITS-TPC-TOF,TPC-TRD-TOF,ITS-TPC-TRD-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0,TRD --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-tpc-scdcalib-interpolation-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0  --disable-root-input --pipeline tpc-track-interpolation:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-dpl-output-proxy --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=zeromq,rateLogging=1" | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0

For the aggregator:

when on the same node:

WORKFLOWMODE=print WORKFLOW_PARAMETERS=CALIB_PROXIES,CALIB_LOCAL_AGGREGATOR WORKFLOW_DETECTORS=ITS,TPC,TOF,FV0,FT0,FDD,MID,MFT CCDB_POPULATOR_UPLOAD_PATH=http://ccdb-test.cern.ch:8080 /home/zampolli/SOFT/alibuild/ali-o2-dev/O2/prodtests/full-system-test/aggregator-workflow.sh

produces:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1"
Workflow command adding aggregator:
o2-dpl-raw-proxy --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1" |
o2-calibration-mean-vertex-calibration-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 |
o2-calibration-tof-diagnostic-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --tf-per-slot 26400 |
o2-calibration-ccdb-populator-workflow --ccdb-path http://ccdb-test.cern.ch:8080 --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-dpl-run --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 -b

on a different node:

WORKFLOWMODE=print WORKFLOW_PARAMETERS=CALIB_PROXIES WORKFLOW_DETECTORS=ITS,TPC,TOF,FV0,FT0,FDD,MID,MFT CCDB_POPULATOR_UPLOAD_PATH=http://ccdb-test.cern.ch:8080 /home/zampolli/SOFT/alibuild/ali-o2-dev/O2/prodtests/full-system-test/aggregator-workflow.sh

produces:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=zeromq,rateLogging=1"

o2-dpl-raw-proxy --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=zeromq,rateLogging=1" |
o2-calibration-mean-vertex-calibration-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 |
o2-calibration-tof-diagnostic-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --tf-per-slot 26400 |
o2-calibration-ccdb-populator-workflow --ccdb-path http://ccdb-test.cern.ch:8080 --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-dpl-run --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 -b

Chiara

chiarazampolli · 2022-05-11T21:03:45Z

Before merging, let me test it in run mode.

chiarazampolli · 2022-05-11T21:14:13Z

We need this one:
davidrohr#1

With that, I am getting errors on the aggregator:

[65268:aggregator-proxy-barrel]: [23:09:46][INFO] aggregator-proxy-barrel[0]: in: 1 (0.110372 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:47][INFO] aggregator-proxy-barrel[0]: in: 0 (0 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:48][INFO] aggregator-proxy-barrel[0]: in: 0 (0 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:49][INFO] aggregator-proxy-barrel[0]: in: 0 (0 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 0 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 2 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 3 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 4 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 5 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 6 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 7 does not follow the O2 data model, DataHeader missing
[65269:internal-dpl-ccdb-backend]: [23:09:49][ERROR] Dummy creation time is not supported for CCDB objects. Setting creation to last one used.

I need to investigate.

chiarazampolli · 2022-05-11T21:36:21Z

Must have been some shm file left hanging, because cleaning up did the trick.
Now everything works fine.
There is one error left, but not related to the setup but to TOF (@noferini - I will take a look):

97447:calib-lhcclock-calibration]: Error in <TFile::TFile>: file /archive/O2/CalibWithBeams/aggregator/localTimeSlewing.root does not exist

But it is harmless.
With 2 EPNs (local) and the aggregator, 100 TFs per EPN were processed correctly. CCDB entries also were uploaded (e.g. GLO/Calib/MeanVertex). There are other small issues with the TOF diagnostic calib (validity of the object). The other calibrations did not get enough statistics.

So I would merge.

chiarazampolli · 2022-05-13T06:52:48Z

Tests ok, merging.

davidrohr added 2 commits May 11, 2022 18:47

Fix misleading output

b8512dc

Use IPC for local calib connections

df8002e

davidrohr requested review from chiarazampolli and shahor02 as code owners May 11, 2022 16:55

chiarazampolli mentioned this pull request May 11, 2022

Calib workflow AliceO2Group/AliceO2#8736

Merged

chiarazampolli approved these changes May 13, 2022

View reviewed changes

chiarazampolli merged commit d892f82 into AliceO2Group:master May 13, 2022

chiarazampolli mentioned this pull request May 13, 2022

Fix option to pass the name of the proxy/channel #392

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use IPC for local calib connections #390

Use IPC for local calib connections #390

Uh oh!

davidrohr commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Use IPC for local calib connections #390

Use IPC for local calib connections #390

Uh oh!

Conversation

davidrohr commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 11, 2022

Uh oh!

chiarazampolli commented May 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants