Calib workflow

[chiarazampolli]

@davidrohr

Needs AliceO2Group/O2DPG#373

Updated after some changes for TRD: also AliceO2Group/O2DPG#375 is needed

[davidrohr]

fine with me, just consider my comment below.

[chiarazampolli]

Hello @martenole , @davidrohr ,

The last commit should enable the calibration in the main workflow, if needed (see discussion in #8668 (review)).

[chiarazampolli]

Back to WIP because it needs AliceO2Group/O2DPG#375

[chiarazampolli]

@martenole , in case you want to test, you need to run with:

WORKFLOW_PARAMETERS="CALIB,${WORKFLOW_PARAMETERS}"

and define the CCDBPATH.

[martenole]

Hi Chiara,
I just tried and ran
WORKFLOW_PARAMETERS=CALIB,CALIB_LOCAL_INTEGRATED_AGGREGATOR WORKFLOWMODE=print $O2_ROOT/prodtests/full-system-test/dpl-workflow.sh
But the workflow does not by default contain any of the aggregator processes. It seems setenv_calib.sh is not sourced anywhere, right?

[chiarazampolli]

Hi Ole,

We changed a few things this morning, and I did not have the time to try yet. So it might well be that something needs to be fixed now. Sorry for the inconvenience.

Chiara

[chiarazampolli]

Hi @martenole ,

Yes, I had removed sourcing of all files, not just checking whether there were sourced :-)
Sorry.
You can retry, but I still have not tested myself, I warn you :-)

Chiara

[chiarazampolli]

For me the following works (with master of O2DPG) - in the following, setenv_extra sets export WORKFLOW_DETECTORS=ITS,TPC,TOF,FV0,FT0,FDD,MID,MFT (no TRD!!!) and extra options to the reco:

1. mode in which we run locally with the calib processes in the main workflow:

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_LOCAL_INTEGRATED_AGGREGATOR IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

--> builds the workflow appending to it the calibration procedures which otherwise should be run on the aggregator. No output nor input proxies:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TOF,ITS-TPC-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0 --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-calibration-mean-vertex-calibration-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 | \
o2-calibration-tof-calib-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 | \
o2-calibration-tof-calib-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 | \
o2-calibration-tof-diagnostic-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --tf-per-slot 26400 | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 

2. mode in which we want to send the calibration specs to an output proxy, to then be taken by the aggregator:

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_PROXIES IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

---> works correctly, the calib processes are not appended, but the output proxy is and the data specs are correctly built:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0
CALIBDATASPEC_CALO =
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,rateLogging=1,transport=zeromq,address=tcp://localhost:30453"
...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TOF,ITS-TPC-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0 --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-dpl-output-proxy --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,rateLogging=1,transport=zeromq,address=tcp://localhost:30453" | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 

3. aggregator run locally:

WORKFLOWMODE=print WORKFLOW_PARAMETERS=CALIB_PROXIES WORKFLOW_DETECTORS=ITS,TPC,TOF,FV0,FT0,FDD,MID,MFT CCDBPATH=none /home/zampolli/SOFT/alibuild/ali-o2-dev/O2/prodtests/full-system-test/aggregator-workflow.sh

---> all correct: specs build correctly, raw proxy as well, calib processes appended:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0
...
o2-dpl-raw-proxy --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,rateLogging=1,transport=zeromq,address=tcp://localhost:30453" |
o2-calibration-mean-vertex-calibration-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 |
o2-calibration-tof-diagnostic-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --tf-per-slot 26400 |
o2-dpl-run --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 -b

(ccdb-populator not attached as we don't pass CCDB_POPULATOR_UPLOAD_PATH).

[chiarazampolli]

Considering AliceO2Group/O2DPG#390 (comment), I am now removing [WIP].

[chiarazampolli]

Hello @shahor02 , @davidrohr , @martenole ,

As discussed via email, I would merge this PR (I need @shahor02 or @davidrohr to do so). It should not affect the data taking, as we have not yet enabled calibrations, but it will help to test more.
I already merged everything in O2DPG, this is the last bit.

Thanks,

Chiara

[ktf]

Can you check if the fullCI failure is unrelated, though?

[chiarazampolli]

Hi @ktf ,

I don't see the full logs, but to me it looks like problems in compilation (memory?) and then in the AODs. So not related to this PR. Can you also take a look?

Chiara

[shahor02]

sim_challenge fails in aod-producer, unrelated, merging