Add XtdfPulses

[philsmt]

This component exposes the pulse informations from the image.pulseId field in XTDF data via the well known interface. I'm not yet quite sure how this should look like, so this is a first draft based on an inspiration I had today.

Apart from the obvious, it offers two special features: The ability to anchor to another pulse pattern and cellId in its pulse index.

As XTDF detectors unfortunately are the biggest offender of using the term pulseId for something completely arbitrary in their own world, the anchor can be used to pull its pulse IDs to an external pulse source, e.g. XrayPulses. The idea behind this is to have an easy way to the global pulse IDs for each frame of an XTDF detector. As some cells are sometimes left intentionally empty at the beginning, the first lit cell can be configured.

By default, the cellId is added as an extra column to the pulse multi index generated by this component. This way any other data can be also connected easily to the cells, if desired.

(Note this currently depends on feat/keydata-train-index from EXtra-data, this is not strictly required)

> p = XtdfPulses(run['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf'],
                 ppt_anchor=XrayPulses(run), first_lit_cell=2)
> p.pulse_ids()

trainId     pulseIndex  cellId
2254954300  0           0         1024
            1           1         1032
            2           2         1040
            3           3         1048
            4           4         1056
                                  ... 
2254957299  197         197       2600
            198         198       2608
            199         199       2616
            200         200       2624
            201         201       2632
Length: 606000, dtype: int32

In addition, there's a LitFrameFinderPulses implementation that can access the same data from Egor's LFF device. I think I would prefer to have both functions (solely using XTDF data vs using LFF data) behind the same interface, maybe similar to TimeserverPulses being able to use both timeserver and PP decoder.

[codecov]

Codecov Report

❌ Patch coverage is 20.75472% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.93%. Comparing base (f57e71c) to head (e859338).

Files with missing lines	Patch %	Lines
src/extra/components/pulses.py	20.75%	84 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #436      +/-   ##
==========================================
- Coverage   72.83%   71.93%   -0.91%     
==========================================
  Files          35       35              
  Lines        6048     6153     +105     
==========================================
+ Hits         4405     4426      +21     
- Misses       1643     1727      +84     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[egorsobolev]

If the detector exposes pulses in the dark, then pulse ids refer the slots in the BunchPatternTable with no x-ray pulse. Is this intended?

[philsmt]

Yes.
This is definitely somewhat of a debatable point, but I'd consider it more useful to expose the entire pulse pattern (which is actually a frame pattern) seen by the detector. If you really want to get the frames exposed to X-rays, you can do p.pulse_ids().xs(True, level='lit') or p.pulse_ids().loc[:, :, True].

There are other components doing this already, e.g. PumpProbePattern showing PPL-only pulses in places where there is no FEL pulse or MaschinePulses exposing whatever is in the bunch pattern table.

[egorsobolev]

Shell we also add XGM energy in every frame: data.energyPerFrame and data.energySigma?

[philsmt]

I would prefer one would rather use the XGM component and combine the data. Of course it will have to be made PulsePattern aware for this to work...

[egorsobolev]

Yeah, there is one rare case... There was at least one experiment, when multiple pulses were exposed in single JF frame...

[philsmt]

Hmmm, that mostly tells me that JF is a weird case we should exclude for now. I don't expect multiple pulses per frame for an XTDF detector?

[egorsobolev]

Potentially it is possible... If the detector is able to expose longer then span between pulses, then it is possible. Not for AGIPD v1, it always works at 4MHz

[egorsobolev]

Note: current implementation of LFF for JF just enumerate frames in detector pulse id.

[philsmt]

Figured as much.
The cell ID mechanism also doesn't support JF burst mode yet, I have to double-check how it exposes memory cells. Not sure it's very useful anyway.