Add presentation Extension

[jacobagilbert]

This extension was written to support suggestive input as to how SigMF Recordings and Metadata should be presented visually to users. A basic example of how this could be used is here: miek/inspectrum#195 miek/inspectrum#208 which is just a quick proof of concept for how this might look.

There is no expectation that applications support all features of this, and the "simple" usage where an optional presentation:color field is added to annotations allows for the features in the above link. A much more descriptive set of features, presentation_style objects, are available that can be used to describe recommended presentation of entire classes of annotations or captures segments without requiring individual specification.

I believe this should be in-tree so as to make these field names common across applications and encourage adoption and convention.

The current implementation is blocked by #146 but this can be changed to work without it.

[gmabey]

Did you consider also allowing strings from this well-defined set of color names? https://www.w3.org/TR/SVG11/types.html#ColorKeywords

[jacobagilbert]

I think this would be ver nice overall but I am not sure if this is burdensome on applications implementing this. Interestingly the merge request with inspectrum does support this (thanks to QT's native support for it).

Open to any thoughts here, especially situations this would be a problem.

[gmabey]

Another thought (I'm not strongly advocating this, but I do like it) is to separate the color and the alpha into two separate fields. That would eliminate the RGBA vs ARGB issue, and simultaneously allow for alpha-izing named colors.

Inasmuch as the .sigmf-meta files are supposed to be "human readable", and inasmuch as I still have a hard time visually decoding hex color strings, named colors would be ... more understandable.

[jacobagilbert]

human readability is a strongly beneficial reason to do this. Not sold on separate alpha but i see your use case...will think more. could be an optional that overrides AARRGGBB....

[bhilburn]

I'm in favor of this, and looks good. I'd say get it going, but deprecate your version to something < v1.0.0. Get feedback from @miek and others and target a v1.0.0 release with next SigMF minor release.

[bhilburn]

Deprecate the version

[gmabey]

That would be duplicate information, no? (Having alpha in 2 places) Also, a separate alpha field has a very natural default value of 1.

…

On Mon, Apr 11, 2022 at 12:15 PM Jacob Gilbert ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In extensions/presentation.sigmf-ext.md <#234 (comment)>: > +application using this extension is not required to implement any particular +feature. It is RECOMMENDED that applications adhering to the `presentation` +extension use default values in this document. It is also RECOMMENDED that +applications implementing this extension describe what fields are implemented. + +## 0 Datatypes + +The `presentation` extension defines the following SigMF datatypes: + +|type|long-form name|description| +|----|--------------|-----------| +|`color`|Hexadecimal Color String|String representing a 24 bit color (with optional alpha value) in hexadecimal form; either "#RRGGBB" or "#AARRGGBB".| + +### 0.1 The `color` datatype + +The `color` datatype is a string which MUST be either 6 or 8 hexadecimal human readability is a strongly beneficial reason to do this. Not sold on separate alpha but i see your use case...will think more. could be an optional that overrides AARRGGBB.... — Reply to this email directly, view it on GitHub <#234 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVTOUBC5DRHR3MPGZPAYJTVERT3DANCNFSM5ROCJGIQ> . You are receiving this because you commented.Message ID: ***@***.***>

[miek]

I've thought about this a bunch and, while I really like the idea of being able to encode some visual information, I worry that this implementation will lead to people creating recordings that only work well with their favourite tool/theme/colour-map. For example, I plan to add options for other colour maps to inspectrum at some point and presentation parameters for annotations created now could be unreadable on other colour maps.

I'd much rather see some way to encode intent rather than specific colours/styles, and then visualisation tools can interpret that in a way that works with their theme. Perhaps the naming scheme from standard logging levels could work, with fatal/error/warn/info/etc.? There could also be a way to define extra named groups, to group similar packets together for example.

[777arc]

I'd much rather see some way to encode intent rather than specific colours/styles, and then visualisation tools can interpret that in a way that works with their theme. Perhaps the naming scheme from standard logging levels could work, with fatal/error/warn/info/etc.? There could also be a way to define extra named groups, to group similar packets together for example.

Great point, although I would rather see integers starting at 0 than fatal/error/warn/info, that way we don't have to all agree what each code means, plus the RFML folks will love integers. And it solves the group problem. Would be nice to at least standardize on green and red, e.g. have 0 = green and 1 = red and then past that it's whatever the tooling thinks looks good?

Then there's the question of how do we support both options, because in the end it would seem weird to not allow specifying a specific color...

[jacobagilbert]

@miek Your input here is great, sorry I sat on this for so long. I'll definitely keep this in mind the next time I'm wondering why one of my inspectrum PRs hasn't been merged yet LOL!

I think your proposal makes a ton of sense and I'd like to figure out a way to capture that effectively. I have run into exactly the concern you raise repeatedly in my playing around with colored annotations in just the single default inspectrum colormap. Im wondering if we can identify a small set of features or annotation groups users would want to allow a tool to color. I think it needs to be a small enough set that its reasonable to find a set of annotation styles that coexist with a given colormap and are distinguishable.

I like taking inspiration from log levels, heres my first-pass take on this:

styles = {
  ALERT,    // should be visible immediately, regardless of the view settings of the dataset
  ERROR,    // should quickly draw attention and be styled in such a way to indicate a problem (e.g.: red)
  WARNING,  // upon inspection it should be obvious that this area may require additional inspection
  INFO,     // this is the standard annotation style
  DEBUG,    // this annotation is of lower importance than standard annotations
  USER_A,   //  \
  USER_B,   //  |
  USER_C,   //   > these styles should be visibly distinct but no relative importance is implied
  USER_D,   //  |
  USER_E    //  /
}

A number of questions:

1. Does 10 styles seem like a lot?
2. Should there be more user classes?
3. Should the ability to specify suggestions or explicit colors still be included?

---

Also, as an aside:

For example, I plan to add options for other colour maps to inspectrum

🙌 🙌 I like to hear this, viridis cant come soon enough :) - Tammojan's suggestion in 204 of using tinycolormap seems like a pretty easy way to get Parula, Heat, Jet, Turbo, Hot, Gray, Magma, Inferno, Plasma, Viridis, Cividis, Github, Cubehelix, and HSV for the cost of adding a drop down, removing Jet from that list and adding a little interface glue.

[777arc]

@jacobagilbert is going to remove some of the fields that havent been used, in order to get a minimal version merged in

[devnulling]

@Teque5 @jacobagilbert ping on this fellow

[marcusmueller]

strike: you cannot "MUST" a hex string and then say "we recommend to support something else in the consumer". That makes no sense. And named colors are a terrible idea, throughout, for a metadata format, to be 100% honest here.

[marcusmueller]

Strike. implementation detail.

[marcusmueller]

not sure I understand this sentence at all. JSON doesn't know "dictionaries"; it speaks of "objects": What is emulated here?

[marcusmueller]

shouldn't this be of type color?

[marcusmueller]

Are we sure this is supposed to be "color of the fill…"? It takes values of "VALID" and "INVALID" in the example

[marcusmueller]

move to "Data Types"

[marcusmueller]

move to "Data types"

[marcusmueller]

Is a subtype, needs to be in data types

[remleduff]

I just wanted to recognize that this implicitly requires #146 also be approved.

With that, I think that using capture_tags in captures is much better than styling annotations by label. I think it would be generally more useful to add an annotation_tags member to annotations which allows semantic tagging (and styling as a side effect) of the annotation separately from its label.

[remleduff]

I think this definition of key's semantic isn't very clear. It should mention that for captures it is keying by capture_tags and for annotations by label.

It may be too late to change this, but it might have been better to have separate "tag" field and "label" field requiring one of them to be specified.

(As I mention elsewhere) Keying annotations by label isn't really flexible enough imho, I think it would be better if there was also an annotation_tags field in annotations.

[remleduff]

I'm not honestly sure how inconvenient this "first wins" policy will be in practice.

(It's probably a bigger issue if annotation_tags were to be added, as I can think of several scenarios where an annotation might have several tags. It would be desirable to be able to "fix" the style to make it clear that it's both VALID and IGNORED or something.

In other words, I could imagine wanting the ability to key by "one_of" or "all_of" on the list of tags.)

[dkozel]

There are a few typos/errors that I've pointed out here.

However the much more substantive issue that I'll object to is the use of display styling in place of meaning. I agree there should be a mechanism for indicating that certain elements should be visually styled in a particular (though maybe not absolute) way.

Introducing visual styling to an element is/should be marking an annotation or capture as being part of some semantic group. The proposed capture_tags in #146 introduces the concept of tags as a way of linking meaning to elements and could be extended to annotation_tags.

Currently an annotation's label does this in part, but there are several reasons not to use it for display styling. There is room for adding more flexibility to users to mark multiple properties of an annotation (without adding a full schema extension) and tags are a viable route there. Label is a mature field which has developed specific meaning to many users and altering it's utility now would be invasive. Third, the decision of what property should be used to decide the display styling should be left as a standalone decision for users.

I was initially sold in on the idea of linking display styles to the _tag(s) value, but now am wondering if that becomes overloading functionality. Marcus raised the comparison with HTML element classes and CSS styling. I'm leaning towards having a single presentation:class optional per annotation or capture. In the global name space it should be required to define

"presentation:classes" : [
  {
    "presentation:class": "group1",
    "presentation:description": "Packets sent by users in the Demo group"
  }, ...
]

Then either in a separate .sigmf-style file or in the global section, have presentation:style definitions which refer to the classes. This decouples meaning from the styling details, allowing easy theming.

I can't think of a situation where the styling of a single annotation or capture should be manually set within the scope of the annotation. Styling an element should be used to convey meaning. If meaning is being conveyed it should be defined. If it's defined it should be done in a reusable way. Once it's defined in the global space it can be used for a single element or many. The ability to embed styling information inside of individual objects should be removed.

Thoughts?

I could see reversing the direction of reference such that a style file (or section) defines a set of available styles, and the presentation classes declare what style to use.

Part of my thinking is that the selection/definition of classes is very end-user/application specific, and the styling is also likely to be tool/environment specific.

The Recording would need to declare what Style file it is intended to be used with. Marcus brought up the idea of having the field support a URI, so styles could optionally be remotely hosted, distributed alongside the recordings, or embedded in tools.

I want to like the idea of defining a default set of style names, like the Log Levels and other attributes @miek mentioned, as long as there's a way of defining new styles and distributing them.

[dkozel]

Description should be "List of available display styles for captures"?

[dkozel]

typo: containing

[dkozel]

typo: style

[dkozel]

Dependency on the currently non-merged addition to the core spec:
#146

[dkozel]

Description should be "List of available display styles for annotations"?

[marcusmueller]

I like @dkozel's approach here, though I have to say that a separate .sigmf-style file sounds like something I'd avoid; especially since, well, citing myself, CSS already exists, and reinventing that wheel in JSON certainly isn't a great idea.

I'd lean towards having a global attribute that refers to the URI (not URL) of a separately available style (in whichever format) rather than inclusion in metadata; comparable to, say, how one references an XML namespace. Representation instructions are not metadata!

[Teque5]

We (@dkozel, @777arc, @KelseyCreekSoftware) just had a long discussion on this topic in the monthly SigMF meeting. We discussed CSS, Matplotlib colormaps, non-web interfaces, hex colors, and more.

Conclusions

• We want to KISS.
• We want to leave color science and color choices to other folks.
• We want to support web and non-web applications.
• We want to support theming and flexibility where possible.

Decision

1. The presentation extension should have something that indicates the class or index of the annotation color from [0, N).

Other ideas

2. The presentation extension should have something that indicates the rank, level, or class of the annotation from [0, N).
3. The presentation extension could support an optional array of named/hex colors that color_index are taken from, or perhaps a defined colormap.

I Suggest

|name|required|type|description|
|----|--------|----|-----------|
|`color_index`|true|int|Color index from colormap. Default 0.|
|`style_index`|false|int|Style index where `0` is highest importance, similar to markdown levels where increasing value is of lower and lower visibility. Default 0.|

or we could consolidate both keys into something called index that just describes the color and style and rank all in one attribute.

@777arc is going to work up a new PR on this.

_edit: removed color suggestions.

[777arc]

Yeah I think consolidating into 1 index is best, to keep it very simple, we can make a note that the first index, 0, could correspond to the signal of interest or friendly signals, and that the second index, 1, could correspond to interferers or adversarial signals, that way if software wants to use green and red for those categories it knows exactly what to do.

Side note, I think the different log level meanings don't translate intuitively into signals in a spectrogram, I like "ALERT" to bring attention to a signal but none of the other ones feel good.

[777arc]

Actually, before I open a new PR, does it make sense to create an entirely new extension if all I'm proposing is adding an optional field to annotations called color_index which is an integer starting at 0 and goes as high as you want? It would be up to the software to decide how to map to color, but we would have the suggested mapping using the following color scheme, which is just matplotlib's default aka tab10 but with green and red as 0 and 1 respectively:

0 #2ca02c (green) - can be used for signal of interest, friendly signals, signals of high importance, etc
1 #d62728 (red) - can be used for interferers, adversarial signals, etc
2 #1f77b4 (blue)
3 #ff7f0e (orange)
4 #9467bd (purple)
5 #8c564b (brown)
6 #e377c2 (pink)
7 #7f7f7f (grey)
8 #bcbd22 (mustard)
9 #17becf (cyan)
<suggested colors only go up to 9 but the color_index can go as high as you want>

There could be example python/javascript/etc code somewhere that provides the mapping so it's easy for folks

So the point here is to not have a standard SigMF way to define styling, but instead leave it up to the software, and have a very light method that folks can choose to use to categorize annotations by color, either N classes, or SOI/interferer style. Someone might even just want a way to mark extra important signals so they only ever use 0 and no other levels.

[marcusmueller]

Because I've been pretty vocal in chat, so that I can't come back later and complain :D :

I agree with #234 (comment) and #234 (comment) , i.e. annotations are classes/indices (not sure why classes need to be numeric, not all classes are orderable and you can't always know which classes you want before, so strings do sound nicer in some contexts, but I'm not going to complain. It feels much more useful when looking things up to have an integer.).

Nice work, thank you for hashing this out.

[Teque5]

Actually, before I open a new PR, does it make sense to create an entirely new extension if all I'm proposing is adding an optional field to annotations called color_index which is an integer starting at 0 and goes as high as you want? It would be up to the software to decide how to map to color, but we would have the suggested mapping using the following color scheme, which is just matplotlib's default aka tab10 but with green and red as 0 and 1 respectively.

I think any choice you make to specify a specific color is not a good idea. Barring any discussion of cultural color symbolism and accessibility concerns, I think colors are left to the GUI and the designers.

If folks are interested in color theory, this talk related to the introduction of the matplotlib viridis colormap is mandatory watching on this topic.

I went back and edited by suggestion to remove tab10.

I also don't mind if we make this an optional key in annotations, it's more likely to be used that way.

[777arc]

Yeah we can leave out the suggested color mapping, I agree software will use its own scheme based on the spectrogram colormap.