Structured cbor #3036

JesusMcCloud · 2025-07-05T05:03:02Z

This PR introduces structured CBOR encoding and decoding

Encoding from/to `CborElement`

Bytes can be decoded into an instance of CborElement with the [Cbor.decodeFromByteArray] function by either manually
specifying [CborElement.serializer()] or specifying [CborElement] as generic type parameter.
It is also possible to encode arbitrary serializable structures to a CborElement through [Cbor.encodeToCborElement].

Since these operations use the same code paths as regular serialization (but with specialized serializers), the config flags
behave as expected

Newly introduced CBOR-specific structures

[CborPrimitive] represents primitive CBOR elements, such as string, integer, float boolean, and null.
CBOR byte strings are also treated as primitives
Each primitive has a [value][CborPrimitive.value]. Depending on the concrete type of the primitive, it maps
to corresponding Kotlin Types such as String, Int, Double, etc.
Note that Cbor discriminates between positive ("unsigned") and negative ("signed") integers!
CborPrimitive is itself an umbrella type (a sealed class) for the following concrete primitives:
- [CborNull] mapping to a Kotlin null
- [CborBoolean] mapping to a Kotlin Boolean
- [CborInt] which is an umbrella type (a sealed class) itself for the following concrete types
  (it is still possible to instantiate it as the invoke operator on its companion is overridden accordingly):
  - [CborPositiveInt] represents all Long numbers ≥0
  - [CborNegativeInt] represents all Long numbers <0
- [CborString] maps to a Kotlin String
- [CborFloat] maps to Kotlin Double
- [CborByteString] maps to a Kotlin ByteArray and is used to encode them as CBOR byte string (in contrast to a list
  of individual bytes)
[CborList] represents a CBOR array. It is a Kotlin [List] of CborElement items.
[CborMap] represents a CBOR map/object. It is a Kotlin [Map] from CborElement keys to CborElement values.
This is typically the result of serializing an arbitrary

Example

bf                                 # map(*)
   61                              #   text(1)
      61                           #     "a"
   cc                              #   tag(12)
      1a 0fffffff                  #     unsigned(268,435,455)
   d8 22                           #   base64 encoded text, tag(34)
      61                           #     text(1)
         62                        #       "b"
                                   #     invalid length at 0 for base64
   20                              #   negative(-1)
   d8 38                           #   tag(56)
      61                           #     text(1)
         63                        #       "c"
   d8 4e                           #   typed array of i32, little endian, twos-complement, tag(78)
      42                           #     bytes(2)
         cafe                      #       "\xca\xfe"
                                   #     invalid data length for typed array
   61                              #   text(1)
      64                           #     "d"
   d8 5a                           #   tag(90)
      cc                           #     tag(12)
         6b                        #       text(11)
            48656c6c6f20576f726c64 #         "Hello World"
   ff                              #   break

Decoding it results in the following CborElement (shown in manually formatted diagnostic notation):

CborMap(tags=[], content={  
    CborString(tags=[],   value=a) = CborPositiveInt( tags=[12],     value=268435455),  
    CborString(tags=[34], value=b) = CborNegativeInt( tags=[],       value=-1),  
    CborString(tags=[56], value=c) = CborByteString(  tags=[78],     value=h'cafe),  
    CborString(tags=[],   value=d) = CborString(      tags=[90, 12], value=Hello World)  
})

Implementation Details

I tried to stick to the existing CBOR codepaths as closely as possible, and the approach to add tags directly to CborElements is the most pragmatic way of getting expressiveness and convenient use. It does come with a caveat (also taken from the Readme:

Tags are properties of CborElements, and it is possible to mixing arbitrary serializable values with CborElements that
contain tags inside a serializable structure. It is also possible to annotate any [CborElement] property
of a generic serializable class with @ValueTags.
This can lead to asymmetric behavior when serializing and deserializing such structures!

The test cases (and comments in the test cases reflect this

Closing Remarks

I also fixed a faulty hex input test vector that I introduced myself, last year, if I pieced it together correctly (see here) and I amended the benchmarks. (see here).

Since the commits from here will be squashed anyways, I did not care for a clean history.

JesusMcCloud · 2025-07-08T10:28:59Z

Full disclosure: This PR incorporates code from a draft generated by Junie (albeit an impressive draft that saved a day of work). This is not a dumb copypasta of AI-generated code. Even if it were already feature-complete It would still not yet be marked ready for review because we have yet to review everything internally. I also want to stress that "we" is not a euphemism. There will be at least two of us reviewing and discussing internally, almost certainly with additional input from other humans in the process of readying this PR.

pdvrieze

I've reviewed the code at a general level. There is a lot of repetition, so I've only commented on the first case, not every one. I guess some of the AI generation is visible in the details.

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

JesusMcCloud · 2025-08-07T11:29:28Z

Performance seems to be OK (fromBytes and toBytes are the baseline on my machine):

Metric / Benchmark	`fromBytes`	`fromStruct`	`structFromBytes`	`toBytes`	`structToBytes`	`toStruct`
Average (ops/ms)	1205.615 ± 20.541	1545.814 ± 50.743	2896.728 ± 74.485	2089.013 ± 30.152	1442.766 ± 32.257	2581.397 ± 32.497
Min	1186.023	1458.225	2796.131	2066.499	1404.482	2550.026
Max	1229.778	1581.420	2960.572	2125.658	1475.015	2619.815
Stdev	13.586	33.563	49.267	19.944	21.336	21.495
CI low (99.9 %)	1185.075	1495.071	2822.244	2058.861	1410.509	2548.900
CI high (99.9 %)	1226.156	1596.557	2971.213	2119.165	1475.023	2613.893

My hot takes:

Deserialising from a structure is fast enough since it is in the same ballpark as deserialising from bytes
Deserialising into a generic CBOR structure takes twice the time than directly deserialising, which is fine, given that we instantiate much more as even primitives need a containing class and an array of tags
Serialising a generic CBOR structure to bytes is faster but in the same ballpark as generic to-byte serialisation of arbitrary serializable data
Serializing to a CBOR structure is slower than to bytes, but OK enough, since it's in the same ballpark and we instantiate more

JesusMcCloud · 2025-08-07T12:12:51Z

I just noticed something that looks weird to me. See this test case here that is failing and closely compare expected vs actual.

the byte string is wrapped twice for the reference. I know there were some discussions, but I don't recall them, so I have to ask: why? did I mess this up last year or is this intentional? Because the way I see it, were' wrapping a bytearray instead of encoding it differently
EDIT: the test vector is faulty as this comparison fails the same way

whyoleg

First round of review from my side :)
Mostly reviewed API surface, will take a look on the implementation details later

whyoleg · 2025-08-08T11:34:55Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+    }
+}
+@Serializable(with = CborIntSerializer::class)
+public sealed class CborInt<T : Any>(


According to RFC 8949, integers are:

type 0 - 0..2^64-1

type 1 - -2^64..-1

This means that only ULong can really fit both positive and negative values.

So I believe that it will be more correct to have a single CborInt implementation, with an API similar to:

public class CborInt(sign: Int, absoluteValue: ULong, tags) { constructor(value: Long) fun toLong(): Long fun toLongOrNull(): Long? }

where:

sign is: -1, 0, 1

toLong - will throw if can't fit - probably a very rare case

Good catch! I agree. The constructors with the current signature (simply passing a number) should stay for convenience.

~~done in 89d671c~~
done in fba0348

whyoleg · 2025-08-08T11:37:07Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+ * traditional methods like [List.get] or [List.size] to obtain CBOR elements.
+ */
+@Serializable(with = CborListSerializer::class)
+public class CborList(


It might make sense to call it CborArray, as spec says, it's array, and Json format also calls it JsonArray because it's called array in spec :)

Sadly, there is an annotation by the very same name ins the same package. Now moving the cbor elements is easily done, but it will probably not be very intuitive (thinking of autocompletion here).

In any case: Good point but not my call!

Oh, yes, how did I miss that :(

But this is on me anyhow because git blame says "Prünster" for the whole file containing the @CborArray annotation. So my short-sighted naming is to blame.

remains an open discussion point

It would be nice to rename @CborArray into something else, and reuse the name here. And it would be also nice to rename @ByteString annotation too.

While CBOR format is experimental, we are free to break things, but it would be nice to make a transition a bit smoother, and, probably, deprecate @CborAray right now and provide an alternative annotation, and as a next step provide the CborElement API with CborArray reused for the aggregate type.

@sandwwraith WDYT?

Some migration path would be nice, as I did mess up with naming. Better sooner than later, given the impact of the kotlinx-datetime / kotlin.time migration

I'm in favor of renaming the annotation, but I'm currently unable to think of a suitable name right now. Maybe @CborAsArray or @CborObjectAsArray, but these look clumsy

@EncodeAsCborArray?

whyoleg · 2025-08-08T11:39:31Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+     * See [RFC 8949 3.4. Tagging of Items](https://datatracker.ietf.org/doc/html/rfc8949#name-tagging-of-items).
+     */
+    @OptIn(ExperimentalUnsignedTypes::class)
+    public var tags: ULongArray = tags


I believe we should make the implementation in a way, that it will be not mutable

We can make it so that there is a val with a custom getter that delegates to the actual field, but the tags should still be simply written to a CborElement as they are collected upon deserialisation, to avoid serialization overhead. Actually, the elements should probably be mutable too internally so the whole container structure that now collects tags and elements for Cbor structures can go away.

In the end this would mean that we have:

an internal var making it possible to collect tags during deserialisation, but a public val tags with a custom getter and no backing field

an internal val elements for array and map so we can also collect elements as we go and get rid of another instantiation, but a public val elements: List<CborElement>

This should boost performance for deserialisation significantly

Of course, this is related to #3036 (comment), but it would also be nice to compare it to how JSON works with lists. Maybe there is some pattern there that could be used inside the CBOR implementation.

Not touching this yet, as it does relate to #3036 (comment). I still stand by my proposal from the comment above, though.

There are two issues with unsigned arrays being a part of public API:

arrays are mutable (which is not very nice in general, but given that tags are used to calculate equals & hashCode, it makes things even worse);

unsigned arrays are experimental and experimental API should be a part of the public API.

And it's definitely should be val, not var.

As of tags mutability (I mean, tags being an array itself), we can introduce a dedicated CborTags collection-type or something, but it's up to discussion.

They will be boxed, though. Theoretically, we can make a defensive copy in a custom getter instead, but I'm not sure it will be faster (less memory pressure tho).
Also, we have already gone down the road with experimentality with @KeyTags/ValueTags.

Also, we have already gone down the road with experimentality with @KeyTags/ValueTags.

True. Yet, the mutability concern holds still (and there's a similar one for CborByteString, BTW), so we need some other way to represent tags than by using an array.

I'll have to dig into the internals and see how this is currently done. If there seems to be too much array copying going on, a list is worth reconsidering. Otherwise, make the public-facing property a val with a custom getter on an internally mutable property, I guess?!

less memory pressure tho

Speaking of memory pressure, the current implementation allocates a fresh ULongArray instance for every CborElement. Assuming that majority of element won't have tags associated with them, it feels pretty excessive.

A bit dirty, but what if we treat null and empty arrays both as "no tags"?

whyoleg · 2025-08-08T11:44:19Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborDecoder.kt

+     * This method is allowed to invoke only as the part of the whole deserialization process of the class,
+     * calling this method after invoking [beginStructure] or any `decode*` method will lead to unspecified behaviour.
+     */
+    public fun decodeCborElement(): CborElement


I believe that there should also be CborEncoder.encoderCborElement as in Json

That just delegates to encodeToByteArray<CborElement>, right?!

Most probably to encodeSerializableValue(CborElementSerializer, element), but yes, just for symmetry

done in 004dd1d

whyoleg · 2025-08-08T11:55:24Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+    )
+
+/*need to expose writer to access encodeTag()*/
+internal fun Encoder.asCborEncoder() = this as? CborWriter


I do think that we should probably make it possible to cast only to CborEncoder here so that we use the same public API as the user.
As far as I see, the only thing needed is encodeTags—so is this a sign that encodeTags might be exposed on the CborEncoder interface?
I'm not sure, though, how useful it could be in reality, but it seems like it should be possible to encode tags manually.

You are probably on to something here. I should investigate tag handling some more

done in 1d97928

whyoleg · 2025-08-08T18:39:16Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+@Serializable(with = CborPositiveIntSerializer::class)
+public class CborPositiveInt(
+    value: ULong,
+    vararg tags: ULong


This vararg feels very unfortunate. For example, currently, it's possible to call CborPositiveInt(1u, 2u, 3u), and it's really hard to tell what's going on here.
UlongArray will probably be better

Overall, it's applied to all other declarations; e.g., CborString("1", 1, 2, 3) doesn't feel better.

I agree with the first part, but the second part is simply in line with how the tagging annotations behave and it is convenient for the user (much more so that ulongArrrayOf().
Yet, I am aware that it should be consistent, so making ints work differently from everything else is also not really nice… In the end, don't really have an opinion. Sort it out internally, it's a straight-forward refactor anyways.

What would help in such cases is a language feature that forces parameter name specification at the call site, but that is not going to happen anytime soon, if ever.

Just in case, for me additional tags field looks very similar to how annotations are applied currently.
Additionally, with Collection Literals it will look like CborString("hello", [1u, 2u, 3u]) or with Named-only parameters it could even be forced to call it like CborString("hello", tags = [1u, 2u, 3u])

Those two features would help, but in the meantime, we need something else for a solution

this remains an open discussion point and is ultimately not my call

whyoleg · 2025-08-08T18:50:23Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+ * The whole hierarchy is [serializable][Serializable] only by [Cbor] format.
+ */
+@Serializable(with = CborElementSerializer::class)
+public sealed class CborElement(


What do you think about the alternative design:

sealed class CborElement { abstract val tags: ULongArray abstract fun withTags(tags: ULongArray): CborElement } // and all other class CborString(value: String): CborElement() { override fun withTags(tags: ULongArray): CborString }

Or, even having CborElement have no tags at all, but have a wrapper element CborTagged?

sealed class CborElement // and all other class CborString(value: String): CborElement() class CborTagged(element: CborElement, tags: ULongArray): CborElement

Both variants may apply some performance penalty, though.
It's just that, reading the spec and current implementation of CborElementSerializers, the current placement of tags seems unfortunate and does not result in straightforward logic.

The first one solves the issues nicely and it would work well with the internally mutable tags array s.t. it would avoid instantiation overhead just for modifying tags. BUT, that would be a footgun for the consumer.

I have thought about CborTagged, also because of the asymmetric serialization/deserialization behaviour, which really is not all that nice, but also very niche (why would anyone in their right mind do this??)

When I started with the implementation, I was imagining how the most straight-forward and usable way would look like: Tags are attached to something and have a hierarchy. An array is enough to represent this and makes for a nice API that mirrors the pattern of @ValueTags, @ObjectTags, and @KeyTags. Now you could get rid of that pesky asymmetry, if you disallow those annotations on CborElement. Then I'd assume we'd be back to straightforward logic again?!

Bottom line: there are issues that need sorting out. The inconsistency issue is documented in the readme, so I was not trying to cover anything up, but I wanted unbiased opinions and fresh ideas because I could not get to a satisfactory solution by myself. It worked once already. I just hope Leonid checks the code first, and the comments later ;-)

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

JesusMcCloud · 2025-10-07T11:47:43Z

Any updates on the open discussion points?

fzhinkin

@JesusMcCloud, sorry for the delay with review. 😿

I didn't thoroughly review the serialization machinery, but focused on the API part for now.

docs/formats.md

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

fzhinkin · 2025-10-13T15:00:00Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+
+/*need to expose writer to access encodeTag()*/
+internal fun Encoder.asCborEncoder() = this as? CborEncoder
+    ?: throw IllegalStateException(


Don't forget to cover this behavior with tests.

fzhinkin · 2025-12-03T20:17:23Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+ */
+@Serializable(with = CborIntSerializer::class)
+@ExperimentalSerializationApi
+public class CborInt(


What do you think about CborInteger instead? It's closer to major type descriptions from the spec (an unsigned integer, a negative integer) and it is farther from Kotlin's Int.

I'm fine with both. Completely up to you guys

Let's make it CborInteger then.

fzhinkin · 2025-12-03T20:27:03Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+    }
+
+    /**
+     * **WARNING! Possible truncation/overflow!** E.g., `-2^64` -> `1`


Silent truncation/overflow is something we should avoid. I'd vote for an exception if a value could not be represented as Long (Int, Short, or Byte - more on that later), and an additional toLongOrNull function, that'll return null on overflow.

It would be nice to align the API with what we already have for JsonElement and instead of providing a member function, provide extension properties.
And it seems reasonable to provide properties returning other integer types too.

toLongOrNull sound good (also for other integer types)!

What would be core functionality (implemented as member functions) and what should be extensions? Or is it really just "look at JsonElement and replicate that" without leaving anything ambiguous?

fzhinkin · 2025-12-03T20:35:07Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+@Serializable(with = CborNullSerializer::class)
+@ExperimentalSerializationApi
+public class CborNull(vararg tags: ULong) : CborPrimitive<Unit>(Unit, tags)
+


Topic for later discussion: do we need a CborUndefined? It's one of the values that could not be parsed with the existing API, yet it is something a valid CBOR document might contain.

Great catch! We need a CborUndefined. I totally forgot about that!

fzhinkin · 2025-12-03T20:42:34Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+ * traditional methods like [List.get] or [List.size] to obtain CBOR elements.
+ */
+@Serializable(with = CborListSerializer::class)
+public class CborList(


It would be nice to rename @CborArray into something else, and reuse the name here. And it would be also nice to rename @ByteString annotation too.

While CBOR format is experimental, we are free to break things, but it would be nice to make a transition a bit smoother, and, probably, deprecate @CborAray right now and provide an alternative annotation, and as a next step provide the CborElement API with CborArray reused for the aggregate type.

@sandwwraith WDYT?

fzhinkin · 2025-12-03T20:50:37Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+            ")"
+    }
+
+}


There are few things that are currently missing, but that would otherwise improve DX when it comes to using CborElement API:

casting functions/properties such as

val CborElement.cborFloat: CborFloat, val CborElement.cborList: CborList

fun CborElement.asCborFloat(): CborFloat, ...

builders for aggregates, similar to JsonObjectBuilder and JsonArrayBuilder

Those could be added later, though.

I agree that these should be there and all of those are easy wins, given a foundation is there and I can add all of these. I'd just like a complete list (even if it is "only" a link to some JSON API Docs or whatever). BUT: I'd really like this PR to make it into the closest possible release, as we have a very concrete need for this.

That being said, I'd propose the following strategy going forward:

I'll address all the open issues that are sorted out already (i. e. stuff that just needs to be done).

All the open discussion points that affect the API should be sorted out ASAP

We will integrate a snapshot build into our codebase, so we have some real-world feedback to check if the API is missing anything or if some behaviour is too limiting

Then we'll finalise this PR

Depending on how large everything is and how pressing the need for builders is, they should be part of this PR or part of a separate one. I just don't want something small being the reason for having to wait for another release cycle…

Depending on how large everything is and how pressing the need for builders is, they should be part of this PR or part of a separate one

We can definitely add them as a follow up:

Those could be added later, though.

Co-authored-by: Filipp Zhinkin <filipp.zhinkin@gmail.com>

fzhinkin · 2025-12-04T21:15:14Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+
+    override fun deserialize(decoder: Decoder): CborFloat {
+        decoder.asCborDecoder()
+        return CborFloat(decoder.decodeDouble())


Tags will be lost here during deserialization (the same is true for CborMap and CborList deserialization routines).

fzhinkin · 2025-12-04T21:16:27Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+    override fun isElementOptional(index: Int): Boolean = original.isElementOptional(index)
+}
+
+private fun CborWriter.encodeTags(value: CborElement) { // Encode tags if present


It is unused

fzhinkin · 2025-12-04T21:18:41Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+        val cborEncoder = encoder.asCborEncoder()
+        cborEncoder.encodeTags(value.tags)
+        //this we really don't want to expose so we cast here
+        (cborEncoder as CborWriter).encodeByteString(value.value)


But the CborEncoder is public and it could be anything else.

fzhinkin · 2025-12-04T21:34:41Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/CborElementSerializers.kt

+        }
+
+        decoder.decodeNull()
+        return CborNull()


Tags will be lost here too

fzhinkin · 2025-12-04T22:08:49Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoder.kt

+        fun finalize() = when (this) {
+            is List -> CborList(content = elements, tags = tags)
+            is Map -> CborMap(
+                content = if (elements.isNotEmpty()) IntRange(0, elements.size / 2 - 1).associate {


It worth adding a check that elements.size is even.

fzhinkin · 2025-12-04T22:15:27Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoder.kt

+            is Map -> CborMap(
+                content = if (elements.isNotEmpty()) IntRange(0, elements.size / 2 - 1).associate {
+                    elements[it * 2] to elements[it * 2 + 1]
+                } else mapOf(),


I would also vote for

buildMap { for (i in 0 until elements.size / 2 - 1) { put(elements[i * 2], elements[i * 2 + 1]) } }

as an alternative to if (..) associate {} or mapOf()

fzhinkin · 2025-12-04T22:17:39Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoder.kt

+        protected val elements = elements
+
+        var tags = tags
+            internal set


CborContainer is internal anyway

fzhinkin · 2025-12-04T22:21:30Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoder.kt

+        class List(tags: ULongArray, elements: MutableList<CborElement> = mutableListOf()) :
+            CborContainer(tags, elements)
+
+        class Primitive(tags: ULongArray) : CborContainer(tags, elements = mutableListOf()) {


Primitive is unused and could be removed, isn't it?

fzhinkin · 2025-12-04T22:28:29Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

+    public fun <T> encodeToCborElement(serializer: SerializationStrategy<T>, value: T): CborElement {
+        val writer = StructuredCborWriter(this)
+        writer.encodeSerializableValue(serializer, value)
+        return writer.finalize()


It does not support primitive values (try Cbor.encodeToCborElement(42)).
It also fails for ByteArrays when alwaysUseByteString = true.

fzhinkin · 2025-12-04T22:41:56Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoder.kt

+
+    // value tags are collects inside beginStructure, so we need to cache them here and write them in beginStructure or encodeXXX
+    // and then null them out, so there are no leftovers
+    private var nextValueTags: ULongArray = ulongArrayOf()


It's better to be a function with a name that'll give away the behavior.

Also, I'm not sure that reallocating array every time we need to append a tag is the best way to collect tags. ProtobufTaggedBase shows a more efficient approach, although for CBOR tagsStack-analogue should be nullable and has null value by default (I assume that elements are usually untagged).

fzhinkin · 2025-12-04T22:44:06Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoder.kt

+        isClass = descriptor.getElementDescriptor(index).kind == StructureKind.CLASS
+
+        encodeByteArrayAsByteString = descriptor.isByteString(index)
+        //TODO check if cborelement and be done


Sorry, I didn't get what's left to do here

fzhinkin · 2025-12-04T22:52:40Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

+ * @throws [IllegalArgumentException] if the decoded input cannot be represented as a valid instance of type [T]
+ */
+@ExperimentalSerializationApi
+public inline fun <reified T> Cbor.decodeFromCborElement(element: CborElement): T =


The following round-trip transformation does not work with elements, but works find when encoding to / decoding from a byte array:

@Serializable data class Wrapped(val x: Int) @Serializable data class Wrapper(val datum: Wrapped?) val wrapper = Wrapper(null) Cbor.decodeFromCborElement<Wrapper>(Cbor.encodeToCborElement(wrapper))

fzhinkin · 2025-12-04T22:54:14Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

+ * Deserializes the given [element] element into a value of type [T] using a deserializer retrieved
+ * from reified type parameter.
+ *
+ * @throws [SerializationException] if the given JSON element is not a valid CBOR input for the type [T]


I think there are no such JsonElements that are valid CBOR inputs ;)

fzhinkin · 2025-12-04T23:31:08Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Decoder.kt

+    override fun isNull() =
+        if (layer.isStructure) layer.peek().let {
+            it is CborNull ||
+                /*THIS IS NOT CBOR-COMPLIANT but KxS-proprietary handling of nullable classes*/


Note that a null-as-an-empty-map and an empty map (or an object w/o properties) have different encoding and are distinguishable from each other.

However, once we have an empty CborMap, it's impossible to say if it represents a null, or an empty map.

As a result, it leads to some ambiguity:

@Serializable class Wrapped @Serializable data class Wrapper(val x: Wrapped?) @Serializable data class StrictWrapper(val x: Wrapped) val b = Wrapper(Wrapped()) val element = Cbor.decodeFromByteArray<CborElement>(Cbor.encodeToByteArray(b)) // Wrapper(x=null), but x is not null here println(Cbor.decodeFromCborElement<Wrapper>(element)) // StrictWrapper(x=kotlinx.serialization.cbor.CborElementTest$cborEncodeDecodeNull$Wrapped@57ae95ca) println(Cbor.decodeFromCborElement<StrictWrapper>(element))

The underlying issue here is the non-compliant default behaviour that requires this special handling of nullable classes, because with correct defaults, we would not need to take this branch for maps, wouldn't we?!

Could you please clarify what do you mean by "non-compliant default behavior"? Null objects being represented by empty maps?

I'm referring to #2848. Probably mo own doing, but I'm still investigating

whyoleg · 2025-12-05T07:26:45Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborEncoder.kt

+     * Appends the given CBOR [element] to the current output.
+     * This method is allowed to invoke only as the part of the whole serialization process of the class,
+     * calling this method after invoking [beginStructure] or any `encode*` method will lead to unspecified behaviour
+     * and may produce an invalid CBOR result.


TBH, this is a very strange limitation (which I also see present in CborDecoder).
Could you please describe it in more detail, and why we can't do this?

whyoleg · 2025-12-05T07:28:54Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborEncoder.kt

+    /**
+     * Encode a negative value as [CborInt]. This function exists to encode negative values exceeding [Long.MIN_VALUE]
+     */
+    public fun encodeNegative(value: ULong)
+
+    /**
+     * Encode a positive value as [CborInt]. This function exists to encode negative values exceeding [Long.MAX_VALUE]
+     */
+    public fun encodePositive(value: ULong)


It feels a bit strange that we expose those.
If users need those, they should be able to just do:

encodeCborElement(CborInteger(...))

whyoleg · 2025-12-05T07:36:52Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt

+@ExperimentalSerializationApi
+public class CborInt(
+    absoluteValue: ULong,
+    public val sign: Sign,


Yes! I'm fine with a flag like isPositive/isNegative.
Should we also then add an assertion that if the value is 0, we should have isPositive=true?

whyoleg · 2025-12-05T07:58:57Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborEncoder.kt

+    public fun encodeCborElement(element: CborElement): Unit = encodeSerializableValue(CborElementSerializer, element)
+
+    /**
+     * Allows manually encoding CBOR tags. Use with caution, as it is possible to produce invalid CBOR if invoked carelessly!


I remember that it was my suggestion to add this function, but I was thinking that it would be just an alternative to *Tags annotations.
As per the specification, tags are encoded as is, followed by the data items. So something like:

encodeTags(...) encodeTags(...) // this should also work fine encodeInt(...) // or anything else

So the only erroneous call to encodeTags is the one which will not be followed by another encode* call, am I right?

In this case, maybe we need to change the signature to something like encodeWithTags(...) { this: Encoder -> } so that we can easily check this? Or, we can do it similarly to inline classes encoding?
Or perhaps it means we need to rethink this API (and maybe all other tag-related API)?

In any case, I think it would be beneficial to explore how real-world use cases (with tags) might be implemented with this API without using the @Serializable annotation.

Here is one of the RFCs, which is not COSE.
Multi-tag example:

d8 c8 # tag(200) envelope d8 c9 # tag(201) leaf 65 # text(5) 416c696365 # "Alice"

Array example:

d8 c8 # tag(200) envelope 82 # array(2) d8 c9 # tag(201) leaf 65 # text(5) 416c696365 # "Alice" a1 # map(1) d8 c9 # tag(201) leaf 65 # text(5) 6b6e6f7773 # "knows" d8 c9 # tag(201) leaf 63 # text(3) 426f62 # "Bob"

JesusMcCloud · 2025-12-05T09:16:49Z

Thanks for alle the comments! I'll have to dig up some memories that have since collected dust to sort some of the issue out and figure some stuff out again from scratch, as I haven't looked into this for many weeks and forgotten about most of the implementation details ;-). So it will take a bit before I'll push changes, addressing issues.

pdvrieze reviewed Jul 8, 2025

View reviewed changes

JesusMcCloud mentioned this pull request Jul 23, 2025

Use Map delegate for COSE/JOSE data structures a-sit-plus/signum#335

Open

JesusMcCloud changed the base branch from master to dev August 5, 2025 11:26

JesusMcCloud force-pushed the feature/structuredCbor branch 2 times, most recently from 674e7ef to aea3f4b Compare August 7, 2025 10:12

JesusMcCloud added 17 commits August 7, 2025 12:13

WIP strucutred cbor

1d3574c

proper when for decoding

d561069

baseline tagging

5857ef0

some cleanups

718422f

cleanups

6984f69

tree encoding half-done

998f01e

tree encoding close

1a46b06

polish encoder

46c8258

visibility fixes

f6bc421

more checks

890b284

WIP decode from CborElement

255f250

more AI slop

4e15cc2

cleanup after Junie

834fa67

clean up more

d13ec50

fix structural issues

9986e7b

fix map size regression

2a61e81

streamlining and cleanups

342500b

JesusMcCloud force-pushed the feature/structuredCbor branch 2 times, most recently from cd963c7 to c89fd46 Compare August 7, 2025 11:27

JesusMcCloud force-pushed the feature/structuredCbor branch from 7cdbe4d to 9b3f0e5 Compare August 8, 2025 13:22

JesusMcCloud added 2 commits August 8, 2025 16:35

benchmarks

f681698

add more tests

b154640

whyoleg reviewed Aug 8, 2025

View reviewed changes

whyoleg reviewed Aug 9, 2025

View reviewed changes

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborElement.kt Show resolved Hide resolved

JesusMcCloud added 3 commits August 27, 2025 17:15

refactor cborint

fba0348

add CborEncoder.encoderCborElement

aeb4c4c

pimp CborEncoder

1d97928

JesusMcCloud force-pushed the feature/structuredCbor branch from 004dd1d to 1d97928 Compare August 27, 2025 15:15

JesusMcCloud requested review from sandwwraith and whyoleg August 27, 2025 15:20

fzhinkin self-assigned this Oct 7, 2025

fzhinkin self-requested a review October 7, 2025 17:18

fzhinkin added the cbor label Oct 14, 2025

fzhinkin requested changes Dec 3, 2025

View reviewed changes

Apply suggestions from code review

1aec1ff

Co-authored-by: Filipp Zhinkin <filipp.zhinkin@gmail.com>

fzhinkin reviewed Dec 4, 2025

View reviewed changes

whyoleg reviewed Dec 5, 2025

View reviewed changes

+                          ")"
+                  }
+              }

                
                    No newline at end of file

Structured cbor #3036

Are you sure you want to change the base?

Structured cbor #3036

Conversation

JesusMcCloud commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Encoding from/to CborElement

Newly introduced CBOR-specific structures

Example

Implementation Details

Closing Remarks

Uh oh!

JesusMcCloud commented Jul 8, 2025

Uh oh!

pdvrieze left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JesusMcCloud commented Aug 7, 2025

Uh oh!

JesusMcCloud commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whyoleg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JesusMcCloud Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JesusMcCloud Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

JesusMcCloud commented Jul 5, 2025 •

edited

Loading

Encoding from/to `CborElement`

JesusMcCloud commented Aug 7, 2025 •

edited

Loading

JesusMcCloud Aug 27, 2025 •

edited

Loading

JesusMcCloud Aug 9, 2025 •

edited

Loading