Fix: #110. Interoperability considerations instead of new parameter

[ioggstream]

This PR

• replaces Fix: #110. Schema Object parameter #118
• adds interoperability considerations about accepting referenced resources.

[handrews]

I think that OAS 3.0 is the first specification with a standardization base,

What does this mean, exactly? OAS 2.0 (which was a tiny bit different from Swagger 2.0, but mostly just small fixes to errors) was the first version released under the OpenAPI Initiative. So it's as standardized as 3.0 or 3.1.

and referencing 2.0 does not help interoperability.

Didn't we define a version media type parameter for content negotiation on this? While I hate that this is true, there are still tools out there that can't use anything after 2.0. I'd love to do anything and everything to make 2.0 go away, but I'm not sure that forbidding it here will really help that goal. And it seems to prevent people who might want to try to bridge 2.0 to 3.x from making an effort in a standardized way that is consistent with what they will do to bridge 3.0 to 3.1+, or to 4.0

Moreover, it's in an FAQ that will be removed at publication.

Does this mean that in practice, 2.0 is not excluded? I'm a bit confused.

[ioggstream]

forbidding it here will really help that goal. And it seems to prevent people who might want to try to bridge 2.0 to 3.x

This makes sense, but I think that migration should be the only reason for that.

Actually, using version: 2.0 is not "forbidden". Just "undefined" here, because:

• I did no proof-reading of the spec wrt OAS 2.0 and there are no statements such as:

dependently on the OAS version, the keyword associated with the version is different between 2.0 (swagger) and 3+ (openapi) ...

• I don't know whether the terms of "OAD", "OAd", ... are still valid for v2.0...
• 2.0 tools generally use application/vnd.swagger.whatever

Anyway, I'm not that into v2.0, but if you/ @darrelmiller think that this document suits OAS 2.0, we can extend the scope.

[ioggstream]

@handrews if we agree on this, let's split and merge this part of the PR.

[handrews]

I'm trying to figure out the cases in my head, as something is bugging me here but I can't quite figure it out. There are few possible reference targets:

1. An OAD document (with an OpenAPI Object at the root) (application/openapi+json or application/openapi+yaml)
2. A JSON Schema document (a Schema Object in its own file) (application/schema+json or whatever we use for schemas, but NOT application/openapi+json)
3. A different type of Object (e.g. Path Item Object) in its own file (This case has no media type more specific than application/json or application/yaml)
4. Some other sort of document used for external docs (External Documentation Object url) or examples (Example Object externalValue) (media type determined as usual- possibly text/html for external docs, while the example could be literally anything, depending on what it is an example of).

Cases 1, 2, and 4 are well-defined.

In the case of 1 and 2, the usual caveats apply, namely that a more generic media type might be returned rather than the specific one with the structured suffix. I don't think there's anything special going on there compared to any other situation where you would ask for a +json and might get back application/json even though the contents are correct for the requested +json format.

For case 3, this is the thing that we may or may not want to mention as possible at all. In #110, it appears we decided to not mention it as a way to discourage people attempting such document organization (it's never necessary in 3.1+, but in 3.0 you can't put Path Item Objects in the Components Object, so people often put then in separate files).

[karenetheridge]

Yes, I don't think we should allow 3 at all. It has no media type and it isn't obvious even from visual inspection what type of object(s) it contains, so I don't think it could be reliably parsed.

Remember, it's not enough to just simply load the file and decode its yaml or json and consider it a $refable target -- an implementation needs to know what type of object it is so it can examine it for embedded schemas (which can contain $id and $anchor keywords which can be used by $ref keywords in other schemas in the OAD); and in the case of operation objects, examine it for operationId keywords (which are referenceable by link objects, as well possibly by outside code e.g. client/code generators).

Also, surprising things might happen if an implementation tries to $ref to a particular object and it is the wrong type -- it is much easier to know the kind of all objects so we can prevent $refs from going to the wrong place before we try to evaluate the keywords found there -- it saves a lot of error checking in the target code, which can just assume that it's using the right type. (In my implementation I keep a table of OAD locations -> entity type, which is easily constructed while performing the initial parse and validation pass on the document, and throw an error if a $ref tries to go the wrong type.)

[handrews]

@karenetheridge

an implementation needs to know what type of object it is so it can examine it for embedded schemas

Also, surprising things might happen if an implementation tries to $ref to a particular object and it is the wrong type

This is addressed in the "Structural Interoperability" sections in 3.0.4, 3.1.2 (and 3.1.1), and the first two subsections of Appendix G of 3.2.

When I mentioned parsing and referencing ambiguities earlier in this thread, @ioggstream commented:

These should probably be left to the relevant OAS specs. We can just warn implementers.

which is probably the best way to handle these details. Older implementations try to do these things, and either deal with it already or don't. Newer implementations should be coming from the 3.1.1+ / 3.2+ world and shouldn't do them at all, and therefore shouldn't have these problems anyway.

[ioggstream]

@handrews this new text provides:

• a way to express the client preference for referenced resources in terms of OAS entry document (e.g., Accept: application/openapi+yaml; version=3.0
• the server to reply with the appropriate media type for the actual returned resource (e.g., Content-Type: application/yaml).

Since application/schema+json is not defined yet, and it has no associated parameters, I'd just use application/whatever.

[ioggstream]

Replying to @handrews @karenetheridge comment from an outdated thread

@ioggstream this does not address the concerns that @karenetheridge and I have been raising about using the same media type for different formats. I can't support this direction.

In the latest text above I tried to address this, and it does not use the same media type for different formats;
instead allows the client to express the preferred media type for a resource.

The client doesn't know the media type of a $ref until the response arrives,
and content negotiation does not involve fragments because:

1. fragments are processed by User Agents
2. they do not appear in requests (see https://www.rfc-editor.org/rfc/rfc9110.html#uri.fragment.identifiers and https://www.rfc-editor.org/rfc/rfc9110.html#name-disclosure-of-fragment-afte)

This means that:

1. $ref: foo.yaml#/components/schemas/Foo triggers the following request;
2. the server doesn't know that the client just wants the #/components/schemas/Foo fragment;
3. it can reply with whatever media type it deems appropriate for the resource it is serving

GET /foo.yaml
Accept: application/openapi+yaml; version=3.0
...

If we want to allow treating documents with other Object types as root objects,
we need a parameter indicating the expected root Object type, or something
similar. I believe @karenetheridge, @darrelmiller , and I are all aligned on
the principle that only a document with an OpenAPI Object at the root is an
unqualified application/openapi+* resource.

This should be addressed via https://github.com/ietf-wg-httpapi/mediatypes/pull/121/files#diff-b22c3a369e1499074590046ad01b71334e21c491c64ccf045f8858e2d3036e56R266

Reference to Schema Objects have been removed.

@ioggstream I would like to hear you address the objections @darrelmiller raised in #110 (comment) — while you have stated your goal several times, you have not addressed the objections.

IMHO the text above does not formalize the existence of fragments of OAS documents,
and the section mentioning "partial" resources present in the previous version of this PR has now been removed.

Thanks again everyone for your help in addressing this topic ❤️

[handrews]

@ioggstream I appreciate that you are operating in good faith here, but I do not feel we are getting much closer to agreement. Perhaps there is a bit of a language barrier issue going on, but also I do not feel like we've really sorted out the concepts here, and we're not going to be able to agree on PR text until we agree on the concepts.

Since this discussion is not moving us forward, I'm going to put up an alternative PR to better illustrate my concerns. We don't necessarily have to merge that PR, but maybe it will at least get us to the point where we're not talking past each other, which I feel is what is happening now.

[ioggstream]

@handrews OK. Can you include a content negotiation exchange to your PR, e.g. maybe in a FAQ or in an example :)

[handrews]

@ioggstream I have not been able to write up a PR today but I have worked through more thoughts and will come back with an update hopefully tomorrow. Please bear with me, my schedule is rather complicated at the moment.

[ioggstream]

@ilovelinux enjoy

[handrews]

@ioggstream let's take a big step back and try to sort out the conceptual framework here.

My understanding of content negotiation is basically as follows (ignoring multiple media types using q-values in Accept as that is pretty orthogonal to our concerns, AFAICT):

• The client asks for a representation in some media type
• The server is generally expected to send back a representation in that media type

There are, of course, several ways in which this might not quite happen:

• The server can send back a more generic media type (e.g. application/json instead of application/openapi+json), which I will discuss more below
• The server can send back an error indicating that it can't fulfill the media type request, e.g. 406 or 415, which is self-explanatory and we need not consider it further
• The server can just completely ignore the media type requested in Accept, which (while allowed by RFC9110) is essentially an error as far as most clients are concerned, so I am not going to pay attention to this option either

Regarding the "more generic media type" option, my expectation if I were writing a client is that the Content-Type: application/json representation would still be the same as what we would get if it set Content-Type: application/openapi+json. This is more about whether the server is configured to use the most specific media type or not. If I ask for an OpenAPI resource and get back some arbitrary other JSON document, there's not much a client can be expected to do with that. In fact, attempting to use such a document could potentially introduce a security vulnerability.

So if I request Accept: application/openapi+json, then whether that comes back as Content-Type: application/openapi+json or Content-Type: application/json, I expect it to have an OpenAPI Object at the root, with the appropriate openapi field (or the 2.0 equivalent, but let's ignore 2.0 for now, that's what the version parameter is for anyway).

I have never seen anything where you ask for media type and get back something that does not conform to that media type at all, and then somehow you are expected to handle it anyway. A document that contains some other Object at its root is in this category.

Note that this has nothing to do with URI fragments. Retrieving a representation with an OpenAPI Object at the root and then using a fragment to access an Object of a different type within that document is fine. No one has an issue with that. It's only when the root Object type is different.

To me, returning a non-OpenAPI-Object-rooted document when application/opnapi+json is requested is objectively wrong, and no amount of wordsmithing this PR will change that view. If you think that I'm wrong, I will need to see an RFC or other specification that clearly explains how this is a valid sort of content negotiation. That is the only way I'm willing to consider what you seem to be proposing here (or if I have misunderstood your proposal, please do let me know, but I can't see any other way to read it right now).

---

The client doesn't know the media type of a $ref until the response arrives

The above statement is not compatible with the OAS, particularly not 2.0 and 3.0. In 2.0 it is explicitly given that the $ref context is the only way you can possibly know what to expect. In fact, if you $ref the same document in different contexts, in OAS 2.0 (explicitly) and in most cases OAS 3.0 (implicitly, by analogy with 2.0 because there is no clear wording on this in 3.0), implementations are expected to interpret that same JSON (or YAML) fragment in two different ways based on the two different $ref contexts.

Please do not ask me to explain or defend this decision, as I was not involved and put a great deal of effort into removing it or at least discouraging it in OAS 3.1+. Recent versions of the OAS explain how this can be a problem (3.0.4, similar sections exist in 3.1.1, 3.1.2, and 3.2.0).

So not only is it untrue that the client doesn't know what media type to expect, it is actually the server which has no idea what the semantics are of the document being returned. Only the client has the context for interpreting the document.

This means that only the client is fully capable of realizing that $ref: /api/description/components/foo is (for example) a Path Item Object and requesting Accept: application/openapi+json; component=Path Item. Or that it's a Schema Object and requesting Accept: application/openapi+json; component=Schema; application/schema+json.

In fact, the same resource might parse either as a valid Path Item Object or as a valid Schema Object, and OAS 2.0 implementations (and many OAS 3.0 implemenations, and possibly 3.1+ implementations) will happily do exactly that.

Particularly in 3.0 and earlier, the Object Type is determined by the referencing context.

Again, don't ask me to defend this. I bashed my head against it for years. There are long discussions of this on the OpenAPI Slack. This is definitely what the Swagger folks intended, according to multiple people involved at the time.

---

content negotiation does not involve fragments

Yes, @karenetheridge and I are well aware of this, we are not talking about URI fragments at any point here, although it's understandable that you would think so. There are difficulties involving fragments and the 2.0 / 3.0 parsing models, they are not relevant here. "Fragmentary documents" are just documents without an OpenAPI Object at the root, they have nothing to do with URI fragments.

IMHO the text above does not formalize the existence of fragments of OAS documents

Because a "fragment of an OAS document" is just a document with something other than an OpenAPI Object at the root, and you are trying to treat such documents as valid resposnes to Accept: application/openapi+json, as far as I can tell your proposal indeed does formalize their existence.

In OAS 3.1 and 3.2, we get around this by treating standalone Schema Objects as JSON Schema resources, and saying that all other Object types are only interoperable if they are in a proper OpenAPI document with an OpenAPI Object at the root.

[ioggstream]

@handrews Thanks for your long reply.

In OAS 3.1 and 3.2, we get around this by treating standalone Schema Objects as JSON Schema resources, and saying that all other Object types are only interoperable if they are in a proper OpenAPI document with an OpenAPI Object at the root.

This can be pointed at in the Interoperability Considerations.

To me, returning a non-OpenAPI-Object-rooted document when application/openapi+json is requested is objectively wrong.

I understand your point of view, but I re-read the whole HTTP and I do not agree (see https://www.rfc-editor.org/rfc/rfc9110.html#section-12.4.3).
IMHO what you describe is achieved via:

• Accept: application/openapi+yaml;q=0.8, */*;q=0

While it is ok to state that the media type openapi+yaml only applies to OpenAPI Objects,
a server is free to reply with the representation it deems more appropriate based on the request.

[handrews]

@ioggstream

I understand your point of view, but I re-read the whole HTTP and I do not agree

I don't agree with your reading. If you're dead-set on this, I'll post a PR to remove my name from this RFC. I think it is the wrong approach, and there are better options that serve the use case more clearly, with much better control and more self-descriptive media types.

You seem uninterested in engaging with the alternate suggestions. I have not heard anything from you about why a component or similar parameter would be a bad solution. Perhaps I am misreading you, but you seem intent on ignoring the possibility.

Regardless, I do not want my name on a spec that requires using application/openapi+yaml or application/openapi+json for something that does not have an OpenAPI Object at the root. I put a tremendous amount of effort into the 3.1 and 3.2 lines to ensure that only documents with an OpenAPI Object at the root are considered proper, fully interoperable OpenAPI documents.

I'm plenty willing to add mechanisms to work around the limitations of 3.0, but I can't support an approach that, in my view, undermines my own work. And violates what I think is the spirit of content negotiation.

[karenetheridge]

To me, returning a non-OpenAPI-Object-rooted document when application/openapi+json is requested is objectively wrong.

I understand your point of view, but I re-read the whole HTTP and I do not agree (see https://www.rfc-editor.org/rfc/rfc9110.html#section-12.4.3).

So I think Henry is saying "if you send an Accept header with application/openapi+json, we're signalling we expect a valid OpenAPI document back, and it would be wrong to send anything else." But what do we use in Accept if we're resolving a reference to a plain old JSON Schema document instead? We ought to use application/schema+json.

I think the problem here is that it is not unambiguously obvious which type of document needs to be requested, based solely on the URI, without using heuristics on the json pointer fragment. Intuitively, I know that $ref: "https://example.com/api#/components/schemas/MySchema" is coming from a Schema Object in an openapi document, and $ref: "https://example.com/schema#/$defs/MySchema" is coming from a JSON Schema document, but should the application be able to tell? If it can't/won't, then we don't know what media type to put in the Accept header of the request.

Roberto seems to be saying "just ask for application/openapi+json, and maybe you'll get something else back instead, and that's ok" and Henry is saying "that's not ok".

Did I capture this dilemma correctly?

[minor edits for typos]

edit 2:

• or is it that Roberto is saying "you can ask for a certain media type, but you won't always get that type, and that's okay", and Henry is saying "the Content-Type of the document you get back must always be correct - don't call it application/openapi+json if it's not an OpenAPI document" -- and if this is more accurate, then there is no disconnect: Because content negotiation (as I understand it) does allow for you getting back a different type than what you asked for. But Henry is also right that the Content-Type must be accurate. So therefore:

  • client sends: Accept: application/openapi+json (the client is saying they expect an openapi document back)
  • server responds with: Content-Type: application/schema+json (the server is saying "you asked for an openapi document, but this is actually a json schema document, so I've labelled it as such").

[karenetheridge]

(I edited my last response - please read in web rather than email)

[handrews]

Thanks, @karenetheridge , you are correct, and you bring up a good point about the ambiguity. Which (as you know) is why we say in more recent versions of the OAS that you can only be sure to understand a referenced document if it:

• has an OpenAPI Object at the root
• can be treated as a JSON Schema document
• the $ref does not have a fragment (although this case is not always interoperable, while the previous two are)

So there are two problems here:

First, how to ask for the right thing?

Sometimes you know what to ask for, down to the Object type, but sometimes you don't. I do not object to using application/openapi+yaml (without any sort of component parameter, even if one is supported) when you can't tell what the target document is. Or even if you can tell that it's a JSON Schema document, but you might need to negotiate based on OAS version (I'd rather negotiate on metaschema, but that's a whole different unresolved can of worms so let's ignore it).

This "what to ask for" ambiguity hasn't been called out as clearly before, so thanks for that! I feel like it is the easier half.

Second, how do you know what you've gotten back?

As @karenetheridge has observed in her second edit, this is where the main disconnect is.

I don't want something without an OpenAPI Object at the root to be given Content-Type: application/openapi+yaml. I think @ioggstream is agreeing to that.

One possibility is that it comes back as Content-Type: application/schema+json (possibly with a schema parameter indicating the metaschema). I'm fine with that and @karenetheridge seems fine with that. I'm not sure whether @ioggstream is fine with it. Ideally, the user would request Accept: application/openapi+yaml, application/schema+yaml, application/schema+json when a schema is needed, but whether or not they do that is not our problem.

The final case is where the disconnect is: The document has neither an OpenAPI Object nor a JSON Schema at the root, e.g. the case that is necessary in OAS 3.0 of having a Path Item Object in its own document/resource. @ioggstream seems to be saying "that's fine, just send back application/yaml, and that's where I disagree. How do I know what to do with that? How do I know that this isn't some random irrelevant YAML document that the server decided to send me?

---

And this is where my point of "I'm willing to put in things to work around OAS 3.0's limitations" come in. It's fairly compatible with 3.0 (and definitely 2.0) to just parse some YAML and attempt to interpret the portion of the file that was referenced as whatever it is. So application/yaml is arguably correct, especially for 2.0.

But I really, really, really do not want to encourage this behavior in 3.1 and later. I don't want to imply that this is correct behavior when asking for documents because we've tried so hard to move away from that.

Maybe there's no other option. But I'd prefer to be able to reply application/openapi+yaml; component=Path Item if the resource is a Path Item Object. And yes, if there's nothing else that can be said about it (because it really is arbitrary non-OpenAPI YAML with a snippet of OpenAPI content embedded somewhere deep in it), then that can be application/yaml.

But I don't want that to be endorsed. I want servers that return known OpenAPI content to be able to indicate that in the Content-Type field somehow.

[handrews]

OK, so... attempting to be productive and realistic here.

• I most want to avoid explicitly advising people to send back application/yaml when any other media type could possibly be relevant.

For example, I would rather say that a schema could be returned "with a suitable media type for a JSON Schema resource" rather than explicitly showing application/yaml. This could be done without modifying anything else.

• My next priority, which is more negotiable, is that I would prefer that responses are as self-descriptive as possible.

A component media type parameter could facilitate this (including for Schema Objects, neatly side-stepping the lack of a formal JSON Schema media type).

The drawback is the one that @darrelmiller noted, which is that it makes what we call fragmentary documents (anything other than an OpenAPI Object at the root, or a standalone JSON Schema) more legitimate.

A possible middle ground could be some other media type or media type parameter to indicate ambiguous OpenAPI content. This might be preferred, because it would correctly express that such usage is not always interoperable, which would be a great outcome.

So perhaps: Accept: applicatin/openapi+yaml; version=3.0 for a resource consisting of only a Path Item Object could be returned as Content-Type: application/openapi-fragment+yaml; version=3.0 or Content-Type: application/openapi+yaml; version=3.0; fragment=true. We could then include a note in interoperability considerations that such responses are not always interoperable.

• I do recognize that in some scenarios, particularly with OAS 2.0, there is nothing that can be done but Content-Type: application/yaml. I would prefer not to mention this at all. If people are doing it, they can do that. But I don't think we should encourage it.

[ioggstream]

This interoperability considerations is warning client implementers that:

1. servers (e.g., github.com) are not required to implement content negotiation;
2. if a client wants to interoperate with such a server, it needs to take into account that behavior.

See https://github.com/ietf-wg-httpapi/mediatypes/pull/132/changes#r2617236017

If the wording is not clear, feel free to improve it.

[ioggstream]

Hi @handrews and @karenetheridge, and thanks again for all your time :)

Karen> it is not unambiguously obvious which type of document needs to be requested

This is exactly the point, thanks @karenetheridge.

Henry> Sometimes you know what to ask for, down to the Object type, but sometimes you don't.
Henry> I do not object to using application/openapi+yaml
Henry> (without any sort of component parameter, even if one is supported)
Henry> when you can't tell what the target document is.

I think that version is key here, so the difference is just that?
I drafted some scenarios including a real world one (github.com)
with requests/responses here https://github.com/ietf-wg-httpapi/mediatypes/pull/132/changes

Henry> how do you know what you've gotten back
Henry> I don't want something without an OpenAPI Object at the root to be given Content-Type: application/openapi+yaml. I think @ioggstream is agreeing to that.

OK.

Henry> I most want to avoid explicitly advising people to send back application/yaml when any other media type could possibly be relevant.

While I think the PR does not advise to send back application/yaml,
instead it warns clients that servers may not behave as the client wishes (e.g., see github example below), feel free to propose an alternate text.

Henry> [..the response..] comes back as Content-Type: application/schema+json

github/gitlab always returns Content-Type: text/plain; charset=utf-8, so if an implementation
wants to resove $refs from github/gitlab it should always process the content even if the content-type is not application/{openapi+,}{yaml,json}.

Henry> ...The document has neither an OpenAPI Object nor a JSON Schema at the root...
Henry> e.g. the case that is necessary in OAS 3.0 of having a Path Item Object in its own document/resource
Henry> How do I know what to do with that? How do I know that this isn't some random irrelevant YAML document.

A User Agent cannot always know from the Content-Type.
Some implementations will decide to process anything including text/plain, others
to only process content-type.endswith(('yaml','json')), others will require a strict match.

A client that decides not to process a Content-Type: text/plain response for example, will not be able to process any $ref resource that's on github.

We thus need to warn implementers that - even if that's not something we endorse - this is what actually happens in the web in case they need to interoperate

Henry> ... component parameter ...

I'm not against a component parameter, I just think that Darrel has a point. We are always free to add it in the future.

HTH + ☮️
R

[handrews]

After thinking on this, I came to a few conclusions:

• I have been fairly burnt out on spec work since the tail end of OAS 3.2, and meant to take December completely off to reset. I should have stuck with that plan, as I think I'm still reacting from a place of burn-out.
• @ioggstream , I think you and I have different use cases and goals for this media type
• It seems like I either do not understand the principles of content negotiation well or deeply enough to review this PR, or it's just that, within the range of what an be done, I prefer a different direction
• There are too many things about the OAS parsing and referencing models, particularly 2.0 and 3.0, that frustrate me, and I keep wanting to help patch over them with the media type rules, which is a goal no one else seems to share (which is fine... maybe just returning application/yaml is the most practical solution, and there are trade-offs in any direction).

So I am going to unblock this effort by bowing out. I do not have the mental or emotional energy to sort through all of my frustrations and burn-out (which are very much my problems and not yours), and I don't think we're trying to reach the same goals anyway.