[Agent][Platform][OpenAI] Add stream usage support

[oesteve]

Q	A
Bug fix?	no
New feature?	yes
Docs?	no
Issues	Fix #753
License	MIT

$result = $agent->call($messages, [
    'stream' => true, // enable streaming of response text
    'stream_options' => [
        'include_usage' => true, // include usage in the response
    ],
]);

/** @var TextChunk $textChunk */
foreach ($result->getContent() as $textChunk) {
    // $textChunk implement \Stringable
    echo $textChunk->getContent();

    // Chunk also contain metadata
    // $textChunk->getMetadata()->get('id'); // Stream id
}

// Output token usage statistics for each call
foreach ($result->getMetadata()->get('calls', []) as $call) {
    echo \PHP_EOL.sprintf(
        '%s: %d tokens - Finish reason: [%s]',
        $call['id'],
        $call['usage']['total_tokens'],
        $call['finish_reason']
    );
}

[OskarStark]

remove?

[OskarStark]

remove

[OskarStark]

Suggested change

// Output text chunks

[chr-hertel]

Thanks for tackling this - one thing that get's me thinking here is that different behavior from dev point of view with extra set of option and metadata key.

usually i'd activate the TokenOutputProcessor and expect to have the key token_usage on the result metadata. that expectation is not working with your current approach leading to an inconsistent API for developers in my opinion.

Is it possible from your point of view to get rid of the extra option and still use the token_usage key on the result metadata?
i get the issue with multiple calls - so maybe #1051 might help?

let me know what you think - thanks already!

[oesteve]

@chr-hertel , thanks for the feedback. Your approach with TokenUsageAggregation looks much cleaner and allows for the reuse of token_usage.


On the other side, the TokenOutputProcessor works well with regular results, but I can’t see how to make an OutputProcessorInterface work with streams. The usage is received at the end of the stream after the processOutput has been called.

[chr-hertel]

Yeah, makes sense - that looks tricky - you needed to do the metadata handling at various places.
Can we go for some kind of middle way tho - maybe skipping the processor but dropping that extra option (is it interpreted anyways?) and provide an aggregated object as token_usage on high level for the devs?