feat: token-threshold strategies, shared utilities, and docs update (v2.1.0)

[IBJunior]

This PR migrates slimcontext strategies to token-threshold logic and updates docs, tests, and examples accordingly.

Summary

• Strategies refactor:
  • TrimCompressor now trims based on token thresholds using model max tokens + configurable threshold.
  • SummarizeCompressor summarizes older messages when over threshold and inserts a system summary before recent messages.
• Shared utilities:
  • New src/strategies/common.ts with TokenBudgetConfig, DEFAULT_* constants, and helpers (normalizeBudgetConfig, computeThresholdTokens, DEFAULT_ESTIMATOR).
• Interfaces:
  • Introduced TokenEstimator type.
• Adapter/docs:
  • LangChain adapter docs updated; one-call helper supports token-threshold options.
  • README, examples, and internal Copilot instructions updated.
• Tests:
  • Updated to deterministic token estimators; all passing (3 files, 12 tests).
• Version:
  • Bumped to 2.1.0 to match CHANGELOG.

Migration notes

• Replace message-count configs with token-based fields:
  • Trim: { maxModelTokens?, thresholdPercent?, estimateTokens?, minRecentMessages? }
  • Summarize: { model, maxModelTokens?, thresholdPercent?, estimateTokens?, minRecentMessages?, prompt? }
• Defaults: maxModelTokens=8192, thresholdPercent=0.7, minRecentMessages (trim=2, summarize=4), estimator ≈ len/4 + 2.

Validation

• Build: PASS
• Tests: PASS (3 files, 12 tests)
• Format: PASS

Closes: n/a

[Copilot]

Pull Request Overview

This PR migrates slimcontext from message-count based compression to token-threshold based compression strategies and updates to version 2.1.0. The key changes focus on making compression decisions based on estimated token usage relative to the model's context window rather than fixed message counts.

• Refactored TrimCompressor and SummarizeCompressor to use token thresholds with configurable max tokens and threshold percentages
• Added shared token budget utilities and a default token estimator in src/strategies/common.ts
• Updated all tests, examples, and documentation to reflect the new token-based configuration approach

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/trim.test.ts	Updated test to use token-threshold configuration with deterministic estimator
tests/summarize.test.ts	Simplified tests to focus on token-threshold behavior, removed complex alignment tests
tests/langchain.test.ts	Updated LangChain adapter tests to use new token-based configuration
src/strategies/trim.ts	Complete rewrite to implement token-threshold trimming logic
src/strategies/summarize.ts	Refactored to use token budgets and simplified summarization logic
src/strategies/common.ts	New file with shared token budget utilities and default estimator
src/interfaces.ts	Added TokenBudgetConfig interface and TokenEstimator type
src/adapters/langchain.ts	Updated adapter documentation and type definitions for token-based options
package.json	Version bump to 2.1.0 and prepare script change
examples/*.md	Updated all examples to show new token-based configuration
README.md	Comprehensive documentation update reflecting token-threshold approach
CHANGELOG.md	Added v2.1.0 entry with breaking changes and migration guide
.github/copilot-instructions.md	Updated internal documentation to reflect new architecture