normalizer.cpp

Jump to bottom Edit New page

makr-code edited this page Dec 21, 2025 · 1 revision

normalizer.cpp

Stand: 5. Dezember 2025
Version: 1.0.0
Kategorie: Src

Path: src/utils/normalizer.cpp

Purpose: Text normalization helpers used by tokenization and search.

Public functions / symbols:

static inline bool is2(unsigned char a, unsigned char b, unsigned char x, unsigned char y) {
for (size_t i = 0; i < n; ) {
if (c == 0xC3 && i + 1 < n) {

ThemisDB v1.3.0 | GitHub | Documentation | Discussions | License

Last updated: December 20, 2025

Übersicht
Home
📋 Dokumentations-Index
📋 Quick Reference
📊 Sachstandsbericht 2025
🚀 Features
🗺️ Roadmap
Ecosystem Overview
Strategische Übersicht
Architektur
- Überblick
- Geo-Architektur
Basismodell
- Base Entity & Keys
- Pfad-Constraints
- Property Graph Modell
Storage & MVCC
- Storage-Layout (Geo/Relational)
- RocksDB Storage
- MVCC-Design
- Transaktionen
- Time-Series Überblick
- Memory Tuning
- Chain of Thought Storage
Indexe & Statistiken
- Indexe
- Index-Statistiken & Wartung
- Index Backup
- Cursor/Pagination
Query & AQL
- Query Engine & AQL
- AQL Syntax
- Explain & Profile
- Rekursive Pfadabfragen
- Temporale Graphen
- Zeitbereichs-Abfragen
- Semantischer Cache
- Hybrid Queries (Phase 1.5)
- AQL Hybrid Queries
- Hybrid Queries README
- Hybrid Query Benchmarks
- Subquery Quick Reference
- Subquery Implementation Summary
Caching
- Cache Invalidation Strategy
- Caching Data Structures
- Caching Lookup Patterns
Content Pipeline
- Content Pipeline
- Architektur-Details
- Ingestion
- JSON Ingestion Spec
- Enterprise Ingestion Interface
- Geo-Processor Design
- Image-Processor Design
Suche
- Hybrid Search Design
- Fulltext API
- Hybrid Fusion API
- Stemming
- Performance Tuning
- Migration Guide
- Future Work
- Pagination Benchmarks
Performance & Benchmarks
- Überblick & Tuning
- Kompression Benchmarks
- Kompression Strategie
- Encryption Metrics
- Pagination
Enterprise Features
- Übersicht
- Scalability Features
- HTTP Client Pool
- Build Guide
- Implementierungs-Status
- Final Report
- Integration Analysis
- Enterprise Strategy
Qualitätssicherung
Vektor & GNN
- Vektor-Operationen
- GNN Embeddings
- HNSW Persistenz & Warmstart
Geo Features
- Geo 3D Games Acceleration
- Geo Execution Plan over Blob
- Geo Feature Tiering
- Geo Research Report MVP
Sicherheit & Governance
- Überblick
- RBAC & Authorization
- RBAC
- Policies (MVP)
- Authentication
  - JWT
  - Benutzerverwaltung (Admin)
- Schlüsselverwaltung
- Verschlüsselung
- TLS & Certificates
  - TLS Setup
  - Certificate Pinning
- PKI & Signatures
- PII Detection
- Vault & HSM
  - Vault
  - HSM Integration
- Audit & Compliance
- Security Audits & Hardening
- Competitive Gap Analysis
Deployment & Betrieb
- Deployment
- Docker
- Tracing & Observability
- Observability
  - Prometheus Metrics
  - Metrics
- Change Data Capture
  - Change Data Capture (CDC)
  - CDC
- Operations Runbook
- Infrastructure Roadmap
- Horizontal Scaling Implementation Strategy
Entwicklung
- Übersicht
- Code Quality Pipeline
- Developers Guide
- Cost Models
- Todo Liste
- Tool Todo
- Core Feature Todo
- Priorities
- Implementation Status
- Roadmap
- Future Work
- Next Steps Analysis
- AQL LET Implementation Guide
- Development Audit
- Sprint Summary (2025-11-17)
- WAL Archiving
- Search Gap Analysis
- Source Documentation Plan
- API Implementations
  - Audit API Implementation
  - SAGA API Implementation
  - Code Audit Mockups/Stubs
- Changefeed
  - Changefeed README
  - Changefeed CMake Patch
  - Changefeed OpenAPI
  - Changefeed OpenAPI Auth
  - Changefeed SSE Examples
  - Changefeed Test Harness
  - Changefeed Tests
- Security Development
  - Security README
  - Content ZSTD HKDF
  - PKI-eIDAS
- Development Overviews
  - Overview README
  - Consolidated Development Overview
  - Feature Status (Changefeed & Encryption)
  - Verification by Area
Publikation & Ablage
Admin-Tools
- Admin Guide
- User Guide
- Feature Matrix
- Suche/Sortierung/Filter
- Demo-Script
APIs
- OpenAPI & Endpunkte
- SSE-Streaming (Changefeed)
- ContentFS API
- Hybrid Search API
Client SDKs
- JavaScript SDK Quickstart
- Python SDK Quickstart
- Rust SDK Quickstart
Implementierungs-Zusammenfassungen
- ThemisDB Implementation Summary
- Database Capabilities Roadmap
- Release Scope Core
Planung & Reports
- Phase 1.5 Completion Report
- Phase 2 Plan
- Phase 3 Plan
- Phase 4 Plan
- Sprint A Plan
Dokumentation
- Dokumentations-Inventar
- Documentation Summary
- Documentation TODO
- Documentation Gap Analysis
- Documentation Consolidation Plan
- Documentation Final Status
- Documentation Phase 3 Report
- Documentation Cleanup Validation Report
Release Notes
- AQL Fulltext
- Temporal Aggregation (2025-11-11)
Styleguide & Glossar
- Styleguide
- Glossar
Roadmap
Changelog
Source Code Documentation
- Übersicht
- Source Documentation
- Main
- Main (Detailed)
- Main Server
- Main Server (Detailed)
- Demo Encryption
- Demo Encryption (Detailed)
- API
  - API README
  - HTTP Server
- Authentication
  - Auth README
  - JWT Validator
- Cache
  - Cache README
  - Semantic Cache
- CDC
  - CDC README
  - Changefeed
- Content
  - Content README
  - Content Manager
  - Content Type
  - Text Processor
- Geo
  - Geo README
  - CPU Backend
  - GPU Backend Stub
- Governance
  - Governance README
  - Policy Engine
- Index
  - Index README
  - Adaptive Index
  - GNN Embeddings
  - Graph Index
  - Property Graph
  - Secondary Index
  - Vector Index
- LLM
  - LLM README
  - LLM Interaction Store
- Query
  - Query README
  - AQL Parser
  - AQL Translator
  - Query Engine
  - Query Optimizer
  - Query Parser
  - Semantic Cache
- Security
  - Security README
  - Encrypted Field
  - Field Encryption
  - Key Cache
  - Mock Key Provider
  - PKI Key Provider
  - Vault Key Provider
- Server
  - Server README
  - [VCCDB Design](src/server/VCCDB Design.md.md)
  - Audit API Handler
  - Auth Middleware
  - Classification API Handler
  - HTTP Server
  - Keys API Handler
  - PII API Handler
  - Policy Engine
  - Ranger Adapter
  - Reports API Handler
  - Retention API Handler
  - SAGA API Handler
  - SSE Connection Manager
- Storage
  - Storage README
  - Base Entity
  - Key Schema
  - RocksDB Wrapper
- Time Series
  - Time Series README
  - Continuous Aggregation
  - Gorilla Compression
  - Retention
  - Time Series
  - TS Store
- Transaction
  - Transaction README
  - SAGA
  - Transaction Manager
- Utils
  - Utils README
  - Audit Logger
  - Cursor
  - HKDF Helper
  - LEK Manager
  - Logger
  - Normalizer
  - PII Detection Engine
  - PII Detector
  - PII Detector (Old)
  - PII Pseudonymizer
  - PKI Client
  - Regex Detection Engine
  - Retention Manager
  - SAGA Logger
  - Serialization
  - Stemmer
  - Stopwords
  - Tracing
  - ZSTD Codec
Archive
- Archive README
- CDC Legacy
- Geo Research Report MVP
- Path Constraints Concept
- Release Scope Core Draft
- Merge Reports
  - Feature Complete Database Capabilities Conflicts

Clone this wiki locally