A production-ready AI framework combining explainability, adaptive learning, and ethical oversight for trustworthy cross-domain recommendation systems.
M.Sc. Thesis Project | Istanbul University - Management Information Systems
Author: Nazlı Özgür | Industry Partner: MindTech
Modern recommendation systems face a critical challenge: they optimize for accuracy but fail to address transparency, adaptability, and fairness. This creates three major issues:
- Black-box decisions → Users and regulators cannot understand why recommendations are made
- Static models → Systems fail to adapt to changing user behavior and data patterns
- Domain isolation → Organizations cannot transfer AI knowledge across business units
With EU AI Act enforcement beginning in 2025, organizations need AI systems that are not only accurate but also explainable, adaptive, and verifiably fair. Current solutions like Clerk.io provide recommendations but lack the transparency, ethical oversight, and cross-domain flexibility required by modern regulations and business needs.
XAE-Frame addresses these challenges by integrating explainability, adaptive learning, cross-domain transfer, and ethical monitoring into a unified framework designed for enterprise deployment.
XAE-Frame is a production-ready AI framework designed for cross-domain personalization with built-in explainability, adaptability, and ethical compliance. Unlike traditional "black-box" AI systems, XAE-Frame ensures:
- Transparency: Every prediction comes with clear, stakeholder-specific explanations (NIST-compliant)
- Adaptability: Automatic drift detection and retraining strategies keep models accurate over time
- Fairness: Continuous bias monitoring and mitigation aligned with EU AI Act requirements
- Cross-Domain: Transfer knowledge across e-commerce, finance, and insurance sectors
- Business Value: Direct mapping of AI performance to business KPIs (ROI, churn reduction, revenue)
- EU AI Act: Mandates transparency, accountability, and human oversight for high-risk AI systems
- GDPR Article 22: Right to explanation for automated decision-making
- Industry Regulations: Financial services (FCRA), insurance (actuarial fairness), healthcare (HIPAA)
According to McKinsey's 2022 research:
- Companies with strong XAI practices see 20%+ EBIT from AI initiatives
- Organizations establishing digital trust through explainability achieve 10%+ annual revenue growth
- XAI is not just compliance—it's a strategic enabler of adoption and business success
- Black-box models (deep learning, ensemble methods) offer high accuracy but low interpretability
- Model drift degrades performance over time without detection mechanisms
- Algorithmic bias can lead to unfair outcomes and reputational damage
- Domain silos prevent organizations from leveraging AI knowledge across business units
XAE-Frame solves these challenges by integrating explainability, adaptability, ethics, and cross-domain transfer into a single, cohesive framework.
Transfer user preferences and behavioral patterns across different business domains (e-commerce, finance, insurance). The framework enables meaningful knowledge transfer through:
- Sector Adaptation Modules: Config-based deployment across e-commerce, finance, and insurance
- Domain-agnostic pipeline: Same LightGBM architecture, SHAP explainability, and fairness monitoring
- Rapid deployment: 1-2 weeks per new sector using configuration templates
- Preserved explainability: SHAP values remain interpretable across domain boundaries
Key benefit: Solve cold-start problems in new domains by leveraging rich knowledge from existing deployments.
Built on SHAP (SHapley Additive exPlanations) with full compliance to NIST's Four Principles of Explainable AI:
- Explanation: Every prediction comes with evidence-based reasoning
- Meaningful: Context-aware explanations adapted to user expertise (technical vs. business)
- Explanation Accuracy: Fidelity metrics ensure explanations truly reflect model behavior (>0.90)
- Knowledge Limits: Confidence thresholds and out-of-distribution detection prevent unreliable predictions
Five types of explanations aligned with NIST standards:
- User benefit: Simple "why" for end users
- Societal acceptance: Trust-building transparency
- Regulatory compliance: Full audit trails
- System development: Technical debugging insights
- Owner benefit: Business value quantification
Stakeholder-specific views:
- Technical users: Detailed feature importance with statistical measures
- Business users: Simplified, actionable insights with KPI mapping
- Compliance officers: Full decision trail with fairness metrics
- Executives: ROI impact and business value dashboard
NEW in v3.5: Complete 6-component integration for production-grade adaptability:
- Redis-based feature store for <100ms predictions
- Session-based context tracking
- Online feature computation with 5-minute cache TTL
- Future-proof: User expectations (Amazon/Netflix-level UX), standard by 2030
Beyond standard drift monitoring - tracks business-critical metrics:
- Click-Through Rate (CTR) monitoring
- Conversion rate trend analysis
- Session duration anomaly detection
- Statistical tests: KS-test, Mann-Whitney U (p-value < 0.05)
- Why critical: McKinsey reports "67% of ML failures due to drift"
MLOps-grade automated retraining with zero human intervention:
- Multi-trigger system: Drift detection, performance drop >10%, time-based (weekly/monthly)
- Champion/Challenger A/B testing: 90/10 split validation before rollout
- Sliding window data selection: Last 90 days for optimal performance
- MLflow integration: Full version control and experiment tracking
- Automatic rollback: If challenger underperforms, champion is restored
Continuous fairness monitoring every 1000 predictions:
- Demographic Parity (threshold >0.80)
- Equal Opportunity metrics
- Disparate Impact analysis
- Auto-pause: Model stops if fairness <0.70
- Legal compliance: EU AI Act (2024) mandatory requirement
Blockchain-inspired tamper-proof logging:
- Hash-chain verification ensures no backdated changes
- Full decision trail: timestamp, user ID, model version, features, SHAP values, fairness score
- 90-day hot storage, 7-year cold archive
- Export formats: JSON, PDF, CSV (regulator-ready)
- Critical for: EU AI Act Article 12 (full traceability)
Automated compliance reporting:
- EU AI Act Articles 10-15 checklist
- GDPR Article 22 (right to explanation)
- Auto-generated PDF audit reports
- Risk assessment templates
- Updates: As regulations evolve (framework designed for extensibility)
Loop orchestration: Monitor → Detect Drift → Trigger Retrain → A/B Test → Audit → Deploy
Continuous bias monitoring across demographic groups using Fairlearn and AIF360:
- Automated fairness metrics: Demographic Parity, Equal Opportunity, Disparate Impact
- Regular audits to detect bias amplification during cross-domain transfers
- Pre/in/post-processing mitigation strategies
- Protected attribute monitoring (age, gender, location)
EU AI Act compliance features:
- ✅ Transparency: Full decision traceability and explainability
- ✅ Accountability: Automated audit logs and compliance reports
- ✅ Fairness: Real-time bias detection with mitigation strategies
- ✅ Robustness: Drift monitoring and adversarial testing
Quantifies AI value through metrics that matter to stakeholders. The framework includes a metric mapping layer that connects technical AI performance to business outcomes:
| XAI Metric | Business KPI |
|---|---|
| Explanation Quality | Customer Trust (surveys) |
| Model Confidence | Conversion Rate |
| Fairness Score | Brand Reputation (NPS) |
| Drift Detection | Revenue Stability |
Real-time dashboard tracks:
- Revenue lift (baseline vs. XAE-Frame)
- Conversion rate improvement
- Churn reduction percentage
- Customer lifetime value (CLV) increase
- A/B test performance metrics
┌─────────────────────────────────────────────────────────────────────┐
│ XAE-FRAME ARCHITECTURE v3.5 │
│ (Explainable, Adaptive, Ethical AI Framework) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ LAYER 1: USER INTERFACE │
├─────────────────────────────────────────────────────────────────────┤
│ ┌──────────────┬──────────────┬──────────────┬─────────────────┐ │
│ │ Executive │ Data Science │ Compliance │ End User │ │
│ │ Dashboard │ Dashboard │ Dashboard │ Interface │ │
│ │ (ROI/KPI) │ (Technical) │ (Audit) │ (Simple Why) │ │
│ └──────────────┴──────────────┴──────────────┴─────────────────┘ │
│ • Multi-sector views (E-commerce, Finance, Insurance) │
│ • Streamlit/React-based multi-view interface │
│ • Role-based access control (RBAC) │
└────────────────────────────┬────────────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────────────┐
│ LAYER 2: ADAPTIVE LEARNING LOOP │
│ (Production-Ready + Future-Proof) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 2A. REAL-TIME PERSONALIZATION LAYER │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ • Redis Feature Store (in-memory cache) │ │
│ │ • <100ms prediction latency │ │
│ │ • Session-based context tracking │ │
│ │ • Online feature computation │ │
│ │ • Cache invalidation strategy (5min TTL) │ │
│ │ │ │
│ │ Why Critical: User expectations (Amazon/Netflix-level UX) │ │
│ │ Future-Proof: Standard by 2030, mandatory by 2040 │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 2B. BEHAVIOR DRIFT DETECTOR │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ Beyond standard drift - Business-critical metrics: │ │
│ │ • Click-Through Rate (CTR) monitoring │ │
│ │ • Conversion rate trend analysis │ │
│ │ • Session duration anomaly detection │ │
│ │ • Purchase funnel drop-off tracking │ │
│ │ • Statistical tests: KS-test, Mann-Whitney U │ │
│ │ • Alert thresholds: CTR drop >15%, p-value <0.05 │ │
│ │ │ │
│ │ Why Critical: McKinsey - "67% of ML failures due to drift" │ │
│ │ Future-Proof: Industry standard by 2030 │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 2C. AUTO-RETRAINING ENGINE │ │
│ ├────────────────────────────────────────────────────────────── ┤ │
│ │ MLOps-grade automated retraining: │ │
│ │ • Multi-trigger system: │ │
│ │ - Behavior drift detected (CTR/conversion) │ │
│ │ - Performance drop >10% │ │
│ │ - Time-based (weekly/monthly) │ │
│ │ - Manual trigger (compliance updates) │ │
│ │ • Champion/Challenger A/B testing (90/10 split) │ │
│ │ • Sliding window data selection (last 90 days) │ │
│ │ • MLflow integration (version control) │ │
│ │ • Automated rollback if challenger underperforms │ │
│ │ │ │
│ │ Why Critical: CI/CD for ML - no human intervention needed │ │
│ │ Future-Proof: Standard by 2030, mandatory by 2040 │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 2D. BIAS SCANNER (Real-Time Enhancement) │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ Continuous fairness monitoring: │ │
│ │ • Real-time checks every 1000 predictions │ │
│ │ • Fairness metrics (Fairlearn/AIF360): │ │
│ │ - Demographic Parity (threshold >0.80) │ │
│ │ - Equal Opportunity │ │
│ │ - Disparate Impact │ │
│ │ • Alert system (email, dashboard, logs) │ │
│ │ • Automatic model pause if fairness <0.70 │ │
│ │ │ │
│ │ Why Critical: EU AI Act (2024) - LEGAL REQUIREMENT! │ │
│ │ Future-Proof: Global regulation standard by 2030 │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 2E. AUDIT LOG ENGINE (Immutable Trail) │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ Blockchain-inspired immutable logging: │ │
│ │ • Hash-chain verification (tamper-proof) │ │
│ │ • Full decision trail: │ │
│ │ - Timestamp (UTC, millisecond precision) │ │
│ │ - User ID + Model version + Prediction ID │ │
│ │ - Input features + Output prediction │ │
│ │ - SHAP values (full explainability) │ │
│ │ - Fairness score + Drift status │ │
│ │ • Retention: 90 days (hot), 7 years (cold archive) │ │
│ │ • Regulator export: JSON/PDF/CSV │ │
│ │ │ │
│ │ Why Critical: EU AI Act Article 12 - Full traceability │ │
│ │ Future-Proof: Mandatory by 2030, immutable logs standard │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 2F. REGULATION PACK (EU AI Act Compliance) │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ Automated compliance reporting: │ │
│ │ • EU AI Act Articles 10-15 checklist │ │
│ │ • GDPR Article 22 (right to explanation) │ │
│ │ • Auto-generated audit reports (PDF) │ │
│ │ • Transparency documentation │ │
│ │ • Risk assessment templates │ │
│ │ │ │
│ │ Why Critical: Legal compliance - non-negotiable │ │
│ │ Future-Proof: Updates needed as regulations evolve │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Loop Orchestration: Monitor → Detect → Retrain → Audit → Deploy │
└────────────────────────────┬────────────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────────────┐
│ LAYER 3: EXPLANATION PERSONALIZATION ENGINE │
├─────────────────────────────────────────────────────────────────────┤
│ • Role Detection & Authorization │
│ • Explanation Complexity Adjustment (5 NIST Categories) │
│ • Language Simplification (Technical → Business) │
│ • Metric Mapping (XAI Metrics → Business KPIs) │
└────────────────────────────┬────────────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────────────┐
│ LAYER 4: CORE XAE-FRAME ENGINES │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 4A. EXPLAINABILITY (XAI) ENGINE │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ Core Components: │ │
│ │ • SHAP Value Computation (TreeExplainer) │ │
│ │ • Multi-Level Explanations (NIST 5 Categories) │ │
│ │ • Explanation Accuracy Tracker (fidelity >0.90) │ │
│ │ • Knowledge Limits & Confidence Module │ │
│ │ │ │
│ │ NIST 4 Principles Implementation: │ │
│ │ 1. Explanation: SHAP-based feature attribution │ │
│ │ 2. Meaningful: Stakeholder-specific views │ │
│ │ 3. Explanation Accuracy: Fidelity metrics │ │
│ │ 4. Knowledge Limits: Confidence thresholds + OOD detection │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 4B. ETHICAL AI MONITOR │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ • Bias Detection (Fairlearn / AIF360) │ │
│ │ • Bias Mitigation Strategies (pre/in/post-processing) │ │
│ │ • Protected attribute monitoring │ │
│ │ • Disparate impact analysis │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 4C. BUSINESS IMPACT MEASUREMENT ENGINE │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ • Real-Time Data Collection Architecture │ │
│ │ • XAI → Business KPI Mapping │ │
│ │ • ROI Calculator │ │
│ │ • A/B Test Framework │ │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────┬────────────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────────────┐
│ LAYER 5: CROSS-DOMAIN ADAPTATION │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 5A. SECTOR ADAPTATION MODULES │ │
│ ├──────────────────────────────────────────────────────────────┤ │
│ │ Config-based multi-sector deployment: │ │
│ │ │ │
│ │ E-Commerce (Primary Demo): │ │
│ │ • Target: Product recommendations │ │
│ │ • Features: Purchase history, reviews, ratings │ │
│ │ • Protected: Verified purchase status │ │
│ │ • Compliance: GDPR, Consumer Protection │ │
│ │ │ │
│ │ Finance (Adaptability Proof): │ │
│ │ • Target: Loan approval, credit scoring │ │
│ │ • Features: Income, credit score, debt ratio │ │
│ │ • Protected: Age, gender, location │ │
│ │ • Compliance: FCRA, GDPR, Basel III │ │
│ │ │ │
│ │ Insurance (Adaptability Proof): │ │
│ │ • Target: Risk assessment, policy recommendation │ │
│ │ • Features: Age, health score, driving record │ │
│ │ • Protected: Age, health status │ │
│ │ • Compliance: Actuarial Fairness, GDPR, ACA │ │
│ │ │ │
│ │ Same Pipeline, Different Sectors: │ │
│ │ ✅ Same LightGBM model architecture │ │
│ │ ✅ Same SHAP explainability engine │ │
│ │ ✅ Same fairness monitoring framework │ │
│ │ ✅ Same adaptive learning loop │ │
│ │ ✅ Sector-specific: Features, targets, compliance │ │
│ │ │ │
│ │ Deployment: Config-driven adaptation │
│ └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────┬────────────────────────────────────────┘
│
┌────────────────────────────▼────────────────────────────────────────┐
│ LAYER 6: MODEL & DATA INFRASTRUCTURE │
├─────────────────────────────────────────────────────────────────────┤
│ • ML Models: LightGBM (production-grade tree ensemble) │
│ • Experiment Tracking: MLflow (versioning, registry, metrics) │
│ • Database: PostgreSQL (metadata, logs, audit trail) │
│ • Cache: Redis (real-time feature store) │
│ • API: FastAPI (RESTful endpoints, <100ms response) │
│ • Deployment: Docker + Docker Compose │
│ • Monitoring: Prometheus + Grafana (optional) │
└─────────────────────────────────────────────────────────────────────┘
| Capability | CDARS (2025) | Clerk.io | Traditional RecSys | XAE-Frame |
|---|---|---|---|---|
| Cross-Domain Transfer | Yes | No | No | Yes |
| Explainability | Yes | Limited | No | NIST-Compliant |
| Adaptive Learning | Yes | Basic | No | Automated Retraining |
| Bias Detection | No | No | No | Yes |
| Knowledge Limits | No | No | No | Yes |
| Explanation Transfer | No | No | No | Yes |
| Context-Aware Explanations | No | No | No | Yes |
| Fairness Across Domains | No | No | No | Yes |
| EU AI Act Compliance | No | Not specified | No | Full Compliance |
| Business Impact Tracking | Partial | Basic analytics | No | Real-time KPI Mapping |
Key differentiators: XAE-Frame is the first framework to:
- Transfer both user preferences AND explanatory logic across domains
- Implement full NIST Four Principles compliance
- Provide automated retraining with feedback loops
- Map technical XAI metrics directly to business KPIs
Dataset: McAuley-Lab/Amazon-Reviews-2023
- Size: 571.54M reviews (May 1996 - Sep 2023)
- Use Cases: Product recommendation, review sentiment analysis, churn prediction
Industry Application: Credit Risk Modeling, Fraud Detection, Scoring Systems
- Use Cases: Automated loan approval processes, real-time fraud detection algorithms, dynamic credit scoring
- Shared Components: Advanced Feature Engineering, SHAP-based Explainability Integration, Ethical and Fairness Constraints
Industry Application: Policy Recommendation, Risk and Claims Prediction
- Use Cases: Driver risk assessment, personalized policy recommendations, claims prediction
- Shared Components: Regression models, fairness metrics, drift detection
Key Insight: The same framework applies across domains because:
- User behavior patterns are universal (collaborative filtering)
- Explainability (SHAP) works for any tree-based model
- Drift detection is domain-agnostic
- Fairness is critical in all regulated industries
- scikit-learn 1.3+: Baseline models, preprocessing
- LightGBM 4.0+: Production gradient boosting
- XGBoost 2.0+: Alternative gradient boosting
- SHAP 0.43+: Model explainability
- SHAP: Feature importance and explanations
- Fairlearn 0.9+: Bias detection and mitigation
- AIF360: IBM's fairness toolkit (optional)
- Alibi-Detect 0.12+: Drift detection
- Pandas 2.0+: Data manipulation
- NumPy 1.24+: Numerical computing
- PostgreSQL: Metadata and logging
- MLflow 2.8+: Experiment tracking, model registry
- FastAPI 0.104+: High-performance API
- Pydantic 2.4+: Data validation
- Docker: Containerization
- Streamlit 1.28+ (or React): Dashboard UI
- Redis: Real-time feature store
- Apache Kafka: Event streaming for large-scale deployments
- Plotly/Dash: Advanced interactive visualizations
--
API & Interface
- FastAPI 0.104+ (REST API), Uvicorn (ASGI server)
Development & Deployment
- Docker + Docker Compose
- pytest (testing), black/flake8 (code quality)
- Git/GitHub
The framework uses the Amazon Reviews 2023 dataset from McAuley Lab at UC San Diego:
- 571.54 million customer reviews (245% larger than previous versions)
- 33 product categories spanning multiple business domains
- Time range: May 1996 to September 2023
- Rich metadata: product descriptions, prices, images, user behavior patterns
- Standard splits: Pre-defined train/validation/test for reproducibility
Selected domains for development:
- E-Commerce (Primary Demo): All_Beauty (701K reviews), Electronics (43.9M reviews)
- Finance (Adaptation): Synthetic credit card transactions + Kaggle datasets
- Insurance (Adaptation): Synthetic policy data + Kaggle safe driver prediction
Prerequisites: Python 3.9+, Git, Docker & Docker Compose (optional, recommended), PostgreSQL (or use Docker)
# Clone the repository
git clone https://github.com/nazliozgur/xae-frame.git
cd xae-frame
# Copy environment template
cp .env.example .env
# Edit .env with your settings (optional for development)
# Start all services (API, Dashboard, PostgreSQL, MLflow)
docker-compose up --build
# Access services:
# - API: http://localhost:8000
# - Dashboard: http://localhost:8501
# - MLflow UI: http://localhost:5000# Clone the repository
git clone https://github.com/nazliozgur/xae-frame.git
cd xae-frame
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Download dataset (Amazon Reviews 2023 - subset)
python scripts/download_data.py --dataset amazon_reviews --category All_Beauty --sample-size 10000
# Initialize database
python scripts/init_db.py
# Run preprocessing
python src/data/preprocess.py --config configs/e_commerce.yaml
# Train baseline model
python src/models/train.py --config configs/e_commerce.yaml
# Start API server
uvicorn src.api.main:app --reload --port 8000
# Start dashboard (new terminal)
streamlit run src/dashboard/app.pyRun Jupyter notebooks:
jupyter labxae-frame/
├── README.md # This file
├── requirements.txt # Python dependencies
├── docker-compose.yml # Multi-container setup
├── Dockerfile # Container definition
│
├── configs/ # Configuration files
│ ├── e_commerce.yaml # E-commerce domain config
│ ├── finance.yaml # Finance domain config
│ └── insurance.yaml # Insurance domain config
│
├── data/ # Data directory (gitignored)
│ ├── raw/ # Original datasets
│ ├── processed/ # Cleaned and featured data
│ └── models/ # Saved model artifacts
│
├── notebooks/ # Jupyter notebooks
│ ├── 01_eda.ipynb # Exploratory data analysis
│ ├── 02_baseline.ipynb # Baseline model experiments
│ ├── 03_xai_analysis.ipynb # SHAP analysis
│ └── 04_fairness.ipynb # Bias detection
│
├── src/ # Source code
│ ├── __init__.py
│ │
│ ├── data/ # Data processing
│ │ ├── __init__.py
│ │ ├── download.py # Dataset downloaders
│ │ ├── preprocess.py # Data cleaning
│ │ └── feature_engineering.py # Feature creation
│ │
│ ├── models/ # Model training
│ │ ├── __init__.py
│ │ ├── train.py # Training pipeline
│ │ ├── predict.py # Inference
│ │ └── evaluate.py # Model evaluation
│ │
│ ├── explainability/ # XAI engine
│ │ ├── __init__.py
│ │ ├── shap_explainer.py # SHAP implementation
│ │ ├── explanation_accuracy.py # Fidelity metrics
│ │ ├── knowledge_limits.py # Confidence & OOD
│ │ └── multi_level_explanations.py # NIST 5 categories
│ │
│ ├── adaptive/ # Adaptive learning
│ │ ├── __init__.py
│ │ ├── drift_detection.py # Alibi Detect wrapper
│ │ ├── retraining_strategy.py # Automated retraining
│ │ └── feedback_loop.py # User interaction learning
│ │
│ ├── ethical/ # Fairness & compliance
│ │ ├── __init__.py
│ │ ├── bias_detection.py # Fairlearn integration
│ │ ├── bias_mitigation.py # Mitigation strategies
│ │ └── eu_ai_act_compliance.py # Compliance mapper
│ │
│ ├── business_impact/ # Business metrics
│ │ ├── __init__.py
│ │ ├── metric_mapping.py # XAI → KPI mapping
│ │ ├── real_time_collector.py # Event logging
│ │ └── roi_calculator.py # Business value computation
│ │
│ ├── api/ # FastAPI application
│ │ ├── __init__.py
│ │ ├── main.py # API entry point
│ │ ├── routes/ # API endpoints
│ │ │ ├── predict.py
│ │ │ ├── explain.py
│ │ │ └── monitor.py
│ │ └── schemas.py # Pydantic models
│ │
│ ├── dashboard/ # Streamlit/React UI
│ │ ├── __init__.py
│ │ ├── app.py # Main dashboard
│ │ ├── views/
│ │ │ ├── executive.py # C-level view
│ │ │ ├── data_science.py # Technical view
│ │ │ ├── compliance.py # Audit view
│ │ │ └── end_user.py # Simple explanations
│ │ └── components/ # Reusable UI components
│ │
│ └── utils/ # Utility functions
│ ├── __init__.py
│ ├── config.py # Config loading
│ ├── logger.py # Logging setup
│ └── db.py # Database connections
│
├── scripts/ # Standalone scripts
│ ├── download_data.py # Data download
│ ├── init_db.py # Database setup
│ └── benchmark.py # Performance testing
│
├── tests/ # Unit and integration tests
│ ├── __init__.py
│ ├── test_explainability.py
│ ├── test_adaptive.py
│ ├── test_ethical.py
│ └── test_api.py
│
└── docs/ # Documentation
├── architecture.md # Detailed architecture
├── nist_compliance.md # NIST mapping
├── eu_ai_act.md # EU AI Act compliance
└── deployment.md # Production deployment guide
This work introduces three methodological innovations to cross-domain recommendation systems:
1. Explanation Knowledge Transfer
Traditional cross-domain systems transfer user-item interaction patterns but lose the semantic reasoning behind preferences. XAE-Frame preserves and transfers explanation patterns across domains.
Example: A user who prefers "award-winning" books (indicating quality focus) receives recommendations for "premium" financial products with explanations tied to the same underlying quality preference, even though the feature spaces are different.
2. NIST-Compliant Adaptive Explanation Generation
Rather than providing uniform SHAP-based explanations, the system implements all four NIST principles with adaptive complexity:
- Explanation: SHAP values for every prediction
- Meaningful: Role-based views (technical deep-dive vs. business summary)
- Explanation Accuracy: Fidelity metrics ensure truthfulness
- Knowledge Limits: Confidence thresholds prevent unreliable predictions
3. Cross-Domain Fairness Monitoring with Automated Retraining
First systematic approach to detect bias propagation in cross-domain transfers while maintaining model accuracy through automated retraining strategies. The system monitors whether explanation quality and recommendation fairness remain consistent across demographic groups when knowledge transfers between domains, and triggers retraining when drift is detected.
Phase 1: Foundation
- Project infrastructure and repository setup
- Dataset acquisition
- Data preprocessing pipeline
- Baseline recommendation models
- MLflow setup
Phase 2: Core Engines
- XAI engine (SHAP integration)
- Drift detection (Alibi Detect)
- Fairness monitoring (Fairlearn)
- Knowledge Limits module
Phase 3: Advanced Features
- Automated retraining strategy
- Feedback loop integration
- Business impact measurement
- EU AI Act compliance mapper
Phase 4: Cross-Domain & Integration
- Transfer learning implementation
- Explanation knowledge transfer
- Finance & insurance adaptations
- Multi-view dashboard
- REST API development
- Testing
- Docker containerization
Phase 5: Finalization
- REST API development
- Testing
- Docker containerization
- Technical documentation
- Thesis writing
| Phase 3: Integration | Business impact module, compliance mapper | | Phase 4: Dashboard | Multi-view dashboard, API endpoints | | Phase 5: Cross-Domain | Finance & insurance adaptations | | Phase 6: Finalization | Documentation, testing, thesis writing |
Target Completion: January 15, 2025
Our framework implements the NIST IR 8312 standards:
| Principle | Implementation | Validation Method |
|---|---|---|
| 1. Explanation | SHAP values for every prediction | Unit tests verify SHAP outputs exist |
| 2. Meaningful | 5 explanation types for different stakeholders | User studies (optional), role-based views |
| 3. Explanation Accuracy | Fidelity metrics (SHAP consistency, feature correlation) | Automated accuracy tracking |
| 4. Knowledge Limits | Confidence thresholds + OOD detection | Reject predictions below threshold |
- User Benefit: "You might like this product because you purchased similar items"
- Societal Acceptance: Transparent reasoning builds trust in AI decisions
- Regulatory Compliance: Audit trails for regulators (e.g., loan denials)
- System Development: SHAP analysis for debugging model behavior
- Owner Benefit: Conversion rate improvement through trustworthy recommendations
- RQ1: Can a unified framework effectively implement NIST's four principles of explainability across multiple domains?
- RQ2: How does automated drift detection and retraining impact long-term model performance compared to static models?
- RQ3: What is the measurable business impact (ROI, conversion rate) of explainability features on user trust?
- RQ4: Can bias mitigation techniques maintain model accuracy while ensuring fairness across protected groups?
- Novel integration of NIST XAI principles into production ML systems
- Empirical study on drift detection effectiveness across domains
- Framework for mapping technical XAI metrics to business KPIs
- Open-source, production-ready framework
- Deployment-ready Docker containers
- Comprehensive documentation for industry adoption
- E-commerce: 5-10% conversion rate improvement (hypothesis)
- Finance: Regulatory audit readiness + reduced compliance costs
- Insurance: Fair pricing with explainable risk assessments
Standards & Frameworks
- Phillips, P.J., et al. (2020). Four Principles of Explainable Artificial Intelligence. NIST IR 8312. Link
- European Commission (2024). EU Artificial Intelligence Act. Link
Cross-Domain Recommendation
- Petruzzelli, A., et al. (2024). Instructing and Prompting Large Language Models for Explainable Cross-Domain Recommendations. ACM RecSys '24.
- Hou, Y., et al. (2024). Bridging Language and Items for Retrieval and Recommendation. arXiv:2403.03952.
Explainable AI
- Lundberg, S.M., & Lee, S.I. (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS.
- McKinsey (2022). Why Businesses Need Explainable AI—and How to Deliver It. Link
Dataset
- McAuley Lab (2023). Amazon Reviews 2023. Link
Full bibliography available in docs/references.md
While the current architecture is future-proof through 2040, these advanced features represent promising research directions:
- Current Status: Domain-specific feature engineering (working well)
- Future Enhancement: Transformer-based cross-domain user representation
- Benefit: Zero-shot recommendations across sectors
- Challenge: Requires mature embedding models and extensive training
- Note: Framework designed to accommodate embeddings when available
- Current Status: Config-based manual adaptation (efficient)
- Future Enhancement: Semantic feature mapping via NLP
- Benefit: Explanation logic transfer between domains
- Challenge: Novel research area, limited prior work
- Note: Sector Adapters provide 95% of value with 10% of complexity
E-Commerce Product recommendations with transparent reasoning, real-time adaptation to browsing patterns, and personalized explanation delivery based on user sophistication.
Financial Services Credit product recommendations with regulatory-compliant explanations, bias-free scoring models, and cross-sell opportunities based on behavior patterns from other domains.
Insurance Policy recommendations with risk factor explanations, fair pricing across demographic groups, and coverage suggestions informed by purchasing behavior in adjacent domains.
This is a Master's thesis project at Istanbul University. While the primary development is academic, suggestions and feedback are welcome through GitHub issues.
For partnership or licensing inquiries, please reach out via email.
This project is licensed under the MIT License. See the LICENSE file for details.
Nazlı Özgür
🎓 M.Sc. Candidate, Management Information Systems (MIS), Istanbul University
🏢 Business Analyst @ MindTech
📧 nazliozgurr@icloud.com
🔗 LinkedIn | GitHub
This project constitutes the final thesis for the Master of Science program in Management Information Systems at Istanbul University.
- NIST for the foundational XAI principles framework
- McKinsey QuantumBlack for industry insights on XAI business value
- McAuley Lab (UC San Diego) for the Amazon Reviews 2023 dataset
- MindTech for real-world problem context from fintech, insurance, and telecom sectors
- Istanbul University for academic support
⭐ If you find this project useful, please consider starring the repository!
Last Updated: December 2025