Supporting blog content - Local rag with lightweight Elasticsearch (#488)

Author: Delacrobix

supporting-blog-content/local-rag-with-lightweight-elasticsearch/Dataset/meeting_QA-team_wednesday.txt

@@ -0,0 +1,38 @@
+MEETING TRANSCRIPT - QA TEAM
+Date: Wednesday, September 18, 2025
+Time: 10:00 AM - 11:30 AM
+Participants: Maria (QA Lead), Tom (Senior QA Engineer), Lisa (QA Automation Engineer), Roberto (Manual Testing Specialist)
+
+[10:02] Maria: Let's review CRM migration testing progress. Tom, report on data import tests?
+
+[10:03] Tom: Found critical issues. Import failures with special characters in addresses and names.
+
+[10:06] Tom: UTF-8 parsing problems with accents, currency symbols, and Asian characters.
+
+[10:08] Tom: 12% of records affected - about 15,000 out of 125,000 total records.
+
+[10:09] Roberto: Confirmed. Also, failed imports corrupt entire batches.
+
+[10:12] Lisa: No atomic transactions for batches?
+
+[10:13] Tom: Correct. Each record processed independently without rollback.
+
+[10:15] Roberto: Found referential integrity issues - orphaned references between contacts and companies.
+
+[10:19] Maria: Need three validation types: pre-import, during import, and post-import.
+
+[10:25] Tom: Recommend smaller migration batches to reduce risk?
+
+[10:26] Maria: Excellent. Batches of 5,000 records with validation between each.
+
+[10:30] Maria: Four recommendations: UTF-8 parser fix, atomic transactions, handle orphaned references, small batch migration.
+
+[10:33] Roberto: Also need concurrency testing during migration.
+
+[10:40] Maria: Complete additional testing in one week. Feasible?
+
+[10:42] Tom: Will share test cases today.
+
+[10:44] Maria: Friday 2 PM meeting before management review.
+
+[10:45] Lisa: Will prepare testing metrics dashboard.

supporting-blog-content/local-rag-with-lightweight-elasticsearch/Dataset/meeting_development-team_monday.txt

@@ -0,0 +1,32 @@
+MEETING TRANSCRIPT - DEVELOPMENT TEAM
+Date: Monday, September 16, 2025
+Time: 09:00 AM - 10:15 AM
+Participants: Alice (Tech Lead), John (Senior Developer), Sarah (Backend Developer), Mike (DevOps Engineer)
+
+[09:02] Alice: Let's review the search API deployed last week. Any issues?
+
+[09:03] Sarah: API works but performance degrades with 1,000+ queries per minute. Response times jump from 200ms to 3 seconds.
+
+[09:05] John: Database queries and no caching layer?
+
+[09:06] Sarah: Exactly. Complex queries are slow, and we need a caching layer.
+
+[09:07] Mike: Also hitting CPU limits during spikes. Need auto-scaling.
+
+[09:08] Alice: Three priorities: query optimization, cache implementation, and infrastructure scaling.
+
+[09:11] Sarah: Propose 15-minute TTL cache with event-based invalidation.
+
+[09:13] John: I'll optimize bool queries and add calculated index fields.
+
+[09:17] Mike: Can set up auto-scaling by tomorrow - scale to 6 instances at 70% CPU.
+
+[09:18] Sarah: Starting cache implementation today, basic version by Wednesday.
+
+[09:19] John: New indexes and query optimization ready for testing Wednesday.
+
+[09:24] Alice: Clear plan. Mike handles scaling, Sarah implements cache, John optimizes queries.
+
+[09:26] Alice: I'll coordinate with product team on deployment impacts and QA for load testing.
+
+[09:30] Alice: Meeting Wednesday 3 PM to review progress. Thanks team!

supporting-blog-content/local-rag-with-lightweight-elasticsearch/Dataset/meeting_management-sync_friday.txt

@@ -0,0 +1,36 @@
+MEETING TRANSCRIPT - MANAGEMENT SYNC
+Date: Friday, September 20, 2025
+Time: 02:00 PM - 03:00 PM
+Participants: David (Project Manager), Alice (Tech Lead), Maria (QA Lead), Emma (Product Manager), Carlos (DevOps Manager)
+
+[14:03] Emma: Good progress. Users report 40% search speed improvement, but support tickets show peak hour performance issues.
+
+[14:05] Alice: We've identified bottlenecks. Working on cache layer implementation and search engine query optimization.
+
+[14:06] David: Can we resolve issues without impacting October migration date?
+
+[14:09] Alice: Recommend two-week extension for complete migration due to performance issues.
+
+[14:10] Maria: QA agrees. Found data import blockers with special characters and integrity issues.
+
+[14:12] Maria: Need one week to fix issues, another for complete re-testing.
+
+[14:14] Carlos: Infrastructure supports extension for proper rollback and disaster recovery testing.
+
+[14:15] Emma: Could we do partial migration on original date?
+
+[14:17] Alice: Yes. Contact management module first, reports and analytics in phase two.
+
+[14:21] Maria: Phased migration ideal for QA - validate each module independently.
+
+[14:22] David: Proposal: Phase 1 - Contact management October 15th. Phase 2 - Complete migration October 30th.
+
+[14:23] Alice: Reasonable timeline for performance fixes.
+
+[14:24] Emma: Works from product perspective. Will update stakeholder communications.
+
+[14:25] Maria: QA commits to these timelines.
+
+[14:26] Carlos: Will prepare deployment strategies for both phases.
+
+[14:32] David: Carlos, send deployment calendar by Monday. Thanks team!

supporting-blog-content/local-rag-with-lightweight-elasticsearch/Dataset/report_QA-team.txt

@@ -0,0 +1,31 @@
+WEEKLY REPORT - QA TEAM
+Week of September 16-20, 2025
+Prepared by: Maria Gonzalez, QA Lead
+
+=== EXECUTIVE SUMMARY ===
+QA team identified critical issues in CRM migration testing. Significant problems in legacy data import and referential integrity require immediate attention.
+
+=== TESTING COMPLETED ===
+- Functional: Contact management (100%), Authentication (100%), Search (75%), Analytics (60%)
+- Data import: 125,000 legacy records tested, 12 critical issues found
+- Performance: Core modules complete, identified issues with 500+ concurrent users
+
+=== CRITICAL ISSUES ===
+**QA-2025-001 - Data Import Failures**
+- UTF-8 parsing problems with special characters
+- 15,000 records affected (12% of total)
+- Escalated to development
+
+**QA-2025-002 - Transaction Integrity**  
+- Failed imports leave batches in inconsistent state
+- No atomic transactions for batches
+- Requires architecture redesign
+
+**QA-2025-003 - Orphaned References**
+- 2,300 records with invalid company/contact references
+- Pending business logic decision
+
+=== METRICS ===
+- Test cases executed: 847 of 1,200 (70.6%)
+- Pass rate: 79.3%, Automation coverage: 36%
+- Bugs: 28 total (4 critical, 8 high, 12 medium, 4 low)

supporting-blog-content/local-rag-with-lightweight-elasticsearch/Dataset/report_development-team.txt

@@ -0,0 +1,30 @@
+WEEKLY REPORT - DEVELOPMENT TEAM
+Week of September 16-20, 2025
+Prepared by: Alice Thompson, Tech Lead
+
+=== EXECUTIVE SUMMARY ===
+Development team completed critical infrastructure components but identified performance bottlenecks requiring attention before production deployment.
+
+=== KEY ACCOMPLISHMENTS ===
+- Database schema and indexes completed for CRM
+- 12 of 18 API endpoints integrated with authentication
+- Contact management: 95% complete, Search: 80%, Analytics: 70%
+
+=== TECHNICAL CHALLENGES ===
+- Critical: Search API degrades at 1,000+ queries/minute (200ms to 3+ seconds)
+- Root cause: Complex database queries without caching layer
+- Multi-filter searches average 1.2 seconds execution time
+
+=== ACTION PLAN NEXT WEEK ===
+1. Cache system implementation (Sarah) - Basic by Wednesday, complete by Friday
+2. Search engine query optimization (John) - Testing ready Wednesday  
+3. Auto-scaling setup (Mike) - Scale to 6 instances at 70% CPU
+
+=== METRICS ===
+- Story points: 43 of 50 completed (86%)
+- Bugs: 7 reported, 12 resolved
+- Code coverage: 78% (target: 80%)
+
+=== TIMELINE ===
+- October 15 Contact Management: 85% confidence, 2 sprints remaining
+- October 30 Complete Migration: 90% confidence, 4 sprints remaining

supporting-blog-content/local-rag-with-lightweight-elasticsearch/README.md

@@ -0,0 +1,48 @@
+# Build a Local lightweight RAG System with Elasticsearch
+
+Simple RAG (Retrieval-Augmented Generation) system using Elasticsearch for semantic search and Local AI as model provider. This application serves as supporting content for the blog post [Build a Local lightweight RAG System with Elasticsearch](https://www.elastic.co/search-labs/blog/local-rag-with-lightweight-elasticsearch)
+
+## Prerequisites
+
+- Docker
+- Python 3.11+
+
+## Quick Start
+
+### 1. Activate Virtual Environment
+
+```bash
+python -m venv venv
+source venv/bin/activate 
+```
+
+### 2. Install Python Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### 3. Configure Environment Variables
+
+Create an `.env` and put there your settings:
+
+```yaml
+# Elasticsearch Configuration
+ES_URL=http://localhost:9200
+ES_API_KEY="your_elasticsearch_api_key_here"
+INDEX_NAME=team-data
+
+# Local AI Configuration
+LOCAL_AI_URL=http://localhost:8080/v1
+
+# Dataset Configuration
+DATASET_FOLDER=./Dataset
+```
+
+### 4. Run the Script
+
+```bash
+python script.py
+```
+
+

supporting-blog-content/local-rag-with-lightweight-elasticsearch/app-logs/llama-smoltalk-3.2-1b-instruct_results.md

@@ -0,0 +1,26 @@
+🔍 Search: 'Can you summarize the performance issues in the API?'
+
+🤖 Asking to model: llama-smoltalk-3.2-1b-instruct
+
+## 💡 Question: 
+Can you summarize the performance issues in the API?
+
+## 📝 Answer: 
+Based on the documents, it appears that the main performance issue with the API is related to the search query optimization. The API degrades to around 1,000+ queries per minute (QP/min) when there are 12 of 18 API endpoints integrated with authentication. This issue is caused by complex queries without a caching layer, leading to performance degrades and slow response times.
+
+However, there is also a smaller issue with the "Search" API, where it degrades to around 3+ seconds after 1.2 seconds execution time. This is likely due to multi-filter searches and the need for a caching layer to improve performance.
+
+To address these issues, the team is working on implementing a caching layer (Sarah) and optimizing bool queries and adding calculated index fields (John) to improve query efficiency. They are also working on setting up auto-scaling for the database (Mike) to ensure that it can handle increased traffic.
+
+A meeting was held to discuss these issues and a plan for improvement was agreed upon. The team will work together to implement a caching layer and optimize the queries, and the team will work with product team to ensure that the migration is completed on time and does not impact the October migration date.
+
+📚 Citations:
+  [1] report_development-team.txt
+  [2] meeting_development-team_monday.txt
+  [3] meeting_management-sync_friday.txt
+
+
+## Stats
+🔍 Search Latency: 12ms
+
+🤖 AI Latency: 21019ms | 5.8 tokens/s

supporting-blog-content/local-rag-with-lightweight-elasticsearch/app-logs/results.md

@@ -0,0 +1,27 @@
+
+🔍 Search: 'Can you summarize the performance issues in the API?'
+
+🤖 Asking to model: dolphin3.0-qwen2.5-0.5b
+
+## 💡 Question: 
+Can you summarize the performance issues in the API?
+## 📝 Answer: 
+
+The performance issues in the API can be summarized as follows:
+
+1. **Search API degrades with 1,000+ queries per minute**: The search API has degraded, with performance degrades to a time of 3 seconds compared to the target of 200ms.
+
+2. **Complex queries are slow and no caching layer**: Complex queries take longer to execute as expected.
+
+3. **CPU limits during spikes**: The CPU usage is at 100% at the peak hour and limits to 70% at the peak hour.
+
+📚 Citations:
+  [1] report_development-team.txt
+  [2] meeting_development-team_monday.txt
+  [3] meeting_management-sync_friday.txt
+
+## Stats
+
+🔍 Search Latency: 14ms
+
+🤖 AI Latency: 16044ms | 9.5 tokens/s

supporting-blog-content/local-rag-with-lightweight-elasticsearch/app-logs/smollm2-1.7b-instruct_results.md

@@ -0,0 +1,24 @@
+🔍 Search: 'Can you summarize the performance issues in the API?'
+
+🤖 Asking to model: smollm2-1.7b-instruct
+
+## 💡 Question: 
+
+Can you summarize the performance issues in the API?
+## 📝 Answer: 
+
+The development team identified two key technical challenges for the API:
+
+1.  The search API degrades at 1,000+ queries per minute, causing average execution times to jump from 200ms to 3 seconds.
+2.  The root cause is complex database queries without a caching layer, leading to poor query performance.
+
+📚 Citations:
+  [1] report_development-team.txt
+  [2] meeting_development-team_monday.txt
+  [3] meeting_management-sync_friday.txt
+
+## Stats
+
+🔍 Search Latency: 16ms
+
+🤖 AI Latency: 47561ms | 4.8 tokens/s

supporting-blog-content/local-rag-with-lightweight-elasticsearch/app-logs/why-is-the-sky-blue.md

@@ -0,0 +1,18 @@
+>>> Why is the sky blue?
+
+## Raw Response
+
+```json
+{"created":1763994585,"object":"chat.completion","id":"d96f7e7a-003f-4167-bb07-501aa41ef8e7","model":"dolphin3.0-qwen2.5-0.5b","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"The sky being blue is primarily due to the scattering of sunlight by the Earth's atmosphere. Here are a few key factors contributing to the blue color of the sky:\n\n1. **Scattering of Light**: Blue light is scattered more than other colors by the Earth's atmosphere. This scattering occurs because blue light has a shorter wavelength, which is more likely to travel through the Earth's atmosphere than other wavelengths. The scattering occurs as the light bounces off particles in the atmosphere, such as dust and water vapor, which have a higher concentration in the upper atmosphere.\n\n2. **Atmospheric Composition**: The Earth's atmosphere is composed of gases like nitrogen, oxygen, and water vapor. These gases have different properties, and their interactions with sunlight can affect the colors they transmit. Nitrogen and oxygen gases have a scattering effect that is more intense at shorter wavelengths than water vapor, which results in the blue color observed in the sky.\n\n3. **Atmospheric Reflection**: The blue color of the sky also depends on the amount of light reflecting off the Earth's surface. Clouds, for instance, can reflect a lot of sunlight, particularly blue and green light, which creates a blue tint to the sky. Additionally, the reflection of sunlight from the Earth's surface can contribute to the blue color observed in the sky.\n\n4. **Clouds and Precipitation**: The presence of clouds and precipitation can also affect the color of the sky. Clouds can scatter blue light, but they can also trap it, making the sky appear a bit less blue. Precipitation, particularly heavy rain or snow, can also scatter light, making the sky appear a bit darker and more gray.\n\nIn summary, the blue color of the sky is a result of the scattering of sunlight by the Earth's atmosphere, with the primary contributor being the scattering of blue light and the absorption of other wavelengths by particles in the atmosphere."}}],"usage":{"prompt_tokens":14,"completion_tokens":384,"total_tokens":398}}
+```
+
+## Answer
+
+The sky being blue is primarily due to the scattering of sunlight by the Earth's atmosphere. Here are a few key factors contributing to the blue color of the sky:
+
+1. **Scattering of Light**: Blue light is scattered more than other colors by the Earth's atmosphere. This scattering occurs because blue light has a shorter wavelength, which is more likely to travel through the Earth's atmosphere than other wavelengths. The scattering occurs as the light bounces off particles in the atmosphere, such as dust and water vapor, which have a higher concentration in the upper atmosphere.
+2. **Atmospheric Composition**: The Earth's atmosphere is composed of gases like nitrogen, oxygen, and water vapor. These gases have different properties, and their interactions with sunlight can affect the colors they transmit. Nitrogen and oxygen gases have a scattering effect that is more intense at shorter wavelengths than water vapor, which results in the blue color observed in the sky.
+3. **Atmospheric Reflection**: The blue color of the sky also depends on the amount of light reflecting off the Earth's surface. Clouds, for instance, can reflect a lot of sunlight, particularly blue and green light, which creates a blue tint to the sky. Additionally, the reflection of sunlight from the Earth's surface can contribute to the blue color observed in the sky.
+4. **Clouds and Precipitation**: The presence of clouds and precipitation can also affect the color of the sky. Clouds can scatter blue light, but they can also trap it, making the sky appear a bit less blue. Precipitation, particularly heavy rain or snow, can also scatter light, making the sky appear a bit darker and more gray.
+
+In summary, the blue color of the sky is a result of the scattering of sunlight by the Earth's atmosphere, with the primary contributor being the scattering of blue light and the absorption of other wavelengths by particles in the atmosphere.