Skip to content

Commit ed84e30

Browse files
docg1701claude
docg1701
and
claude
authored
Story 1.3: Data Tier Deployment (PostgreSQL + Redis) (#37)
* Story 1.3: Data Tier Deployment - PO validation and corrections Created and validated Story 1.3 for deploying PostgreSQL 18 with pgvector and Redis 8.2 on dedicated data tier (VPS 2). Story includes: - Complete acceptance criteria for data tier deployment - 9 detailed tasks with granular subtasks - Integration verification criteria (IV1-IV3) - Comprehensive Dev Notes with architecture references - Testing standards and verification procedures PO Validation corrections applied (v1.0 → v1.2): CRITICAL fixes: - Added Task 5: Docker Swarm Secrets and Configs preparation - 7 secrets creation (postgres_password, 5 db passwords, redis_password) - 4 configs creation with versioning pattern (_v1) - Added "Deployment Workflow" section with scp transfer instructions - Clarified all commands execute from VPS 1 manager node SHOULD-FIX improvements: - Task 2.5.1: Specific postgresql.conf adaptation steps for 4GB RAM - Task 6.5: Rollback procedure for failed deployments - Important Notes #7: pgvector for Chatwoot documentation clarification - Important Notes #8: Resource monitoring guidance Enhancements: - Health check timing guidance (30-60 seconds) - Cross-references between sections - Updated task numbering after Task 5 insertion Story status: APPROVED - Ready for implementation Implementation Readiness Score: 10/10 Confidence Level: HIGH 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Atualizar Evolution API para v2.3.6 (evoapicloud/evolution-api) Atualização da imagem do container Evolution API no PRD e toda documentação relacionada à Story 1.7. Mudanças: - Evolution API: atendai/evolution-api:v2.2.3 → evoapicloud/evolution-api:v2.3.6 - Atualizado em 7 arquivos de documentação (PRD + shards + architecture docs) - Total de 9 ocorrências atualizadas Arquivos alterados: - docs/prd.md - docs/prd/technical-constraints-and-integration-requirements.md - docs/architecture.md (3 ocorrências) - docs/architecture/service-architecture-deep-dive.md - docs/architecture/target-architecture-to-be-docker-swarm-cluster.md - docs/architecture/high-level-architecture-current-state-as-is.md - docs/services.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Story 1.3: Data Tier Deployment - Development Complete Implementa configuração completa para deployment de PostgreSQL 18 + Redis 8.2 no VPS 2 (data tier) usando Docker Swarm. Arquivos Criados: - stacks/stack-worker-data.yml: Stack Swarm com PostgreSQL e Redis - scripts/deploy-data-tier.sh: Script automático de deployment - tests/deployment/verify-data-tier-swarm.sh: 18 testes de verificação Arquivos Modificados: - config/postgresql/init-databases.sh: Suporte a Docker Swarm secrets - config/postgresql/postgresql.conf: Adaptado para perfil Minimal (4GB RAM) * shared_buffers: 8GB → 1GB * effective_cache_size: 24GB → 2.5GB * max_connections: 200 → 100 * work_mem: 20MB → 64MB Configurações: - PostgreSQL 18 com pgvector (image: pgvector/pgvector:pg18) - Redis 8.2 com persistência RDB + AOF (image: redis:8.2-alpine) - 5 databases isolados: n8n, chatwoot, metabase, directus, evolution - pgvector habilitado em: n8n_db, chatwoot_db, directus_db - 7 Docker Swarm secrets (senhas) - 4 Docker Swarm configs (arquivos de configuração) - Sem portas expostas (rede interna apenas) - Health checks configurados - Resource limits baseados em perfil Minimal Segurança: - Todas as senhas via Docker Swarm secrets (criptografadas) - PostgreSQL: scram-sha-256 password encryption - Redis: protected-mode com autenticação por senha - Script de deployment gera senhas aleatórias seguras Próximos Passos: - Tasks 5-8 requerem execução no VPS 1 (scripts prontos) - Deployment: bash scripts/deploy-data-tier.sh - Verificação: bash tests/deployment/verify-data-tier-swarm.sh Status: Ready for Review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix: PostgreSQL 18+ volume mount path PostgreSQL 18+ requer volume em /var/lib/postgresql ao invés de /var/lib/postgresql/data conforme documentação oficial. Ref: docker-library/postgres#1259 * Fix: PostgreSQL config syntax - use integer MB values PostgreSQL não aceita valores decimais com unidades (ex: 2.5GB). Corrigido: effective_cache_size = 2.5GB → 2560MB Erro: 'syntax error near token GB' na linha 35 * Fix: Corrigir script de verificação para Docker Swarm O script estava tentando docker ps no VPS 1, mas containers estão no VPS 2. Correções: - Test 4: SSH para data node para obter container PostgreSQL - Test 5-10: Executar comandos PostgreSQL via SSH no data node - Test 15-17: SSH para data node para Redis - Test 16: Obter senha do secret e testar autenticação Redis - Corrigir aspas em comandos SQL dentro de SSH (usar aspas simples) Agora o script executa corretamente em ambiente Swarm distribuído. * Fix: Criar scripts de verificação corretos (sem SSH entre nodes) PROBLEMA: Script anterior tentava SSH de VPS1 para VPS2 - gambiarra ridícula. SOLUÇÃO CORRETA: - verify-data-tier-manager.sh: Roda no VPS 1 (manager) * Verifica via comandos Swarm (service ls, service ps, inspect) * 8 testes: service exists, placement, state, ports - verify-data-tier-local.sh: Roda no VPS 2 (data node) * Verifica containers localmente (docker exec direto) * 12 testes: connectivity, databases, pgvector, config, Redis auth, persistence Removido: - verify-data-tier-swarm.sh (script quebrado com SSH) Uso correto: 1. VPS 1: ./tests/deployment/verify-data-tier-manager.sh 2. VPS 2: ./tests/deployment/verify-data-tier-local.sh * Fix: Remover set -e dos scripts de verificação set -e fazia script parar após primeiro teste quando incrementava variável. Removido para permitir execução completa dos testes. * Story 1.3: Final story update - deployment complete Atualizou story file com resultados finais do deployment: - Status: Ready for Review → Complete - Testes: 20/20 passed (8 manager + 12 local) - Scripts de verificação que funcionaram: verify-data-tier-manager.sh + verify-data-tier-local.sh - Correções aplicadas: volume path PG18, syntax config, remoção SSH, fix set -e Mudanças documentadas: 1. PostgreSQL 18 volume: /var/lib/postgresql/data → /var/lib/postgresql 2. Config syntax: effective_cache_size 2.5GB → 2560MB 3. Abordagem verificação: 2 scripts separados (sem SSH entre nodes) 4. Script bug: Removido set -e que causava exit no primeiro teste Deployment verificado e completo em VPS 1 e VPS 2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Mark Story 1.3 as Done: Data Tier Deployment (PostgreSQL + Redis) Finalizing story branch with all implementation and documentation updates. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d9e1c57 commit ed84e30

14 files changed

+1471
-26
lines changed

‎config/postgresql/init-databases.sh‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,42 @@ GREEN='\033[0;32m'
2424
BLUE='\033[0;34m'
2525
NC='\033[0m' # No Color
2626

27+
# ===========================================================================
28+
# Docker Swarm Secrets Support
29+
# ===========================================================================
30+
# Read passwords from secret files if *_FILE environment variables are set
31+
# This pattern matches official PostgreSQL Docker image behavior
32+
# ===========================================================================
33+
34+
# Helper function to read secret from file or use environment variable directly
35+
file_env() {
36+
local var="$1"
37+
local fileVar="${var}_FILE"
38+
local def="${2:-}"
39+
40+
if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
41+
echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
42+
exit 1
43+
fi
44+
45+
local val="$def"
46+
if [ "${!var:-}" ]; then
47+
val="${!var}"
48+
elif [ "${!fileVar:-}" ]; then
49+
val="$(< "${!fileVar}")"
50+
fi
51+
52+
export "$var"="$val"
53+
unset "$fileVar"
54+
}
55+
56+
# Load passwords from secret files or environment variables
57+
file_env 'N8N_DB_PASSWORD'
58+
file_env 'CHATWOOT_DB_PASSWORD'
59+
file_env 'DIRECTUS_DB_PASSWORD'
60+
file_env 'EVOLUTION_DB_PASSWORD'
61+
file_env 'METABASE_DB_PASSWORD'
62+
2763
echo -e "${BLUE}[BorgStack]${NC} Initializing databases and users..."
2864

2965
# ===========================================================================

‎config/postgresql/postgresql.conf‎

Lines changed: 23 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# BorgStack - PostgreSQL Performance Configuration
33
# ============================================================================
44
#
5-
# Target Environment: 36GB RAM dedicated server
5+
# Target Environment: 4GB RAM VPS (Minimal Profile)
66
# PostgreSQL Version: 18.0
77
# Workload Type: Mixed OLTP (n8n, Chatwoot, Directus, Evolution API)
88
#
@@ -14,7 +14,7 @@
1414
#
1515
# Configuration generated using PostgreSQL best practices:
1616
# - shared_buffers: ~25% of RAM (PostgreSQL recommendation)
17-
# - effective_cache_size: ~66% of RAM (OS file cache estimate)
17+
# - effective_cache_size: ~62.5% of RAM (OS file cache estimate)
1818
# - work_mem: Calculated for max_connections to avoid OOM
1919
# ============================================================================
2020

@@ -24,22 +24,22 @@
2424

2525
# shared_buffers: Amount of memory dedicated to PostgreSQL for caching data
2626
# Rule: 25% of total RAM for dedicated database servers
27-
# Calculation: 36GB * 0.25 = 9GB (using 8GB for safety margin)
27+
# Calculation: 4GB * 0.25 = 1GB
2828
# Impact: Higher values reduce disk I/O by keeping more data in memory
29-
shared_buffers = 8GB
29+
shared_buffers = 1GB
3030

3131
# effective_cache_size: Estimate of memory available for disk caching by OS
32-
# Rule: ~66% of total RAM (includes PostgreSQL buffers + OS file cache)
33-
# Calculation: 36GB * 0.66 = 24GB
32+
# Rule: ~62.5% of total RAM (includes PostgreSQL buffers + OS file cache)
33+
# Calculation: 4GB * 0.625 = 2.5GB = 2560MB
3434
# Impact: Influences query planner decisions for index vs sequential scans
35-
effective_cache_size = 24GB
35+
effective_cache_size = 2560MB
3636

3737
# work_mem: Memory for sorting and hash operations PER operation PER connection
3838
# Rule: (Total RAM - shared_buffers) / (max_connections * 2-3)
39-
# Calculation: (36GB - 8GB) / (200 * 3) ≈ 20MB
39+
# Calculation: (4GB - 1GB) / (100 * 3) ≈ 10MB, increased to 64MB for better performance
4040
# Impact: Higher values improve sort/hash performance but risk OOM if too high
4141
# Note: Each complex query may use multiple work_mem allocations
42-
work_mem = 20MB
42+
work_mem = 64MB
4343

4444
# maintenance_work_mem: Memory for maintenance operations (VACUUM, CREATE INDEX)
4545
# Rule: Up to 2GB for large databases (higher than work_mem is safe)
@@ -68,14 +68,14 @@ hba_file = '/etc/postgresql/pg_hba.conf'
6868
password_encryption = scram-sha-256
6969

7070
# max_connections: Maximum number of concurrent database connections
71-
# Calculation: 4 services * ~50 connections each = 200 total
71+
# Calculation: 4 services * ~25 connections each = 100 total (Minimal profile)
7272
# Service allocation estimate:
73-
# - n8n: 50 connections (workflow executions)
74-
# - Chatwoot: 50 connections (web workers + sidekiq)
75-
# - Directus: 50 connections (API requests)
76-
# - Evolution API: 50 connections (WhatsApp message processing)
73+
# - n8n: 25 connections (workflow executions)
74+
# - Chatwoot: 25 connections (web workers + sidekiq)
75+
# - Directus: 25 connections (API requests)
76+
# - Evolution API: 25 connections (WhatsApp message processing)
7777
# Note: Services should implement connection pooling for efficiency
78-
max_connections = 200
78+
max_connections = 100
7979

8080
# ----------------------------------------------------------------------------
8181
# WRITE-AHEAD LOG (WAL) CONFIGURATION
@@ -152,11 +152,17 @@ default_statistics_target = 100
152152
# ----------------------------------------------------------------------------
153153
#
154154
# Summary:
155-
# ✅ Memory optimized for 36GB RAM server
156-
# ✅ Connection pooling supported (max_connections=200)
155+
# ✅ Memory optimized for 4GB RAM VPS (Minimal Profile)
156+
# ✅ Connection pooling supported (max_connections=100)
157157
# ✅ WAL configured for reliability and performance
158158
# ✅ SSD-optimized query planner settings
159159
#
160+
# Profile Settings Applied:
161+
# - shared_buffers: 1GB (25% of 4GB RAM)
162+
# - effective_cache_size: 2560MB (62.5% of 4GB RAM)
163+
# - max_connections: 100
164+
# - work_mem: 64MB
165+
#
160166
# Performance Verification:
161167
# Run these commands to verify settings are loaded:
162168
# - SELECT name, setting FROM pg_settings WHERE name = 'shared_buffers';

‎docs/architecture.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -182,7 +182,7 @@ BorgStack is currently a development/testing environment running on a single VPS
182182
| Redis | `redis:8.2-alpine` | Cache + n8n message broker | ✅ Migrate to VPS 2 |
183183
| Caddy | `caddy:2.10-alpine` | Reverse proxy + SSL | ✅ Migrate to VPS 3 |
184184
| n8n | `n8nio/n8n:stable` (1.112.6→stable) | Workflow automation (main mode) | ✅ Migrate to VPS 4 (queue mode) |
185-
| Evolution | `atendai/evolution-api:v2.2.3` | WhatsApp API gateway | ✅ Migrate to VPS 4 |
185+
| Evolution | `evoapicloud/evolution-api:v2.3.6` | WhatsApp API gateway | ✅ Migrate to VPS 4 |
186186
| Chatwoot | `chatwoot/chatwoot:v4.7.0-ce` | Customer service platform | ✅ Migrate to VPS 3 |
187187
| Metabase | `metabase/metabase:v0.56.2` | BI and analytics | ✅ Migrate to VPS 3 |
188188
| Directus | `directus/directus:11` | Headless CMS | ✅ Migrate to VPS 4 |
@@ -449,7 +449,7 @@ External Webhook Sources → Caddy → n8n (webhook receiver)
449449

450450
#### Evolution API (WhatsApp Gateway)
451451

452-
**Image:** `atendai/evolution-api:v2.2.3`
452+
**Image:** `evoapicloud/evolution-api:v2.3.6`
453453

454454
**Current Configuration:**
455455
- **Database:** PostgreSQL (`evolution_db`)
@@ -1668,7 +1668,7 @@ services:
16681668
replicas: 2 # Scalable: 1-5 via Portainer UI
16691669
16701670
evolution:
1671-
image: atendai/evolution-api:v2.2.3
1671+
image: evoapicloud/evolution-api:v2.3.6
16721672
placement:
16731673
constraints:
16741674
- node.labels.processing == true

‎docs/architecture/high-level-architecture-current-state-as-is.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@
5252
| Redis | `redis:8.2-alpine` | Cache + n8n message broker | ✅ Migrate to VPS 2 |
5353
| Caddy | `caddy:2.10-alpine` | Reverse proxy + SSL | ✅ Migrate to VPS 3 |
5454
| n8n | `n8nio/n8n:stable` (1.112.6→stable) | Workflow automation (main mode) | ✅ Migrate to VPS 4 (queue mode) |
55-
| Evolution | `atendai/evolution-api:v2.2.3` | WhatsApp API gateway | ✅ Migrate to VPS 4 |
55+
| Evolution | `evoapicloud/evolution-api:v2.3.6` | WhatsApp API gateway | ✅ Migrate to VPS 4 |
5656
| Chatwoot | `chatwoot/chatwoot:v4.7.0-ce` | Customer service platform | ✅ Migrate to VPS 3 |
5757
| Metabase | `metabase/metabase:v0.56.2` | BI and analytics | ✅ Migrate to VPS 3 |
5858
| Directus | `directus/directus:11` | Headless CMS | ✅ Migrate to VPS 4 |

‎docs/architecture/service-architecture-deep-dive.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ External Webhook Sources → Caddy → n8n (webhook receiver)
104104

105105
### Evolution API (WhatsApp Gateway)
106106

107-
**Image:** `atendai/evolution-api:v2.2.3`
107+
**Image:** `evoapicloud/evolution-api:v2.3.6`
108108

109109
**Current Configuration:**
110110
- **Database:** PostgreSQL (`evolution_db`)

‎docs/architecture/target-architecture-to-be-docker-swarm-cluster.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ services:
199199
replicas: 2 # Scalable: 1-5 via Portainer UI
200200

201201
evolution:
202-
image: atendai/evolution-api:v2.2.3
202+
image: evoapicloud/evolution-api:v2.3.6
203203
placement:
204204
constraints:
205205
- node.labels.processing == true

‎docs/prd.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -339,7 +339,7 @@ The new **custom dashboard** (per SIMPLE_DASHBOARD.md) will serve as the primary
339339
- Caddy: `caddy:2.10-alpine`
340340
- Chatwoot: `chatwoot/chatwoot:v4.7.0-ce`
341341
- Metabase: `metabase/metabase:v0.56.2`
342-
- Evolution API: `atendai/evolution-api:v2.2.3`
342+
- Evolution API: `evoapicloud/evolution-api:v2.3.6`
343343
- Directus: `directus/directus:11`
344344
- FileFlows: `revenz/fileflows:25.09`
345345
- SeaweedFS: `chrislusf/seaweedfs:3.97`

‎docs/prd/technical-constraints-and-integration-requirements.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@
6464
- Caddy: `caddy:2.10-alpine`
6565
- Chatwoot: `chatwoot/chatwoot:v4.7.0-ce`
6666
- Metabase: `metabase/metabase:v0.56.2`
67-
- Evolution API: `atendai/evolution-api:v2.2.3`
67+
- Evolution API: `evoapicloud/evolution-api:v2.3.6`
6868
- Directus: `directus/directus:11`
6969
- FileFlows: `revenz/fileflows:25.09`
7070
- SeaweedFS: `chrislusf/seaweedfs:3.97`

‎docs/services.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ docker compose exec redis redis-cli -a ${REDIS_PASSWORD}
229229
**Documentação Oficial:** https://doc.evolution-api.com/
230230

231231
**Configuração BorgStack:**
232-
- **Imagem:** `atendai/evolution-api:v2.2.3`
232+
- **Imagem:** `evoapicloud/evolution-api:v2.3.6`
233233
- **Função:** Multi-instâncias WhatsApp via QR code (WhatsApp Web Protocol)
234234
- **Database:** PostgreSQL (`evolution_db`)
235235
- **Autenticação:** Header `apikey: ${EVOLUTION_API_KEY}`

0 commit comments

Comments
 (0)