Merge pull request #1 from redis-applied-ai/terraform

terraform cleanup

Author: rbs333

.gitignore

@@ -192,5 +192,4 @@ ai
 TASK_MEMORY.md
 .env*
 
-terraform/
 infra/

CLAUDE.md

@@ -17,7 +17,6 @@ Important: DO NOT USE Redis Stack or other earlier versions of Redis.
 - `tasks.py`: Task definitions with retry logic
 - `rag.py`: RedisVL vector search + OpenAI generation
 - `web_search.py`: Tavily API integration
-- `glean_search.py`: Enterprise knowledge search
 - `db.py`: Vector index management
 
 ### Pipeline System (`pipelines/`)

Dockerfile.collector

@@ -5,7 +5,7 @@
 ![Language](https://img.shields.io/github/languages/top/redis-applied-ai/redis-slack-worker-agent)
 ![GitHub last commit](https://img.shields.io/github/last-commit/redis-applied-ai/redis-slack-worker-agent)
 
-The code in this repo shows a reference architecture for a Slack-integrated agent application running on ECS. The architecture is designed so that agent workers can scale horizontally to demand while keeping a minimal API instance running. Additionally, it implements the [agent memory server](https://github.com/redis/agent-memory-server) as a tool call for storing and automatically summarizing and persisting short and long term memory to Redis.
+The code in this repo shows a reference architecture for a Slack-integrated agent application running on ECS. The architecture is designed so that agent workers, API containers, and agent-memory-server can scale in a decoupled way. [The agent memory server](https://github.com/redis/agent-memory-server) itself is implemented as a tool call for storing and automatically summarizing and persisting short and long term memories.
 
 ## Business objective
 
@@ -16,22 +16,22 @@ Internally at Redis, this bot extends the Applied AI engineering team by assisti
 
 ![Application Architecture](resources/haink_task_flow.png)
 
-**TLDR Flow**: Slack → FastAPI webhook → Redis task queue → Agent workers pick up and execute tasks → Agent workers perform tool calls and schedule tasks until determining to respond and invoke Slack callback.
+**TLDR Flow**: Slack → FastAPI webhook → Redis task queue → Agent workers pick up and execute tasks → Agent workers perform tool calls and ReAct style execution -> invoke Slack callbacks async.
 
 ## Core Components
 
-- **FastAPI App**: Webhook handler with health checks
-- **Agent Engine**: ReAct methodology that runs search tools (curated AI team knowledge, internal Glean search, and web search via Tavily) in an agentic loop
-- **Agent memory**: Remembers past interactions with users via the [Agent Memory Server](https://github.com/redis/agent-memory-server) and personalizes responses
-- **Docket Workers**: Background task processing with retry logic
-- **Redis**: Vector database (RedisVL) + streams-based task queue + caching
+- **FastAPI App**: Webhook handler.
+- **Agent Engine**: ReAct methodology runs search tools in an agentic loop.
+- **Agent memory**: Remembers past interactions with users via the [Agent Memory Server](https://github.com/redis/agent-memory-server).
+- **Docket Workers**: Core task processing utility.
+- **Redis**: Vector database + streams-based task queue + memory store.
 
 ## Quick Start
 
 ### Prerequisites
 - Python 3.12+, Redis, UV package manager
 - Slack app credentials, OpenAI API key
-- Optional: Tavily API key for web search
+- Tavily API key for web search
 
 ### Development Setup
 ```bash
@@ -54,27 +54,27 @@ uv run python -m app.worker
 # Start API (Terminal 2)
 uv run fastapi dev app/api/main.py
 
-# Local otel collector (Optional Terminal 3)
-docker compose -f docker-compose.collector.yml up
 ```
 
-### Local development with Slack
+### Development with Slack
 
-To test local changes in Slack, you can run an ngrok server locally and then connect Slack to the ngrok endpoint that proxies to your local machine.
+To test changes in Slack, you can run an ngrok server locally or setup with ALB with terraform (see below) and then connect Slack to the respective endpoint.
 
-First, run ngrok:
+For ngrok, run:
 
 ```bash
 ngrok http 8000
 ```
 
 Then, in the Slack API console, update "Event Subscriptions -> Request URL" with the proxy URL ngrok gives you. It will look like `https://<ngrok proxy>/slack/events` (e.g.: `https://3cfaf9a1bcff.ngrok-free.app/slack/events`).
 
+![Application Architecture](resources/slack_subscription.png)
+
 Additionally, if persisting answer feedback locally, update "Interactivity & Shortcuts -> Request URL" with the URL `https://<ngrok proxy>/slack/interactive`.
 
 ## Usage
 
-**Slack**: Mention `@bot` in any channel. The bot processes questions using ReAct methodology with search tools (curated AI knowledge, internal docs with Glean, and web search).
+**Slack**: Mention `@bot` in any channel. The bot processes questions using ReAct methodology with search tools (curated AI knowledge and web search).
 
 **API**:
 - Health: `GET /health`
@@ -89,12 +89,139 @@ Essential environment variables:
 SLACK_BOT_TOKEN=xoxb-your-bot-token
 SLACK_SIGNING_SECRET=your-signing-secret
 OPENAI_API_KEY=your-openai-key
-
-# Optional
 TAVILY_API_KEY=your-tavily-key  # Web search tool
 REDIS_URL=redis://localhost:6379/0
 ```
 
+## Deployment (AWS, single environment)
+
+This reference deploys a working agent stack on AWS with a single `terraform apply`:
+VPC, ALB, ECS Fargate (API, Worker, Memory Server), ECR, S3, IAM, and basic CloudWatch.
+No domain or SSL is required; the ALB exposes HTTP for testing.
+
+Development environment requirements
+- Terraform v1.10 or later
+- AWS CLI configured with credentials
+- Docker (to build and push images)
+- Agent Memory Server [credentials](https://redis.github.io/agent-memory-server/authentication/?h=secret#token-management-commands)
+
+Prerequisites
+- AWS account with permissions for VPC, ECS (Fargate), ECR, ALB, IAM, S3, CloudWatch
+- Slack app credentials (bot token, signing secret)
+- OpenAI API key
+- Cloud-accessible Redis URL (e.g., Upstash or ElastiCache). Localhost will not work from ECS.
+- Tavily API key (web search)
+
+Terraform modules
+- VPC: networking, subnets, security groups
+- ECR: container repositories
+- ECS: Fargate cluster, API and worker services, Memory Server sidecar (separate service)
+- ALB: public Load Balancer routing to API and Memory Server health
+- S3: content bucket for uploads/examples
+- IAM: roles and policies for ECS/ECR/S3/SSM
+- Monitoring: CloudWatch dashboard and alarms (basic)
+
+Step 1) Configure minimal variables
+Create or edit `terraform/terraform.tfvars` (example):
+```hcl
+aws_region   = "us-east-1"
+project_name = "my-ai-agent"           # choose your own unique name
+bucket_name  = "my-ai-agent-content-1234"  # must be globally unique
+```
+
+Step 2) Seed required secrets into AWS SSM
+- Copy the example and fill in values: `cp .env.example .env && edit .env`
+- Then run the loader script to write SSM parameters under `/${project_name}/...`
+- IMPORTANT: Use a cloud Redis URL for `REDIS_URL` (not localhost)
+```bash
+set -a; source .env; \
+export PROJECT_NAME="my-ai-agent" AWS_REGION="us-east-1"; \
+# Required for cloud: set a token and the base URL the app will call
+export AGENT_MEMORY_SERVER_API_KEY="generate-a-strong-token"; \
+# Get ALB DNS dynamically (ALB routes /v1/* to memory server):
+export AGENT_MEMORY_SERVER_URL="http://$(terraform -chdir=terraform output -raw alb_dns_name)"; \
+set +a; sh ./scripts/load_secrets.sh
+```
+See the full list of parameters in `terraform/SSM_PARAMETERS.md`.
+
+Step 3) Deploy the infrastructure
+Use the helper script or terraform directly:
+```bash
+# Using helper script
+./terraform/deploy.sh apply
+
+# Or manually
+terraform -chdir=terraform init
+terraform -chdir=terraform validate
+terraform -chdir=terraform plan -out=tfplan
+terraform -chdir=terraform apply tfplan
+```
+After apply, note the `application_url` output (ALB HTTP URL).
+
+Step 4) Build and push images to ECR
+
+**CRITICAL**: ECS Fargate runs on X86_64 (amd64) architecture. Always build with `--platform linux/amd64` to avoid "exec format error" failures.
+
+```bash
+ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
+AWS_REGION=us-east-1
+ECR="$ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"
+aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin "$ECR"
+
+# API (build for linux/amd64)
+docker build --platform linux/amd64 -f Dockerfile.api -t "$ECR/my-ai-agent-api:latest" .
+docker push "$ECR/my-ai-agent-api:latest"
+
+# Worker (build for linux/amd64)
+docker build --platform linux/amd64 -f Dockerfile.worker -t "$ECR/my-ai-agent-worker:latest" .
+docker push "$ECR/my-ai-agent-worker:latest"
+```
+
+**Note**: If you encounter "No space left on device" during `uv sync`, increase Docker Desktop disk allocation (Settings → Resources → Disk image size).
+
+**Optional - Test locally before pushing**:
+```bash
+# Verify architecture
+docker image inspect "$ECR/my-ai-agent-api:latest" --format '{{.Architecture}}'  # should show "amd64"
+
+# Quick smoke test (requires .env file)
+docker run --rm --env-file .env -p 3000:3000 "$ECR/my-ai-agent-api:latest" &
+sleep 5 && curl -sf http://localhost:3000/health && echo "✓ API health OK"
+docker stop $(docker ps -q --filter ancestor="$ECR/my-ai-agent-api:latest")
+
+# Verify git binary is present (prevents GitPython errors)
+docker run --rm "$ECR/my-ai-agent-worker:latest" git --version
+docker run --rm "$ECR/my-ai-agent-worker:latest" python -c "import git; print('GitPython OK')"
+```
+
+Step 5) Force ECS services to deploy latest images
+```bash
+aws ecs update-service --cluster my-ai-agent-cluster \
+  --service my-ai-agent-api-service --force-new-deployment
+aws ecs update-service --cluster my-ai-agent-cluster \
+  --service my-ai-agent-worker-service --force-new-deployment
+```
+
+Step 6) Verify health
+```bash
+APP_URL=$(terraform -chdir=terraform output -raw application_url)
+curl -i "$APP_URL/health"
+curl -i "$APP_URL/v1/health"
+```
+
+Step 7) Configure Slack and test
+- Event Subscriptions → Enable → Request URL: `<application_url>/slack/events`
+- Interactivity & Shortcuts → Enable → Request URL: `<application_url>/slack/interactive`
+- Subscribe to bot events (at minimum): `app_mention`, `message.channels`, `message.im`
+Test by mentioning the bot in a channel or DM.
+
+> See Development with Slack section for visual on how to update this value.
+
+Cleanup
+```bash
+terraform -chdir=terraform destroy
+```
+
 ## Testing
 
 ```bash

README.md

@@ -19,11 +19,8 @@
 
 # Re-export all tools
 from .tools import (
-    GleanSearchService,
-    get_glean_service,
     get_search_tool_config,
     perform_web_search,
-    search_glean,
     search_knowledge_base,
     search_knowledge_base_with_metadata,
 )
@@ -38,11 +35,8 @@
     "is_brief_satisfied_response",
     "SYSTEM_PROMPT",
     # Tool functions
-    "GleanSearchService",
-    "get_glean_service",
     "get_search_tool_config",
     "perform_web_search",
-    "search_glean",
     "search_knowledge_base",
     "search_knowledge_base_with_metadata",
     # All task functions are available via the wildcard import

app/agent/__init__.py

@@ -19,7 +19,7 @@
 from .tasks import *
 
 # Re-export all tools
-from .tools import get_glean_search_tool, get_search_tool, get_web_search_tool
+from .tools import get_search_knowledge_base_tool, get_web_search_tool
 
 # For backward compatibility, also export the main functions directly
 __all__ = [
@@ -31,8 +31,7 @@
     "is_brief_satisfied_response",
     "retrieve_context",
     # Tool functions
-    "get_search_tool",
+    "get_search_knowledge_base_tool",
     "get_web_search_tool",
-    "get_glean_search_tool",
     # All task functions are available via the wildcard import
 ]

app/agent/agent.py

@@ -19,11 +19,7 @@
 from redisvl.index.index import AsyncSearchIndex
 from redisvl.utils.vectorize import OpenAITextVectorizer
 
-from app.agent.tools import (
-    get_glean_search_tool,
-    get_search_knowledge_base_tool,
-    get_web_search_tool,
-)
+from app.agent.tools import get_search_knowledge_base_tool, get_web_search_tool
 from app.utilities.openai_client import get_instrumented_client
 
 logger = logging.getLogger(__name__)
@@ -334,7 +330,6 @@ async def answer_question(
     tools = [
         get_search_knowledge_base_tool(),
         get_web_search_tool(),
-        get_glean_search_tool(),
         *MemoryAPIClient.get_all_memory_tool_schemas(),
     ]
 

app/agent/core.py

@@ -4,12 +4,6 @@
 This package contains all tool modules for the agent.
 """
 
-from .glean_search import (
-    GleanSearchService,
-    get_glean_search_tool,
-    get_glean_service,
-    search_glean,
-)
 from .search_knowledge_base import (
     get_search_knowledge_base_tool,
     get_search_tool_config,
@@ -19,10 +13,6 @@
 from .web_search import get_web_search_tool, perform_web_search
 
 __all__ = [
-    "GleanSearchService",
-    "get_glean_search_tool",
-    "get_glean_service",
-    "search_glean",
     "get_search_knowledge_base_tool",
     "get_search_tool_config",
     "search_knowledge_base",