Memory System Setup Guide

This guide will help you set up the Clear-AI Memory System with Neo4j, Pinecone, and Ollama for intelligent memory management.

Prerequisites

Before setting up the memory system, you'll need:

Node.js (v18 or higher)
Neo4j (Desktop or Server)
Ollama (for local embeddings)
Pinecone (optional, for semantic memory)

Quick Start

1. Install Dependencies

# Install Clear-AI packages
yarn install

# Build shared package
yarn build:shared

2. Set up Neo4j

Option A: Neo4j Desktop (Recommended)

Download Neo4j Desktop
- Go to https://neo4j.com/download/
- Download and install Neo4j Desktop
Create a Project
- Open Neo4j Desktop
- Click "New Project"
- Name it "Clear-AI Memory"
Add a Database
- Click "Add Database" → "Local DBMS"
- Set password (remember this!)
- Click "Create"
Start the Database
- Click the "Start" button on your database
- Wait for it to show "Active"

Option B: Neo4j Server

# Using Docker
docker run \
  --name neo4j-memory \
  -p 7474:7474 -p 7687:7687 \
  -d \
  -v $HOME/neo4j/data:/data \
  -v $HOME/neo4j/logs:/logs \
  -v $HOME/neo4j/import:/var/lib/neo4j/import \
  -v $HOME/neo4j/plugins:/plugins \
  --env NEO4J_AUTH=neo4j/password \
  neo4j:latest

3. Set up Ollama

Install Ollama

# macOS/Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
# Download from https://ollama.ai/download

Start Ollama
```
ollama serve
```
Pull the Embedding Model
```
ollama pull nomic-embed-text
```

4. Set up Pinecone (Optional)

Create Pinecone Account
- Go to https://www.pinecone.io/
- Sign up for a free account
Create a Project
- Create a new project
- Note your API key and environment
Create an Index
- Create an index named clear-ai-memories
- Set dimensions to 768 (for nomic-embed-text)
- Use cosine similarity metric

5. Configure Environment

Create a .env file in packages/server/:

# Neo4j Configuration
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=your_password_here
NEO4J_DATABASE=neo4j

# Pinecone Configuration (optional)
PINECONE_API_KEY=your_pinecone_api_key
PINECONE_ENVIRONMENT=your_pinecone_environment
PINECONE_INDEX_NAME=clear-ai-memories

# Memory System Settings
MEMORY_EMBEDDING_MODEL=nomic-embed-text
MEMORY_EMBEDDING_DIMENSIONS=768
MEMORY_MAX_CONTEXT_MEMORIES=50
MEMORY_SIMILARITY_THRESHOLD=0.7
OLLAMA_BASE_URL=http://localhost:11434

# LangChain Configuration (optional)
OPENAI_API_KEY=your_openai_api_key
OPENAI_MODEL=gpt-3.5-turbo

6. Test the Setup

# Test Ollama embeddings
node test-ollama-embeddings.js

# Test Neo4j connection
node test-neo4j-connection.js

# Test memory system
node test-neo4j-only.js

# Test full system (if Pinecone is configured)
node test-memory-system.js

7. Start the Test Server

# Start memory test server
node test-memory-server.js

The server will start on port 3006 and provide test endpoints.

Detailed Setup

Neo4j Configuration

Database Setup

Access Neo4j Browser
- Open http://localhost:7474
- Login with username neo4j and your password

Create Initial Schema (Optional)

// Create indexes for better performance
CREATE INDEX user_id_index IF NOT EXISTS FOR (u:User) ON (u.id);
CREATE INDEX memory_timestamp_index IF NOT EXISTS FOR (m:EpisodicMemory) ON (m.timestamp);
CREATE INDEX memory_user_index IF NOT EXISTS FOR (m:EpisodicMemory) ON (m.userId);

Verify Connection

// Test query
RETURN "Neo4j is working!" as message;

Performance Tuning

// Set memory limits
CALL dbms.listConfig() YIELD name, value
WHERE name CONTAINS 'memory'
RETURN name, value;

// Adjust heap size if needed
// Add to neo4j.conf:
// dbms.memory.heap.initial_size=512m
// dbms.memory.heap.max_size=2G

Ollama Configuration

Model Management

# List available models
ollama list

# Pull additional models
ollama pull llama2
ollama pull codellama

# Remove unused models
ollama rm model-name

Performance Optimization

# Set GPU acceleration (if available)
export OLLAMA_GPU=1

# Set memory limits
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_MAX_QUEUE=512

Pinecone Configuration

Index Settings

# Python example for index creation
import pinecone

pinecone.init(api_key="your-api-key", environment="your-environment")

# Create index
pinecone.create_index(
    name="clear-ai-memories",
    dimension=768,
    metric="cosine",
    spec={
        "serverless": {
            "cloud": "aws",
            "region": "us-east-1"
        }
    }
)

Monitoring

# Check index status
index = pinecone.Index("clear-ai-memories")
print(index.describe_index_stats())

Testing and Validation

Test Scripts

The following test scripts are available:

1. Ollama Embeddings Test

node test-ollama-embeddings.js

Expected Output:

🧪 Testing Ollama embeddings...
✅ Ollama embeddings working!
📊 Embedding dimension: 768
🔢 First 5 values: [0.2762, 1.6010, -3.6177, -1.7872, 0.7215...]
🔗 Similarity between texts: 0.4876
✅ Good! Different texts have low similarity

2. Neo4j Connection Test

node test-neo4j-connection.js

Expected Output:

🔍 Testing Neo4j connection...
Trying bolt://localhost:7687...
✅ Connected to Neo4j at bolt://localhost:7687
✅ Query test successful: 1

🎉 Neo4j is working! Use these settings in your .env file:
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=your_password
NEO4J_DATABASE=neo4j

3. Memory System Test

node test-neo4j-only.js

Expected Output:

🧪 Testing Neo4j memory system...
Neo4j memory service initialized successfully
✅ Neo4j memory service initialized
✅ Stored memory: 57aa123b-6194-4745-8fe6-1ab111a8d2a1
✅ Retrieved memory: User asked about machine learning algorithms
✅ Found 1 memories
✅ Memory stats: 1 memories
✅ Cleaned up test memories
✅ Neo4j connection closed

🎉 Neo4j memory system is working perfectly!
📋 Next step: Set up Pinecone for semantic memory

4. Full System Test

node test-memory-system.js

Expected Output:

🧪 Testing complete memory system...
✅ Neo4j memory service initialized
✅ Pinecone memory service initialized
✅ Memory context service initialized
✅ Stored episodic memory: memory-123
✅ Stored semantic memory: concept-456
✅ Found 1 episodic memories
✅ Found 1 semantic memories
✅ Enhanced context with memories
✅ Memory stats: 1 episodic, 1 semantic
✅ Cleaned up test data
✅ All services closed

🎉 Complete memory system is working perfectly!

API Testing

Test Memory Storage

curl -X POST http://localhost:3006/test/episodic \
  -H "Content-Type: application/json" \
  -d '{}'

Test Memory Retrieval

curl http://localhost:3006/test/stats/test-user

Test Chat Integration

curl -X POST http://localhost:3001/api/memory-chat/chat \
  -H "Content-Type: application/json" \
  -d '{
    "userId": "user123",
    "sessionId": "session456",
    "message": "Hello, what can you remember about me?",
    "includeMemories": true
  }'

Troubleshooting

Common Issues

Neo4j Connection Failed

Error:

Failed to connect to server. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver.

Solutions:

Ensure Neo4j is running
Check the port (default: 7687)
Verify password is correct
Check firewall settings
Try different connection URI formats:
- bolt://localhost:7687
- neo4j://localhost:7687
- bolt://127.0.0.1:7687

Ollama Not Working

Error:

Failed to generate embedding with Ollama: fetch failed

Solutions:

Ensure Ollama is running: ollama serve
Check if model is available: ollama list
Pull the model: ollama pull nomic-embed-text
Check Ollama logs: ollama logs
Verify port 11434 is accessible

Pinecone Connection Error

Error:

PineconeConnectionError: Request failed to reach Pinecone

Solutions:

Verify API key is correct
Check environment name
Ensure index exists
Check network connectivity
Verify index dimensions (768)

Memory Service Not Initialized

Error:

Memory service not initialized

Solutions:

Check all environment variables
Ensure Neo4j is running
Verify Ollama is accessible
Check Pinecone credentials (if using)
Review service logs

Debug Commands

# Check Neo4j status
lsof -i :7687

# Check Ollama status
curl http://localhost:11434/api/tags

# Check Pinecone status
curl -H "Api-Key: your-api-key" \
  "https://controller.your-environment.pinecone.io/actions/whoami"

# Check memory service
node test-memory-server.js

Log Analysis

Neo4j Logs

# Check Neo4j logs
tail -f $HOME/neo4j/logs/neo4j.log

Ollama Logs

# Check Ollama logs
ollama logs

Application Logs

# Check application logs
tail -f logs/memory-service.log

Performance Optimization

Neo4j Optimization

Memory Settings

# Add to neo4j.conf
dbms.memory.heap.initial_size=1G
dbms.memory.heap.max_size=4G
dbms.memory.pagecache.size=2G

Index Optimization

// Create indexes for frequently queried fields
CREATE INDEX user_id_index FOR (u:User) ON (u.id);
CREATE INDEX memory_timestamp_index FOR (m:EpisodicMemory) ON (m.timestamp);
CREATE INDEX memory_user_index FOR (m:EpisodicMemory) ON (m.userId);

Query Optimization

// Use EXPLAIN to analyze query performance
EXPLAIN MATCH (m:EpisodicMemory {userId: "user123"}) RETURN m;

Ollama Optimization

GPU Acceleration

# Enable GPU if available
export OLLAMA_GPU=1

Memory Management

# Limit loaded models
export OLLAMA_MAX_LOADED_MODELS=2

Model Optimization

# Use quantized models for better performance
ollama pull nomic-embed-text:latest

Pinecone Optimization

Index Configuration
- Use appropriate similarity metric (cosine)
- Set correct dimensions (768)
- Choose appropriate pod type
Query Optimization
- Use appropriate topK values
- Implement query caching
- Batch operations when possible

Security Considerations

Neo4j Security

Authentication

# Enable authentication
dbms.security.auth_enabled=true

Encryption

# Enable encryption
dbms.connector.bolt.tls_level=REQUIRED

Access Control

// Create users with specific roles
CREATE USER memory_user SET PASSWORD 'secure_password';
GRANT ROLE reader TO memory_user;

API Security

Rate Limiting

// Implement rate limiting
const rateLimit = require('express-rate-limit');

const memoryLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100 // limit each IP to 100 requests per windowMs
});

app.use('/api/memory', memoryLimiter);

Input Validation

// Validate input data
const validateMemoryData = (data) => {
  if (!data.userId || !data.content) {
    throw new Error('Invalid memory data');
  }
};

Authentication Middleware

// Add authentication
const authenticateUser = (req, res, next) => {
  const token = req.headers.authorization;
  if (!token) {
    return res.status(401).json({ error: 'Unauthorized' });
  }
  // Verify token
  next();
};

Monitoring and Maintenance

Health Checks

// Health check endpoint
app.get('/api/memory/health', async (req, res) => {
  try {
    const neo4jHealth = await checkNeo4jConnection();
    const ollamaHealth = await checkOllamaConnection();
    const pineconeHealth = await checkPineconeConnection();
    
    res.json({
      status: 'healthy',
      services: {
        neo4j: neo4jHealth,
        ollama: ollamaHealth,
        pinecone: pineconeHealth
      }
    });
  } catch (error) {
    res.status(500).json({ status: 'unhealthy', error: error.message });
  }
});

Monitoring Metrics

Memory Usage
- Track episodic memory count
- Monitor semantic memory usage
- Set up alerts for limits
Performance Metrics
- Query response times
- Embedding generation time
- API response times
Error Tracking
- Log all errors
- Set up error alerts
- Monitor service availability

Maintenance Tasks

Regular Cleanup

// Clean up old memories
const cleanupOldMemories = async () => {
  const cutoffDate = new Date(Date.now() - 90 * 24 * 60 * 60 * 1000); // 90 days ago
  await memoryService.cleanupOldMemories(cutoffDate);
};

Index Maintenance

// Rebuild indexes periodically
DROP INDEX user_id_index;
CREATE INDEX user_id_index FOR (u:User) ON (u.id);

Backup Strategy

# Backup Neo4j database
neo4j-admin dump --database=neo4j --to=/backup/neo4j-backup.dump

# Backup Pinecone index (export data)
# Use Pinecone API to export vectors

Next Steps

Explore the APIs: Check out Memory System APIs
Learn Integration: See Memory Integration Examples
Build Applications: Use the Client SDK
Customize Configuration: Adapt settings for your use case
Monitor Performance: Set up monitoring and alerts

Support

If you encounter issues:

Check the logs for error messages
Run the test scripts to identify problems
Review the troubleshooting section above
Check the documentation for configuration details
Open an issue on GitHub if problems persist

Prerequisites​

Quick Start​

1. Install Dependencies​

2. Set up Neo4j​

Option A: Neo4j Desktop (Recommended)​

Option B: Neo4j Server​

3. Set up Ollama​

4. Set up Pinecone (Optional)​

5. Configure Environment​

6. Test the Setup​

7. Start the Test Server​

Detailed Setup​

Neo4j Configuration​

Database Setup​

Performance Tuning​

Ollama Configuration​

Model Management​

Performance Optimization​

Pinecone Configuration​

Index Settings​

Monitoring​

Testing and Validation​

Test Scripts​

1. Ollama Embeddings Test​

2. Neo4j Connection Test​

3. Memory System Test​

4. Full System Test​

API Testing​

Test Memory Storage​

Test Memory Retrieval​

Test Chat Integration​

Troubleshooting​

Common Issues​

Neo4j Connection Failed​

Ollama Not Working​

Pinecone Connection Error​

Memory Service Not Initialized​

Debug Commands​

Log Analysis​

Neo4j Logs​

Ollama Logs​

Application Logs​

Performance Optimization​

Neo4j Optimization​

Ollama Optimization​

Pinecone Optimization​

Security Considerations​

Neo4j Security​

API Security​

Monitoring and Maintenance​

Health Checks​

Monitoring Metrics​

Maintenance Tasks​

Next Steps​

Support​

Related Documentation​

Prerequisites

Quick Start

1. Install Dependencies

2. Set up Neo4j

Option A: Neo4j Desktop (Recommended)

Option B: Neo4j Server

3. Set up Ollama

4. Set up Pinecone (Optional)

5. Configure Environment

6. Test the Setup

7. Start the Test Server

Detailed Setup

Neo4j Configuration

Database Setup

Performance Tuning

Ollama Configuration

Model Management

Performance Optimization

Pinecone Configuration

Index Settings

Monitoring

Testing and Validation

Test Scripts

1. Ollama Embeddings Test

2. Neo4j Connection Test

3. Memory System Test

4. Full System Test

API Testing

Test Memory Storage

Test Memory Retrieval

Test Chat Integration

Troubleshooting

Common Issues

Neo4j Connection Failed

Ollama Not Working

Pinecone Connection Error

Memory Service Not Initialized

Debug Commands

Log Analysis

Neo4j Logs

Ollama Logs

Application Logs

Performance Optimization

Neo4j Optimization

Ollama Optimization

Pinecone Optimization

Security Considerations

Neo4j Security

API Security

Monitoring and Maintenance

Health Checks

Monitoring Metrics

Maintenance Tasks

Next Steps

Support

Related Documentation