Maintenance & Operations

Backup, monitoring, updates, and day-to-day operations

Maintenance & Operations

Keep your AFFiNE instance running smoothly with regular maintenance, backups, and monitoring.

Database Backups

Regular backups are critical for data safety. AFFiNE uses PostgreSQL, which provides multiple backup methods.

Automated Daily Backups

Create a backup script that runs daily:

backup-affine.sh

#!/bin/bash
set -e

# Configuration
BACKUP_DIR="/var/backups/affine"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Backup database
echo "Backing up database..."
docker compose exec -T postgres pg_dump -U affine affine | gzip > "$BACKUP_DIR/database_$DATE.sql.gz"

# Backup storage volumes
echo "Backing up storage..."
tar -czf "$BACKUP_DIR/storage_$DATE.tar.gz" -C ~/.affine/self-host storage/

# Backup configuration
echo "Backing up configuration..."
tar -czf "$BACKUP_DIR/config_$DATE.tar.gz" -C ~/.affine/self-host config/

# Remove old backups
echo "Cleaning up old backups..."
find "$BACKUP_DIR" -name "*.gz" -mtime +$RETENTION_DAYS -delete

echo "Backup completed: $DATE"

Make it executable and schedule with cron:

chmod +x backup-affine.sh

# Add to crontab (runs daily at 2 AM)
crontab -e
0 2 * * * /path/to/backup-affine.sh >> /var/log/affine-backup.log 2>&1

Manual Backup

Create a backup manually:

1. Stop the application (optional)

   For consistent backups, stop the application:

   docker compose stop affine
   
2. Backup the database

   docker compose exec postgres pg_dump -U affine affine > affine-backup-$(date +%Y%m%d).sql
   
3. Backup storage and config

   tar -czf affine-storage-$(date +%Y%m%d).tar.gz ~/.affine/self-host/storage/
   tar -czf affine-config-$(date +%Y%m%d).tar.gz ~/.affine/self-host/config/
   
4. Restart the application

   docker compose start affine
   

Continuous Backup with WAL Archiving

For zero-downtime backups and point-in-time recovery:

# Configure PostgreSQL for WAL archiving
docker compose exec postgres psql -U affine -c "ALTER SYSTEM SET wal_level = replica;"
docker compose exec postgres psql -U affine -c "ALTER SYSTEM SET archive_mode = on;"
docker compose exec postgres psql -U affine -c "ALTER SYSTEM SET archive_command = 'test ! -f /var/lib/postgresql/wal_archive/%f && cp %p /var/lib/postgresql/wal_archive/%f';"

# Restart PostgreSQL
docker compose restart postgres

Test your backups regularly by restoring to a separate environment. A backup you can’t restore is worthless.

Restoring from Backup

Database Restoration

1. Stop the application

   docker compose down
   
2. Start only the database

   docker compose up -d postgres
   
3. Drop and recreate the database

   docker compose exec postgres psql -U affine -c "DROP DATABASE affine;"
   docker compose exec postgres psql -U affine -c "CREATE DATABASE affine;"
   
4. Restore from backup

   # From uncompressed backup
   docker compose exec -T postgres psql -U affine affine < affine-backup.sql
   
   # From compressed backup
   gunzip < affine-backup.sql.gz | docker compose exec -T postgres psql -U affine affine
   
5. Restore storage and config

   tar -xzf affine-storage-backup.tar.gz -C ~/.affine/self-host/
   tar -xzf affine-config-backup.tar.gz -C ~/.affine/self-host/
   
6. Start the application

   docker compose up -d
   

Updates and Upgrades

Updating to Latest Stable Version

1. Check current version

   docker compose exec affine node -e "console.log(require('./package.json').version)"
   
2. Backup before updating

   Always create a backup before updating:

   ./backup-affine.sh
   
3. Pull new image

   docker compose pull affine
   
4. Review changelog

   Check release notes for breaking changes.

5. Apply update

   docker compose down
   docker compose up -d
   
6. Verify

   Check logs for successful startup:

   docker compose logs -f affine
   

Switching Release Channels

Change between stable, beta, or canary:

.env

# Switch to beta
AFFINE_REVISION=beta

# Or canary (not recommended for production)
AFFINE_REVISION=canary

Then update:

docker compose down
docker compose pull
docker compose up -d

Downgrading from canary or beta to stable may cause issues. Always test in a non-production environment first.

Monitoring

Health Checks

Monitor service health:

# Check all containers
docker compose ps

# Check specific service health
docker compose exec postgres pg_isready -U affine
docker compose exec redis redis-cli ping

Log Monitoring

View and follow logs:

# All services
docker compose logs -f

# Specific service
docker compose logs -f affine

# Last 100 lines
docker compose logs --tail=100 affine

# Since specific time
docker compose logs --since 2024-01-01T00:00:00 affine

Prometheus Metrics

Enable metrics in configuration:

config.json

{
  "metrics": {
    "enabled": true
  }
}

Metrics are exposed at http://localhost:3010/metrics.

Example Prometheus Configuration

prometheus.yml

scrape_configs:
  - job_name: 'affine'
    static_configs:
      - targets: ['localhost:3010']
    metrics_path: '/metrics'
    scrape_interval: 15s

Key metrics to monitor:

• http_request_duration_seconds - Request latency
• http_requests_total - Request count by status
• process_resident_memory_bytes - Memory usage
• nodejs_heap_size_used_bytes - Heap usage
• websocket_connections_active - Active WebSocket connections

Database Monitoring

Monitor database performance:

-- Active connections
SELECT count(*) FROM pg_stat_activity;

-- Database size
SELECT pg_size_pretty(pg_database_size('affine'));

-- Table sizes
SELECT 
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

-- Slow queries
SELECT pid, now() - pg_stat_activity.query_start AS duration, query 
FROM pg_stat_activity 
WHERE state = 'active' 
AND now() - pg_stat_activity.query_start > interval '5 seconds';

Performance Optimization

Database Maintenance

Run regular maintenance:

# Vacuum and analyze
docker compose exec postgres vacuumdb -U affine -d affine -z -v

# Reindex
docker compose exec postgres reindexdb -U affine -d affine

Schedule with cron:

# Weekly vacuum (Sundays at 3 AM)
0 3 * * 0 docker compose -f /path/to/docker-compose.yml exec -T postgres vacuumdb -U affine -d affine -z

Redis Optimization

Configure Redis memory limits:

docker-compose.yml

redis:
  image: redis
  command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru

Resource Limits

Set container resource limits:

docker-compose.yml

affine:
  image: ghcr.io/toeverything/affine:stable
  deploy:
    resources:
      limits:
        cpus: '2'
        memory: 4G
      reservations:
        cpus: '1'
        memory: 2G

Troubleshooting Common Issues

High Memory Usage

Check Node.js heap usage:

docker compose exec affine node -e "console.log(process.memoryUsage())"

Restart the application to clear memory:

docker compose restart affine

Consider increasing container memory limits if needed.

Database Connection Pool Exhausted

Check active connections:

docker compose exec postgres psql -U affine -c "SELECT count(*) FROM pg_stat_activity;"

Adjust connection pool in Prisma configuration:

config.json

{
  "db": {
    "prisma": {
      "connection_limit": 100
    }
  }
}

Slow Performance

1. Check database query performance (see Database Monitoring)
2. Review slow query log
3. Verify sufficient resources (CPU, RAM, disk I/O)
4. Check for Docker volume performance issues
5. Consider using named volumes instead of bind mounts

Disk Space Issues

Check disk usage:

df -h
docker system df

Clean up old Docker resources:

docker system prune -a --volumes

Check database size and clean old data:

docker compose exec postgres psql -U affine -c "SELECT pg_size_pretty(pg_database_size('affine'));"

Scaling Considerations

Horizontal Scaling

AFFiNE application servers are stateless and can be scaled horizontally:

docker-compose.yml

affine:
  image: ghcr.io/toeverything/affine:stable
  deploy:
    replicas: 3
  environment:
    - REDIS_SERVER_HOST=redis
    - DATABASE_URL=postgresql://affine:password@postgres:5432/affine

Use a load balancer (Nginx, HAProxy, Traefik) to distribute traffic.

Database Scaling

For high-traffic deployments:

• Use read replicas for PostgreSQL
• Consider managed database services (AWS RDS, Google Cloud SQL)
• Implement connection pooling with PgBouncer

Storage Scaling

Move to S3-compatible storage for better scalability:

config.json

{
  "storages": {
    "blob": {
      "storage": {
        "provider": "aws-s3",
        "bucket": "affine-production",
        "config": {
          "endpoint": "https://s3.amazonaws.com",
          "region": "us-east-1"
        }
      }
    }
  }
}

Disaster Recovery

Recovery Plan

1. Prepare

   • Maintain multiple backup locations (local + remote)
   • Document recovery procedures
   • Test recovery process quarterly
2. Detect

   • Monitor health checks and metrics
   • Set up alerts for failures
   • Have on-call procedures
3. Respond

   • Follow incident response procedures
   • Communicate with users
   • Switch to backup systems if needed
4. Recover

   • Restore from latest backup
   • Verify data integrity
   • Resume normal operations
5. Review

   • Conduct post-mortem
   • Update procedures
   • Implement preventive measures

High Availability Setup

For mission-critical deployments:

1. Load Balancer: Use HAProxy or Nginx with multiple AFFiNE instances
2. Database: PostgreSQL with streaming replication and automatic failover (e.g., Patroni)
3. Redis: Redis Sentinel or Redis Cluster for high availability
4. Storage: S3 with cross-region replication
5. Monitoring: Prometheus + Grafana with alerting

Next Steps

• Security best practices
• Configuration reference
• Deployment guide