Troubleshooting

Diagnose and resolve common Sockudo issues

Overview

This guide covers common issues you may encounter with Sockudo and provides solutions for debugging and resolving them.

Debug Logging

Enable Debug Mode

Environment Variable (Recommended):

# Enable debug logging (takes precedence over DEBUG_MODE)
DEBUG=true ./target/release/sockudo

# Alternative method
DEBUG_MODE=true ./target/release/sockudo

Rust Log Levels:

# Set log level to debug
RUST_LOG=debug ./target/release/sockudo

# Debug specific modules
RUST_LOG=sockudo::websocket=debug ./target/release/sockudo

# Multiple modules
RUST_LOG=sockudo::websocket=debug,sockudo::adapter=trace ./target/release/sockudo

Log Output Formats

Human-Readable Format (Development):

# Human format with colors (default)
LOG_OUTPUT_FORMAT=human 
LOG_COLORS_ENABLED=true

JSON Format (Production):

# JSON output for log aggregation systems
LOG_OUTPUT_FORMAT=json ./target/release/sockudo

JSON format (LOG_OUTPUT_FORMAT=json) must be set via environment variable at startup. It cannot be configured in config files due to tracing library limitations.

Configuration File:

{
  "logging": {
    "colors_enabled": false,
    "include_target": true
  }
}

You can disable colors via config file, but JSON format requires an environment variable.

Common Issues

Connection Issues

WebSocket connections failing

Symptoms:

• Clients cannot establish WebSocket connections
• Connection attempts timeout
• 502 Bad Gateway errors

Diagnosis:

# Check if Sockudo is running
curl http://localhost:6001/up/app-id

# Enable WebSocket debug logging
RUST_LOG=sockudo::websocket=debug ./target/release/sockudo

Solutions:

1. Check port binding:

   # Verify Sockudo is listening
   netstat -tlnp | grep 6001
   

2. Verify app credentials:

   # Check app configuration
   RUST_LOG=sockudo::app=debug ./target/release/sockudo
   

3. Check firewall rules:

   # Allow port 6001
   sudo ufw allow 6001/tcp
   

4. Validate origin headers (if origin validation is enabled):

   # Check logs for "Origin not allowed" errors
   grep "Origin not allowed" /var/log/sockudo.log
   

Connection drops / disconnects

Symptoms:

• Clients randomly disconnect
• Connection closed with error codes 4100, 4200, 4201
• Frequent reconnection attempts

Diagnosis:

# Monitor connection lifecycle
RUST_LOG=sockudo::websocket=debug ./target/release/sockudo

# Check buffer full errors
grep "buffer full" /var/log/sockudo.log

Solutions:

1. Increase WebSocket buffer limits:

   WEBSOCKET_MAX_MESSAGES=2000
   WEBSOCKET_MAX_BYTES=2097152  # 2MB
   

2. Adjust idle timeout:

   WEBSOCKET_IDLE_TIMEOUT=300  # 5 minutes
   

3. Enable auto-ping:

   WEBSOCKET_AUTO_PING=true
   WEBSOCKET_PING_INTERVAL=30
   

4. Check slow consumer handling:

   # Drop messages instead of disconnecting
   WEBSOCKET_DISCONNECT_ON_BUFFER_FULL=false
   

Origin not allowed errors

Symptoms:

• Error code 4009: “Origin not allowed”
• Browser clients rejected, direct clients work

Diagnosis:

# Check origin validation logs
grep "origin_not_allowed" /var/log/sockudo.log

# View current app configuration
RUST_LOG=sockudo::app=debug ./target/release/sockudo

Solutions:

1. Add allowed origins:

   # For default app
   SOCKUDO_DEFAULT_APP_ALLOWED_ORIGINS=https://app.example.com,*.staging.example.com
   

2. Update config file:

   {
     "app_manager": {
       "array": {
         "apps": [{
           "id": "my-app",
           "allowed_origins": [
             "https://app.example.com",
             "*.staging.example.com",
             "http://localhost:3000"
           ]
         }]
       }
     }
   }
   

3. Allow all origins (development only):

   {
     "allowed_origins": ["*"]
   }
   

Memory Issues

High memory usage

Symptoms:

• Memory consumption grows over time
• Out of memory errors
• Process killed by OOM killer

Diagnosis:

# Monitor memory usage
ps aux | grep sockudo

# Check Prometheus metrics
curl http://localhost:9601/metrics | grep memory

Solutions:

1. Reduce WebSocket buffer sizes:

   WEBSOCKET_MAX_MESSAGES=500
   WEBSOCKET_MAX_BYTES=524288  # 512KB per connection
   

2. Lower cleanup queue buffer:

   CLEANUP_QUEUE_BUFFER_SIZE=1000
   

3. Reduce cache capacity:

   CACHE_MAX_CAPACITY=5000
   

4. Enable aggressive cleanup:

   CACHE_CLEANUP_INTERVAL=30  # Cleanup every 30 seconds
   CACHE_TTL_SECONDS=180  # Shorter TTL
   

5. Check for memory leaks:

   # Enable trace logging for allocations
   RUST_LOG=trace ./target/release/sockudo
   

Cleanup queue memory growth

Symptoms:

• Memory grows during mass disconnections
• “Queue near capacity” warnings
• Cleanup queue overflowing

Diagnosis:

# Monitor cleanup queue
grep "Cleanup batch completed" /var/log/sockudo.log

# Check queue size metrics
curl http://localhost:9601/metrics | grep cleanup_queue

Solutions:

1. Increase batch processing speed:

   CLEANUP_BATCH_SIZE=100  # Larger batches
   CLEANUP_BATCH_TIMEOUT_MS=10  # Faster processing
   

2. Add more worker threads:

   CLEANUP_WORKER_THREADS=4  # More parallelism
   

3. Reduce queue buffer (if memory constrained):

   CLEANUP_QUEUE_BUFFER_SIZE=1000
   

4. Emergency: Disable async cleanup:

   CLEANUP_ASYNC_ENABLED=false
   systemctl restart sockudo
   

Performance Issues

High CPU usage

Symptoms:

• CPU usage near 100%
• Slow message processing
• Increased latency

Diagnosis:

# Monitor CPU usage
top -p $(pgrep sockudo)

# Check operation latency
curl http://localhost:9601/metrics | grep duration

Solutions:

1. Disable unnecessary features:

   # Disable socket counting
   ADAPTER_ENABLE_SOCKET_COUNTING=false
   

2. Adjust cleanup workers:

   # Use auto-scaling
   CLEANUP_WORKER_THREADS=auto
   

3. Increase batch timeouts:

   CLEANUP_BATCH_TIMEOUT_MS=100  # Less frequent processing
   

4. Check database connection pools:

   # Increase pool size
   DATABASE_POOL_MAX=20
   

Slow message delivery

Symptoms:

• Messages delayed or arrive out of order
• High latency between publish and receive
• Backpressure warnings

Diagnosis:

# Enable adapter debug logging
RUST_LOG=sockudo::adapter=debug ./target/release/sockudo

# Check message queue depth
curl http://localhost:9601/metrics | grep queue

Solutions:

1. Increase Redis concurrency:

   QUEUE_REDIS_CONCURRENCY=10
   

2. Optimize adapter configuration:

   # For Redis Cluster
   REDIS_CLUSTER_QUEUE_CONCURRENCY=10
   

3. Check network latency:

   # Test Redis latency
   redis-cli --latency -h redis-host
   

4. Increase buffer sizes:

   WEBSOCKET_WRITE_BUFFER_SIZE=32768  # 32KB
   

Database & Backend Issues

Redis connection failures

Symptoms:

• “Failed to connect to Redis” errors
• Adapter operations timing out
• Falling back to local adapter

Diagnosis:

# Test Redis connection
redis-cli -h $DATABASE_REDIS_HOST -p $DATABASE_REDIS_PORT ping

# Enable Redis debug logging
RUST_LOG=sockudo::adapter::redis=debug ./target/release/sockudo

Solutions:

1. Verify Redis is running:

   systemctl status redis
   

2. Check connection settings:

   DATABASE_REDIS_HOST=localhost
   DATABASE_REDIS_PORT=6379
   DATABASE_REDIS_PASSWORD=your-password
   

3. Test with Redis URL:

   REDIS_URL=redis://:password@host:6379/0
   

4. Increase connection timeout:

   # For NATS (similar concept)
   NATS_CONNECTION_TIMEOUT_MS=10000
   

Database pool exhaustion

Symptoms:

• “Connection pool timeout” errors
• Slow app lookups
• “Failed to acquire connection” errors

Diagnosis:

# Check pool metrics
curl http://localhost:9601/metrics | grep pool

# Enable database debug logging
RUST_LOG=sockudo::app=debug ./target/release/sockudo

Solutions:

1. Increase pool size:

   DATABASE_POOL_MAX=20
   DATABASE_MYSQL_POOL_MAX=32
   

2. Enable connection pooling:

   DATABASE_POOLING_ENABLED=true
   

3. Reduce pool minimum:

   DATABASE_POOL_MIN=1  # Slower startup, lower baseline memory
   

4. Enable app caching:

   {
     "app_manager": {
       "cache": {
         "enabled": true,
         "ttl": 300
       }
     }
   }
   

Issue Reference Table

Issue	Error Code	Common Cause	Quick Fix
Connection refused	-	Port not open	Check firewall, verify port binding
Origin not allowed	4009	Missing origin in allowed list	Add origin to allowed_origins
Buffer overflow	4100	Slow consumer	Increase WEBSOCKET_MAX_MESSAGES
Auth failed	4009	Invalid credentials	Verify app key/secret
Connection timeout	-	Network/firewall issue	Check network connectivity
Memory exhaustion	-	High connection count	Reduce buffer sizes, enable limits
Queue overflow	-	Mass disconnections	Increase CLEANUP_QUEUE_BUFFER_SIZE
CPU saturation	-	Too many workers	Reduce CLEANUP_WORKER_THREADS

Diagnostic Commands

Health Checks

# WebSocket health check
curl http://localhost:6001/up/app-id

# Prometheus metrics
curl http://localhost:9601/metrics

# Check active connections
curl http://localhost:6001/apps/app-id/channels

Real-time Monitoring

# Watch logs in real-time
tail -f /var/log/sockudo.log

# Monitor cleanup performance
tail -f /var/log/sockudo.log | grep -E "(Cleanup batch|Queue.*full|Failed to queue)"

# Watch connection events
tail -f /var/log/sockudo.log | grep -E "(connected|disconnected)"

Testing Tools

# Test WebSocket connection
wscat -c ws://localhost:6001/app/app-id

# Generate auth signature for private channels
echo -n "socket_id:channel_name" | openssl dgst -sha256 -hmac "app_secret" -hex

# Load test connections
for i in {1..1000}; do
  wscat -c ws://localhost:6001/app/app-id &
done

Emergency Procedures

Quick Restart

# Graceful restart (respects SHUTDOWN_GRACE_PERIOD)
systemctl reload sockudo

# Force restart
systemctl restart sockudo

Disable Features

# Disable async cleanup (emergency)
CLEANUP_ASYNC_ENABLED=false systemctl restart sockudo

# Disable rate limiting
RATE_LIMITER_ENABLED=false

# Disable metrics
METRICS_ENABLED=false

Fallback to Local Mode

# Use local adapter (no Redis required)
ADAPTER_DRIVER=local
CACHE_DRIVER=memory
QUEUE_DRIVER=memory
RATE_LIMITER_DRIVER=memory

systemctl restart sockudo

Getting Help

If you’re still experiencing issues:

1. Enable debug logging: DEBUG=true RUST_LOG=debug
2. Collect logs: Save relevant log sections
3. Gather metrics: Export Prometheus metrics
4. Document steps: Record what you tried
5. Report issue: https://github.com/soketi/sockudo/issues

Next Steps

• Performance Tuning - Optimize your deployment
• Security Best Practices - Secure your installation
• Monitoring & Metrics - Set up comprehensive monitoring