vibetunnel/benchmark/README.md

VibeTunnel Protocol Benchmark Tool

A comprehensive benchmarking tool for testing VibeTunnel server performance and protocol compliance.

Features

• Session Management: Test session creation, retrieval, and deletion performance
• SSE Streaming: Benchmark Server-Sent Events streaming latency and throughput
• Concurrent Load: Simulate multiple users for load testing
• Protocol Compliance: Full VibeTunnel HTTP API client implementation

Installation

cd benchmark
go mod tidy
go build -o vibetunnel-bench .

Usage

Basic Connectivity Test

./vibetunnel-bench session --host localhost --port 4031 --count 5

Session Management Benchmark

# Test session lifecycle with 10 sessions
./vibetunnel-bench session --host localhost --port 4031 --count 10 --verbose

# Custom shell and working directory
./vibetunnel-bench session --host localhost --port 4031 --shell /bin/zsh --cwd /home/user

SSE Streaming Benchmark

# Test streaming performance with 3 concurrent sessions
./vibetunnel-bench stream --host localhost --port 4031 --sessions 3 --duration 30s

# Sequential streaming test
./vibetunnel-bench stream --host localhost --port 4031 --sessions 5 --concurrent=false

# Custom commands to execute
./vibetunnel-bench stream --host localhost --port 4031 --commands "echo test,ls -la,date"

Concurrent Load Testing

# Simulate 20 concurrent users for 2 minutes
./vibetunnel-bench load --host localhost --port 4031 --concurrent 20 --duration 2m

# Load test with custom ramp-up period
./vibetunnel-bench load --host localhost --port 4031 --concurrent 50 --duration 5m --ramp-up 30s

Command Reference

Global Flags

• --host: Server hostname (default: localhost)
• --port: Server port (default: 4026)
• --verbose, -v: Enable detailed output

Session Command

• --count, -c: Number of sessions to create (default: 10)
• --shell: Shell to use (default: /bin/bash)
• --cwd: Working directory (default: /tmp)
• --width: Terminal width (default: 80)
• --height: Terminal height (default: 24)

Stream Command

• --sessions, -s: Number of sessions to stream (default: 3)
• --duration, -d: Benchmark duration (default: 30s)
• --commands: Commands to execute (default: ["echo hello", "ls -la", "date"])
• --concurrent: Run streams concurrently (default: true)
• --input-delay: Delay between commands (default: 2s)

Load Command

• --concurrent, -c: Number of concurrent users (default: 10)
• --duration, -d: Load test duration (default: 60s)
• --ramp-up: Ramp-up period (default: 10s)

Example Output

Session Benchmark

🚀 VibeTunnel Session Benchmark
Target: localhost:4031
Sessions: 10

Testing connectivity... ✅ Connected

📊 Session Lifecycle Benchmark
✅ Created 10 sessions in 0.45s
✅ Retrieved 10 sessions in 0.12s
✅ Listed 47 sessions in 2.34ms
✅ Deleted 10 sessions in 0.23s

📈 Performance Statistics
Overall Duration: 0.81s

Operation Latencies (avg/min/max in ms):
  Create:  45.23 /  38.12 /  67.89
  Get:     12.45 /   8.23 /  18.67
  Delete:  23.78 /  19.45 /  31.23

Throughput:
  Create: 22.3 sessions/sec
  Get:    83.4 requests/sec
  Delete: 43.5 sessions/sec

Stream Benchmark

🚀 VibeTunnel SSE Stream Benchmark
Target: localhost:4031
Sessions: 3
Duration: 30s
Concurrent: true

📊 Concurrent SSE Stream Benchmark

📈 Stream Performance Statistics
Total Duration: 30.12s

Overall Results:
  Sessions: 3 total, 3 successful
  Events: 1,247 total
  Data: 45.67 KB
  Errors: 0

Latency (average):
  First Event: 156.3ms
  Last Event: 29.8s

Throughput:
  Events/sec: 41.4
  KB/sec: 1.52
  Success Rate: 100.0%

✅ All streams completed successfully

Protocol Implementation

The benchmark tool implements the complete VibeTunnel HTTP API:

• POST /api/sessions - Create session
• GET /api/sessions - List sessions
• GET /api/sessions/{id} - Get session details
• POST /api/sessions/{id}/input - Send input
• GET /api/sessions/{id}/stream - SSE stream events
• DELETE /api/sessions/{id} - Delete session

Performance Testing Tips

1. Start Small: Begin with low concurrency and short durations
2. Monitor Resources: Watch server CPU, memory, and network usage
3. Baseline First: Test single-user performance before load testing
4. Network Latency: Account for network latency in benchmarks
5. Realistic Workloads: Use commands and data patterns similar to production

Troubleshooting

Connection Refused

• Verify server is running: curl http://localhost:4031/api/sessions
• Check firewall and port accessibility

High Error Rates

• Reduce concurrency level
• Increase timeouts
• Check server logs for resource limits

Inconsistent Results

• Run multiple iterations and average results
• Ensure stable network conditions
• Close other applications using system resources