Ask HN: Burning $100K/week on LLM tokens – what are you doing to cut costs?

# LLM Token Cost Optimization Strategy

## Immediate Wins (30-40% savings)
- [ ] Enable prompt caching on Anthropic/OpenAI for system prompts
- [ ] Implement model tiering: Haiku for linting/simple tasks, Sonnet for code gen
- [ ] Set aggressive history truncation (keep last N turns only)
- [ ] Monitor cache hit rates; target >60% for cached prompts

## Medium-term (40-60% savings)
- [ ] Integrate token compression middleware (LLMLingua) at gateway
- [ ] Build context router: analyze task type → send minimal relevant context
- [ ] Implement tool trace summarization before re-sending to API
- [ ] A/B test smaller models (Haiku) for more task types

## Long-term (60%+ savings)
- [ ] Self-host models for long-tail simple tasks (linting, formatting)
- [ ] Build local embedding cache to avoid redundant context lookups
- [ ] Implement request deduplication at gateway level
- [ ] Consider fine-tuned smaller models for domain-specific tasks

## Monitoring
- Track cost per task type (linting vs. code gen vs. debugging)
- Monitor cache hit rates by prompt type
- Measure latency impact of compression/routing optimizations
- Set cost alerts by provider and model tier

# OpenClaw Gateway Middleware Stack

```
Incoming Request
    ↓
[1] Request Deduplication
    - Hash request signature
    - Return cached response if identical request in-flight
    ↓
[2] Token Compression
    - Apply LLMLingua or similar
    - Target: 20-40% token reduction
    ↓
[3] Context Router
    - Analyze task type from request
    - Extract only relevant context from knowledge base
    - Drop redundant system prompts if cached
    ↓
[4] Prompt Cache Lookup
    - Check Anthropic/OpenAI cache for matching prefix
    - Route to cached endpoint if hit
    ↓
[5] Model Selection
    - Route simple tasks → Haiku
    - Route complex tasks → Sonnet
    - Route specialized tasks → Self-hosted
    ↓
[6] Provider Routing
    - Load balance across Claude, GPT, Gemini
    - Prefer cheapest provider for equivalent quality
    ↓
API Call → Response
```

## Implementation Priority
1. Prompt cache lookup (easiest, ~30% savings)
2. Model tiering (straightforward, ~15% savings)
3. History truncation (quick win, ~10% savings)
4. Token compression (moderate effort, ~20% savings)
5. Context routing (high effort, ~25% savings)
6. Self-hosting (long-term, 50%+ for long tail)

# SQL-style cost tracking for multi-provider LLM usage

SELECT 
    task_type,
    model_used,
    COUNT(*) as request_count,
    SUM(input_tokens) as total_input_tokens,
    SUM(output_tokens) as total_output_tokens,
    SUM(cost_usd) as total_cost,
    AVG(cost_usd) as avg_cost_per_request,
    SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) as cache_hits,
    ROUND(100.0 * SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) / COUNT(*), 2) as cache_hit_rate,
    SUM(CASE WHEN compression_applied THEN compressed_tokens ELSE input_tokens END) as effective_tokens
FROM llm_requests
WHERE timestamp >= NOW() - INTERVAL '7 days'
GROUP BY task_type, model_used
ORDER BY total_cost DESC;

# Identify cost anomalies
SELECT 
    task_type,
    DATE(timestamp) as date,
    SUM(cost_usd) as daily_cost,
    AVG(input_tokens) as avg_context_size,
    MAX(input_tokens) as max_context_size
FROM llm_requests
WHERE timestamp >= NOW() - INTERVAL '30 days'
GROUP BY task_type, DATE(timestamp)
HAVING SUM(cost_usd) > (SELECT AVG(daily_cost) * 1.5 FROM (
    SELECT DATE(timestamp), SUM(cost_usd) as daily_cost FROM llm_requests GROUP BY DATE(timestamp)
) t)
ORDER BY daily_cost DESC;