Summary

Details

• No partition key sharding/salting
• No write coalescing or batching
• No client-side admission control before hitting DynamoDB
• RateLimiterUnavailable is raised but the caller has already been delayed

Impact

• Availability: High-traffic entities experience elevated latency and rejected requests beyond what their rate limits specify
• Fairness: Other entities sharing the same DynamoDB partition may experience collateral throttling
• Multi-tenant risk: In a shared LLM proxy scenario, one tenant's burst traffic could degrade service for others

Reproduction

1. Create an entity with high rate limits (e.g., 100,000 rpm)
2. Send sustained traffic at 1,000+ req/sec to a single entity
3. Observe DynamoDB ThrottledRequests CloudWatch metric increasing
4. Observe acquire() latency spikes and RateLimiterUnavailable exceptions

Remediation Design: Pre-Shard Buckets

• Move buckets to PK={ns}/BUCKET#{entity}#{resource}#{shard}, SK=#STATE — one partition per (entity, resource, shard)
• Auto-inject wcu:1000 reserved limit on every bucket — tracks DynamoDB partition write pressure in-band (name may change during implementation)
• Shard doubling (1→2→4→8) triggered by client on wcu exhaustion or proactively by aggregator
• Shard 0 at suffix #0 is source of truth for shard count. Aggregator propagates to other shards
• Original limits stored on bucket, effective limits derived: original / shard count. Infrastructure limits (wcu) not divided
• Shard selection: random/round-robin. On application limit exhaustion, retry on another shard (max 2 retries)
• Lazy shard creation on first access
• Bucket discovery via GSI3 (KEYS ONLY) + BatchGetItem. GSI2 for resource aggregation unchanged
• Cascade: parent unaware, protected by own wcu
• Aggregator: parse new PK format, key by shard id, effective limits for refill, filter wcu from snapshots
• Clean break migration: schema version bump, old buckets ignored, new buckets created on first access
• $0.625/M preserved on hot path