perf(runtime): Hermes Phase 1-3 — prompt caching + parallel tools + smart retry
Phase 1: Anthropic prompt caching - Add cache_control ephemeral on system prompt blocks - Track cache_creation/cache_read tokens in CompletionResponse + StreamChunk Phase 2A: Parallel tool execution - Add ToolConcurrency enum (ReadOnly/Exclusive/Interactive) - JoinSet + Semaphore(3) for bounded parallel tool calls - 7 tools annotated with correct concurrency level - AtomicU32 for lock-free failure tracking in ToolErrorMiddleware Phase 2B: Tool output pruning - prune_tool_outputs() trims old ToolResult > 2000 chars to 500 chars - Integrated into CompactionMiddleware before token estimation Phase 3: Error classification + smart retry - LlmErrorKind + ClassifiedLlmError for structured error mapping - RetryDriver decorator with jittered exponential backoff - Kernel wraps all LLM calls with RetryDriver - CONTEXT_OVERFLOW recovery triggers emergency compaction in loop_runner
This commit is contained in:
@@ -24,6 +24,10 @@ pub enum StreamChunk {
|
||||
input_tokens: u32,
|
||||
output_tokens: u32,
|
||||
stop_reason: String,
|
||||
#[serde(default)]
|
||||
cache_creation_input_tokens: Option<u32>,
|
||||
#[serde(default)]
|
||||
cache_read_input_tokens: Option<u32>,
|
||||
},
|
||||
/// Error occurred
|
||||
Error { message: String },
|
||||
|
||||
Reference in New Issue
Block a user