๐ง Caching in Kebogyro
Caching improves speed and cost efficiency by preventing repeated tool spec fetches and model calls.
๐ง Interface: AbstractLLMCache
class AbstractLLMCache:
async def aget_value(self, key: str): ...
async def aset_value(self, key: str, value: dict, expiry_seconds: int): ...
async def adelete_value(self, key: str): ...
async def is_expired(self, key: str, expiry_seconds: int): ...
๐ Dual Caching Strategy
Kebogyro caches both:
- ๐ง Tool metadata (via
BBServerMCPClient
) - ๐ฅ Model responses + tool resolution (via
LLMClientWrapper
)
๐ฆ Example Adapter
class MyCache(AbstractLLMCache):
async def aget_value(self, key): ...
async def aset_value(self, key, value, expiry_seconds): ...
async def adelete_value(self, key): ...
async def is_expired(self, key, expiry_seconds): ...
๐งช Usage
Pass to both:
llm = LLMClientWrapper(
...,
llm_cache=MyCache()
)
mcp = BBServerMCPClient(
...,
cache_adapter=MyCache()
)
๐ฏ Tips
- Use Redis or DB cache in production.
- Avoid caching extremely dynamic responses unless necessary.
- Align expiry time with tool volatility.
Next โ Extending Providers