Skip to content

๐Ÿง  Caching in Kebogyro

Caching improves speed and cost efficiency by preventing repeated tool spec fetches and model calls.


๐Ÿ”ง Interface: AbstractLLMCache

class AbstractLLMCache:
    async def aget_value(self, key: str): ...
    async def aset_value(self, key: str, value: dict, expiry_seconds: int): ...
    async def adelete_value(self, key: str): ...
    async def is_expired(self, key: str, expiry_seconds: int): ...

๐Ÿ”„ Dual Caching Strategy

Kebogyro caches both:

  • ๐Ÿ”ง Tool metadata (via BBServerMCPClient)
  • ๐Ÿ“ฅ Model responses + tool resolution (via LLMClientWrapper)

๐Ÿ“ฆ Example Adapter

class MyCache(AbstractLLMCache):
    async def aget_value(self, key): ...
    async def aset_value(self, key, value, expiry_seconds): ...
    async def adelete_value(self, key): ...
    async def is_expired(self, key, expiry_seconds): ...

๐Ÿงช Usage

Pass to both:

llm = LLMClientWrapper(
    ...,
    llm_cache=MyCache()
)

mcp = BBServerMCPClient(
    ...,
    cache_adapter=MyCache()
)

๐ŸŽฏ Tips

  • Use Redis or DB cache in production.
  • Avoid caching extremely dynamic responses unless necessary.
  • Align expiry time with tool volatility.

Next โ†’ Extending Providers