Quick review of Claudette

Introduction

Yesterday Answer.AI released a new higher level SDK for working with Anthropic Claude models: Introducing Claudette, a new friend that makes Claude 3.5 Sonnet even nicer. Here is my quick review of that library.

It is simplistic and minimal, with a stark contrast with nearly all of the other LLM libraries. The core functionality is realized in two files, one less than 300 lines, and second with just 30 lines. This simplicity in the code is impressive, yet the library covers a broad range of functionalities. It is released as Claude SDK - but when you look into the code there are signs that it is planned (for example see the comment on streaming) as a general LLM library. Maybe it will come at a later version.

The source code

The code in Github is automatically compiled from a Literate Programming version of it. The literate programming aspect is interesting and I think that it might eventually work with the right tooling. For now I consider it still an experiment. The literate version's length makes the code harder, not easier, to follow in many aspects. This might be alleviated by switching between the compiled version and the literate version - but I still haven’t got accustomed to that.

The code of the library uses a lot of @patch, changing libraries external to it. This approach also makes the code harder to understand. It might be a good trade-off but as Brian Kernighan once said: "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." Maybe it is just too much above my level.

The API

The documentation advertises the following features provided:

• A Chat class that creates stateful dialogs

• Support for prefill, which tells Claude what to use as the first few words of its response

• Convenient image support

• Simple and convenient support Tool Use API.

That is all you need to write most LLM applications but for my own work I found some limitations, that I list below, making it not ready to replace my current solutions.

I stumbled for a bit on the examples in the documentation like:

chat = Chat(model, sp="You are a helpful and concise assistant.")

chat("I'm Jeremy")

chat("What's my name?")

Here it seems as if there was a chat function imported that works with a global variable that keeps the history of the chat. But in fact chat here is a callable object which keeps the history in its own state. I was not paying attention.

When using function calling, to generate function schemas, it uses another library from Answer.AI called: toolslm. It is not evident from that library web page - but it has a get_schema function for that. And that library in turn uses docments for adding descriptions to function parameters:

def sums(

a:int, # First thing to sum

b:int=1 # Second thing to sum

) -> int: # The sum of the inputs

"Adds a + b."

print(f"Finding the sum of {a} and {b}")

return a + b

I really like it! I am thinking about adding this to LLMEasyTools. The argument against it is that these additional libraries are rather heavy. The argument for is that the double annotations of Annotated type annotations, that I use currently, are very awkward - comments seem so much more natural here.

Current limitations

That said I am not ready to use it for my own project. This is not only because I don’t want to limit it to just one LLM, even if it is currently the best one in many benchmarks, but also because of some assumptions it makes about tool usage.

String tools

Number one assumption is that it only uses tools that return a string. In my code I often want structured output from LLM - and then I use a function call for that - for example I want to get the final answer together with the reasoning that leads to it. I guess I could just parse the LLM answer for that or make a workaround with two LLM. I think they can easily incorporate tools returning objects by requiring that the objects have a sane stringification and then calling str(output) when generating the message for the LLM. That would mean that the history would be a bit more heterogenous - but I have a similar solution in my code and it works for me.

No way to add to history inside a toolloop

A very powerful technique in using LLMs is the agentic loop where we let the LLM use tools until it can fulfill a goal. I write a lot about that. In Claudette the mechanism designed for making an agentic loop is the toolloop. Overall that mechanism is refreshingly simple to use. It is such a surprise that you can do an agentic loop so easily. But in my work I discovered that I really want to add additional steps into that loop, like for example reflections that prompt the LLM to do some additional thinking about the retrieved information. I do these reflections in clean context - in a new chat using the Claudette terminology - so that I can make the LLM to focus well on some particular aspects of that additional thinking. And I don’t see how I could do that with the current toolloop design. Maybe I am missing something and I could use trace_funct for that. Certainly I can run another chat session in that - but the point is that I want to store some results of that reflection in the original chat history - so that the next tool call can be informed by it.

Immutable tool list

The third limiting assumption is about the fact that the chat always uses the same tools. In my code I try to limit the tools I offer the LLM to use to only those that can be useful at the particular stage of the chat. For example when I let the LLM to use Wikipedia - I give it tools for getting pages and for looking up information on a current page (this is needed when I don’t want to give the LLM the whole page to read - because of token limits or because I want it to focus better). I don’t want to offer the page related tools when there is no current page yet.

Conclusions

It is a wonderful library for simple LLM projects, but I still cannot use it for my more advanced work. This might be personal - but I also don’t like such extensive usage of @patch. I don’t know the authors plans for this new library - but none of the limitations that I mentioned seem fundamental - so maybe in some next version I’ll get what I need and remove huge parts of my own code from my project?

Updates

24.06.2024

It looks that toolslm does not work with more complex types. Here is an example that works with LLMEasyTools (which uses pydantic for that):

from toolslm.funccall import get_schema

from datetime import datetime, timedelta

from llm_easy_tools import get_tool_defs

def date_manip( date: datetime, action: str, amount: int):

if action == 'add':

return date + timedelta(days=amount)

elif action == 'subtract':

return date - timedelta(days=amount)

else:

raise ValueError(f'Unknown action: {action}')

print(get_tool_defs([date_manip]))

print(get_schema(date_manip))

The output is:
[{'type': 'function', 'function': {'name': 'date_manip', 'description': '', 'parameters': {'properties': {'date': {'format': 'date-time', 'type': 'string'}, 'action': {'type': 'string'}, 'amount': {'type': 'integer'}}, 'required': ['date', 'action', 'amount'], 'type': 'object'}}}]

Traceback (most recent call last):

File "/home/zby/llm/answerbot/scripts/tmp/claudette.py", line 17, in <module>

print(get_schema(date_manip))

^^^^^^^^^^^^^^^^^^^^^^

…
I truncated the error message.