Filly now runs on Claude Sonnet 5.5. Before we switched, we put both models through 60+ real analytics tasks to see how they went.
Filament brings all your data together to answer the questions you always wanted to know about your nonprofit or business. We run and train an AI data scientist to do the analysis and dashboard-building for you. This is a story about their brain.
It would almost be a full time job keeping up with new AI models this year. When they arrive, they’re always touted as state-of-the-art, complete with doom-to-humanity numbers to prove it.
We’ve found that it’s one thing for a model to be book smart, and another to be good at interpreting imperfect human questions into accurate analysis. We need life smarts too.
So, for each model that comes out we have to decide if the hype is worth it. Is this actually better for the questions you ask day to day in Filament? Cause ultimately, we need a model to combine raw intelligence with tool calling and accurately interpreting a non-technical question.
Now, Filament mostly runs on Anthropic models. They’re excellent at “agentic” work (aka.. doing stuff) and are good at picking up when a human has been a little sloppy with their prompt. Sonnet is also quite consistent, compared to Gemini or Kimi K2.6 whose correctness was starkly contrast with their ability to go down completely imaginary and incorrect tangents.
A few weeks ago, Anthropic released Claude Sonnet 5.5. I use Opus 5.5 every day to help build Filament and I am a big fan, so I thought Sonnet 5.5 might be a game changer for Filament. Was it worth the switch?
I won’t bury the lede (I know your attention span). Filly runs on Sonnet 5.5 now. In our internal benchmark of 60+ real world analytics tasks, Sonnet 5.5 answered 23% more tasks with complete accuracy and did that 24% faster than Sonnet 5.
Filament’s last exam
The easiest way to benchmark an AI analyst would be to write a bunch of questions, hand write the answers, and see how many each model gets right. Exam-style.
The trouble is that’s not how people use Filament. Usually, a conversation with the analyst in Filament is tangent after tangent until you find the insight that changes your strategy. Along the way we’re making charts, running SQL, building data definitions, and laying out dashboards.
Our benchmark was built around simulated real-world analysis scenarios that required multiple turns. After each, we inspected and judged the artefact made at the end.
Each scenario was weighted to match how often each kind of task happens in Filament. They ranged from straightforward data lookups, to multi-step analysis that contains a deliberate trap like pre-aggregated or pre-calculated data.
The benchmarks run on real data (our own business) and the correct answers are calculated live from it. We use an AI grader to score each model and each response against a written matrix.
We fail an answer immediately if it returns one wrong number. Even when an answer is correct, we rate a more precise respond higher than a general one.
We saw this in the numbers. Sonnet 5.5 scored a 90/100 average over Sonnet 5’s 84/90. High Distinctions all round.
But non-determinism
If you work with AIs regularly, you’ll know they’re non-deterministic. It’s a fancy way of calling them a little imaginative.
Sometimes they choose a more creative path.
We’ve found in our benchmarks that we need to measure and control for imagination (noise, would be the statistical term). To do this, we run our benchmark over and over again on the same model to determine how much drift there is those answers.
This tells us how much of Sonnet 5.5’s improvement is within the usual range of performance, against what is statistically significant. Our 23% result is beyond the noise we typically see in models (+-5%).
What we learnt
Across the board, Sonnet 5.5 answered 23% more tasks completely correctly (every checklist our benchmark scored was correct) than Sonnet 5, with extremely high overall accuracy on analysis questions.
Your everyday work improved a lot:
- Straightforward lookups improved the most, with around three-quarters more answers fully correct.
- Multi-step analysis improved clearly.
- Questions with a built-in trap improved more subtly.
That last group is what keeps me up at night. The expensive mistakes in analytics are rarely the hard sums; they’re the confident answer built on a number that was already an average, or on the wrong table. Filly is now noticeably better at stopping to check.
It’s more consistent, too. We only counted a task as consistently solved when Filly got it right three times out of three, and more tasks cleared that bar on Sonnet 5.5. Questions with more than one reasonable approach are more likely to land in the same place whenever you ask.
It holds up on questions it hasn’t seen. We kept some tasks aside to check we hadn’t just taught to the test. The improvement there was even bigger than overall.
It’s faster. Tasks took 21 seconds on average, down from 28, and Filly takes fewer steps to get to an answer.
Where to from here?
AI companies are intensely competitive, and we’ve since run Haiku 5.5, GPT 6.1 Sol, and Grok 4.7 through the same benchmark. None has compared and often we suffer a 10-20% drop in accuracy. In an analytics product, that doesn’t fly.
Sonnet 5.5 is rolled out across Filament. We’re excited to hear how you find it!


