Evaluating Open-Weight LLMs For Our Structured Equity & Financial Research Agent

A field report on substituting Ollama-hosted models (GLM-5.2, DeepSeek-V4-pro, gpt-oss:120b) for Claude Opus

I previously mentioned that we built our own AI agent, inspired by OpenClaw, some months ago. One of the most interesting features of the agent is our ability to swop in and out different LLMs, both to test their capabilities and to minimize token cost. The recent launch of GLM 5.2 has generated quite a stir, especially in light of Anthropic’s latest frontier models currently being restricted by the US Administration, and OpenAI voluntarily restricting distribution of their latest model.

We decided to test out GLM 5.2 in particular as the engine for our Agent. This is what we found. Full disclosure, my brother is the brains behind our Agent. This report reflects his work, lightly edited by me personally.

Executive summary

We ran a controlled, real-world test that swapped open-weight, Ollama-hosted language models in for Claude Opus as the reasoning core of our structured equity and financial research agent. This agent is not a chatbot. It runs a disciplined production pipeline: it assembles a single working draft, puts that draft through automated red-team and fact-check gates, repairs the draft in place, converges, and delivers an institution-grade research note. The reason to attempt the swap was cost. Open models on a flat-rate plan are far cheaper per unit of output than a frontier commercial model billed by the token.

The result, based off two days of production logs, was clear. The candidate open models were not usable as the agent’s main reasoning loop, and were not safe as substitutes for the verification gates, when judged against Claude Opus.

The gap was not in fluency or financial knowledge. The prose these models generate can read perfectly well. The gap was in the three capabilities that actually make the agent function

• Holding a multi-step procedure together

• Returning output in the exact machine-readable shape the pipeline requires

• Satisfying the quality gates on the model’s own merits.

These happen to be the dimensions on which open models most visibly trail frontier models in the public benchmarks, and our logs matched the benchmark story closely. The agent now runs on Claude Opus again for both the reasoning loop and the gates.

What follows describes each problem in general terms, explains the mechanism behind it rather than just the symptom, and points to external evidence that the pattern is general rather than specific to our setup.

Let’s dig in…