Why my agent started to speak Chinese - how to understand and analyze my agent’s conversations with my clients

So, I built an agent using LangGraph. Cool, right? I talked to it. A lot. I mean, when you build something, you wanna see it come to life. And yeah, I know there are a hundred guides about testing and analyzing your agent, but let’s be real, I’m doing this solo. Not in a tragic, violin-music-in-the-background way. Just in a “too busy to breathe” kind of way.
Anyway, I started scrolling through my LangSmith traces to see what my agent was actually saying to people. And BAM, suddenly it’s all looking like Chinese calligraphy, literally talkin’ Chinese, like my agent just came back from a study-abroad semester in Beijing and forgot we were building for English-speaking users.. Like, full-on artistic nonsense. Did I mess up the prompt? Was it a rogue parameter in the invoke function? What the fuck just happened?

Let me be honest upfront: I built a solution for this, and yep, I want you to try it out. Shameless plug? Maybe. But also, I actually believe this can help. Because whether or not you use my solution, the most important thing is this: You need to understand your agent’s conversations with users. That’s what separates someone who’s just shipping an AI toy from someone building a real, great product.
Insight = Product.
No insight = Mystery box that sometimes answers questions in Chinese.

scrolling through endless chat logs is just like looking at a million raw data numbers. It’s chaos. Few years ago, we had some basic semantic tools to analyze conversations. And then — BOOM — LLMs entered the chat. And now? They’re not just the problem. They’re also the solution.
Sounds funny, but yeah — we need agents to analyze agents. (Yes, yes, I hear the purists: “That’s not a real agent, just a sequence of LLM calls!” And you’re technically right. But I’m vibing here, let me have this.)

So here’s the naive idea: throw all your agent conversations into ChatGPT and ask for insights. And honestly, you can do that. But if you’ve built an agent, you probably already know the punchline: the details matter.

Let me break it down with two golden rules I discovered:

1. Know what the hell you’re looking for — Like any real analysis, start with a question and what action it should lead to.
*2. AI ain’t cheap *— LLM tokens = money. So don’t be dumb with how you use them.

Let’s talk example.

I dumped all my agent conversations into my analyzer-agent and asked:
”How can I make my platform better?”
The result? Slow AF. Tons of exceptions from prompt length. And answers that were as clear as mud.
So I got smarter. I reframed the question: Our agent helps users build workflow automations in a way we call vibe-automating (more on that in another post). One of our pain points? Integrations. There are tools, APIs, plugins, and way too much noise about MCP. Still, making killer integrations is hard.

So the new question became:
”Which integrations are users asking for that we don’t yet support?” (And the action: prioritize building those.)
This time, I structured the LLM output into a table with columns like: Chat | Task | Needed Integration. The results? So. Much. Better.

But here’s the next level: optimization
People talk a lot. A lot. Some of these agent chats are novels. Analyzing them can get very expensive. So, we need a smart architecture. Here’s what worked for me:

1. The Cheap Chop
   Split long conversations into context-sized chunks. Throw out irrelevant fluff. Ask for concise, clear, structured answers in your prompt.

2. The 2-Step Power Move — Use a two-stage LLM pipeline:

3. Stage 1 (cheap model) — Pull out only the important parts from all the chats. I used GPT-4.1-nano, but any budget-friendly model works.

4. Stage 2 (fancy model) — Feed that filtered gold into a more advanced LLM for deep insights.

So how’d I build this?

Lazy mode: I used my platform www.nex-craft.com — to build this pipeline with agents. Because I hate wasting time and love working smart. You can build your own version, of course. But if you’re curious, here’s how NexCraft helps:

You can get your conversation data in two ways:

1. Direct DB Integration — Hook up your conversation database. We have an agent that helps you do that. Lazy-friendly.
2. LangSmith Traces Integration— Use our “LangSmith Traces” node and pull convo data straight from your agent logs.

** We’re using Azure OpenAI to run our LLMs, mainly because it guarantees that your conversations are not used for model training. This is huge when you’re working with sensitive or proprietary data. If you’re planning to implement something similar with other providers, make sure to check their data usage policies, and explicitly choose a setup that protects your users’ privacy.

Also — and this is super important — make sure there’s no PII (Personally Identifiable Information) in the data you send to the AI tools. Clean your inputs. Your users trust you, and you don’t want to mess that up by accidentally leaking their emails, names, or grandma’s secret lasagna recipe.

That’s all, folks. Hope this helped! If you’re also wrangling agents, trying to decode what they’re saying to your users — I’d love to learn from your journey too.

Let’s stop building chatbots that hallucinate and start building tools that truly understand the people using them.