EVAL PASSED - gate model "qwen2.5:3b": acc 95.1%, answer 95.7%

I stared at that line for a week feeling pretty good about myself. Then I changed a prompt, ran it again, got 94.6%, and asked the obvious question: is that worse than yesterday, or is that noise?

I scrolled up to check. My terminal had eaten yesterday.

That’s it. That’s the entire reason I put Langfuse in front of a Go agent. Not because I wanted a dashboard, but because a number you can’t compare to last week is not a measurement, it’s a vibe.

The agent is Ava, a research PoC in Go: a voice agent that runs opinion polls out loud and knows when to hang up. She exists because a team here had a voice agent that didn’t know when a conversation was over, so I built the dumbest PoC that worked instead of writing a spec. One afternoon, no document.

This post is what happened after that afternoon, once “it works” stopped being enough and I wanted to know whether it kept working.

And Go is where the first small wall shows up, because Langfuse ships SDKs for Python and JS and nothing for us. Turns out that’s fine. What follows is the whole path: what an eval even is, the crude one I wrote first, and how it got to Langfuse with nothing but the standard OpenTelemetry SDK and net/http.

First, how do you know an LLM app is any good?

If you write a function that adds two numbers, the test is obvious:

if Add(2, 2) != 4 {
    t.Fatal("math is broken")
}

Now write that test for a model. Same input, run it twice, get two different sentences. Both correct. != is useless to you.

This is the part that trips people up coming from normal software: the model is not the unit under test, the behavior is. You are not asserting that the output is a specific string. You are asserting that the output has a property you care about. So the whole job becomes: pick the property, and find a way to score it.

Chip Huyen catalogs the ways to do that in AI Engineering. I ended up using three of them, and I’d learn those three before touching any platform.

1. Functional correctness

Did the system do the thing? Not “does the text look nice” - did it work.

This is the strongest kind of eval and always the one to reach for first, because there’s no interpretation involved. If you ask a model to write gcd(a, b), you don’t grade the code, you run it and check that gcd(15, 20) returns 5. It’s how LeetCode grades you and how HumanEval grades models.

Ava’s version: on every reply, a classifier decides what the conversation does next - advance, re-read the question, ask for clarification, or hang up. That decision has a right answer, so the eval is a comparison:

turn, err := classifier.ClassifyTurn(ctx, c.q, c.reply)
if turn.Intent == c.want {
    report.correct++
}

Boring. == on a label. That is a feature. The classifier’s output is constrained enough to be checked exactly, and everything downstream of that decision is a state machine I can test like any other Go code. When you can push a fuzzy thing into a small set of labels, do it - you get your == back.

2. Similarity against reference data

Some things have no single right answer. Ava’s sign-off is a personalized callback to what the respondent said. There are a thousand good ones.

Here you compare the output against reference data: a labeled corpus of (input, expected) pairs, where expected is a reference answer rather than a label. And “compare” splits in two.

Lexical similarity works on the words themselves. Overlap, edit distance, BLEU, ROUGE. Cheap, deterministic, no model in the loop, and it has no idea that “pricey” and “expensive” mean the same thing.

Semantic similarity works on meaning. You embed both texts into vectors and measure the angle between them, so paraphrase scores high. Costs an embedding call per comparison, and returns a float you now have to pick a threshold for.

In Go both are the same shape of function:

// Lexical: words in common, no model involved.
func WordOverlap(got, reference string) float64

// Semantic: embed both, measure the angle between the vectors.
func CosineSimilarity(ctx context.Context, e Embedder, got, reference string) (float64, error)

Then you score a case with whichever one fits, against a threshold you own:

score := WordOverlap(got, c.reference)
if score < 0.6 {
    report.miss(c, score)
}

I went lexical, and not because I benchmarked anything. I only needed one property: did the agent talk about things the respondent actually said? That question is about which words showed up, so meaning was never the axis.

A dozen lines of strings.Fields and a map[string]bool answered it. UnsupportedWords(line, answers) gives me the words in the sign-off that appear in neither the respondent’s answers nor the agent’s own register. A cheap groundedness check: it doesn’t ask whether the sentence is right, only whether anything in it was invented.

It caught a real one. I added a nice example closing line to the prompt, and the 3B model copied it verbatim into all 13 cases, cheerfully telling a respondent who had only said “Vanilla.” all about lavender. Every exact-match score still read 100%, because none of them were checking whether the words were earned.

There’s a tax, and it’s specifically the lexical tax: a legitimate paraphrase (“scent” for “smell”) lands in that list as a violation. Semantic scoring would forgive it, at the price of an embedding call and a threshold I’d have to defend. Either way I report the number and never gate on it. It tells me where to go look and nothing more.

3. AI as a judge

And then there’s the stuff no string comparison will ever reach. Ava says a short acknowledgment before the next question so she doesn’t sound like a form. Is “Lavender, nice one” a good ack? You know instantly. Your code has no idea.

So you ask a model, constrain it to {"good": bool, "reason": string}, and parse the JSON. In Go that’s an API call and a json.Unmarshal, and there’s nothing clever about it.

Two rules I’d hand anyone doing this. Pin one judge model for every model you evaluate, or your scores stop being comparable to each other. And never let the judge fail your build - it’s a paid, non-deterministic dependency, and a judge outage should not turn CI red.

Cost and reach go up as you move down that list. Trust goes the other way. So gate on the first kind, and merely watch the other two.

The rustic eval: a slice, a loop, and an exit code

Here’s the part I wish someone had said to me earlier: an eval is not a platform. It’s three things.

A dataset:

type evalCase struct {
    q       string
    reply   string
    want    llm.Intent
    clarity llm.Clarity
}

var dataset = []evalCase{
    {"What's your favorite scent?", "Vanilla, definitely.", llm.IntentAnswer, clear},
    {"What could we do better?", "Nothing that comes to my mind actually.", llm.IntentAnswer, clear},
    {"How do you like it?", "(coughing)", llm.IntentUnintellig, na},
    // ~80 of these, hand-labeled
}

A scorer - the loop above, in a worker pool. And a threshold:

minAcc := flag.Float64("min-acc", 0.90, "minimum overall intent accuracy to pass")
minAns := flag.Float64("min-answer", 0.95, "minimum valid-answer acceptance to pass")

// ...

if gate.acc() >= *minAcc && gate.ansRate() >= *minAns {
    fmt.Printf("\nEVAL PASSED - gate model %q: acc %.1f%%, answer %.1f%%\n", ...)
    return
}
os.Exit(1)

go run ./cmd/eval, and a non-zero exit code when the agent’s behavior regresses. That’s a real eval. Zero dependencies, and it caught actual bugs.

NOTA: the two thresholds are deliberately different. Misreading an answer as something else loses a real answer, so it’s gated hard. A flat acknowledgment is cosmetic, so it’s reported and never blocks. Gate on what loses data.

I’m not showing you the crude version because it’s charming. I’m showing it because everything Langfuse gave me afterwards is a view over exactly these three pieces. The scorer is disposable. The dataset is the asset.

Once you see that, the platform reads as storage rather than magic. Which is all I ever wanted from it, since my actual problem was that yesterday’s run no longer existed.

Two doors into Langfuse, and neither one is an SDK

Langfuse has no Go SDK. What it has is two HTTP surfaces, and between them they cover everything:

1. An OTLP endpoint at /api/public/otel/v1/traces. This is the officially supported path for any language without an SDK - their OpenTelemetry docs say it outright: “For other languages, use the native OpenTelemetry API for your language and export spans to Langfuse.” You point the standard OpenTelemetry Go SDK at it. This carries traces.
2. A REST API at /api/public/* for the things OpenTelemetry has no concept of: datasets, experiment runs, and scores. Plain net/http. The Public API docs cover auth and conventions; the full API reference is the page you’ll actually keep open.

That’s the whole architecture. internal/obs in my project is two files, one per door.

Door 1: traces over OTLP

The entire configuration is one exporter:

func Init(ctx context.Context) (shutdown func(context.Context) error, enabled bool, err error) {
    noop := func(context.Context) error { return nil }
    pk := strings.TrimSpace(os.Getenv("LANGFUSE_PUBLIC_KEY"))
    sk := strings.TrimSpace(os.Getenv("LANGFUSE_SECRET_KEY"))
    if pk == "" || sk == "" {
        return noop, false, nil // no creds: tracing off, everything is a no-op
    }

    auth := base64.StdEncoding.EncodeToString([]byte(pk + ":" + sk))
    exp, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpointURL(Host()+"/api/public/otel/v1/traces"),
        otlptracehttp.WithHeaders(map[string]string{
            "Authorization":                "Basic " + auth,
            "x-langfuse-ingestion-version": "4",
        }),
    )
    if err != nil {
        return noop, false, err
    }

    res, err := sdkresource.Merge(sdkresource.Default(),
        sdkresource.NewSchemaless(semconv.ServiceName("voicesurvey")))
    if err != nil {
        return noop, false, err
    }
    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
    otel.SetTracerProvider(tp)
    return tp.Shutdown, true, nil
}

Three things in there are worth more than the rest.

Auth is Basic auth over your key pair. Public key as user, secret key as password, base64, done. No token dance.

x-langfuse-ingestion-version: 4 is the current ingestion contract, and their docs are blunt about it: include it “so that new data appears in real time.” Send it. Without it the endpoint falls back to older mapping behavior and your spans land looking subtly wrong.

NewSchemaless, not NewWithAttributes. This one cost me an afternoon. If you pin your own semconv schema URL on the resource, it conflicts with the one the SDK’s default resource already carries, and Merge fails your whole init on a schema mismatch. Schemaless attributes merge cleanly and survive SDK upgrades.

And notice what happens with no credentials: the global tracer stays a noop, so every instrumented call in the codebase costs nothing. The PoC still runs fully offline, which is not a small thing when your gate model is local.

Instrumenting: wrap, don’t edit

Now, what gets traced? I did not sprinkle spans through the agent. Every LLM caller in the project is an interface, so tracing is a decorator:

func TraceClassifier(inner llm.Classifier, model string) llm.Classifier {
    return &tracedClassifier{inner: inner, model: model}
}

func (t *tracedClassifier) ClassifyTurn(ctx context.Context, question, reply string) (llm.Turn, error) {
    ctx, span := otel.Tracer("voicesurvey").Start(ctx, "classify_turn")
    defer span.End()

    input, _ := json.Marshal(map[string]string{"question": question, "reply": reply})
    span.SetAttributes(
        attribute.String("gen_ai.request.model", t.model),
        attribute.String("langfuse.observation.type", "generation"),
        attribute.String("langfuse.observation.input", string(input)),
        attribute.String("langfuse.trace.name", "classify_turn"),
        attribute.String("langfuse.trace.metadata.prompt_version", llm.ClassifyPromptVersion()),
    )

    turn, err := t.inner.ClassifyTurn(ctx, question, reply)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return turn, err
    }
    output, _ := json.Marshal(turn)
    span.SetAttributes(attribute.String("langfuse.observation.output", string(output)))
    return turn, nil
}

One line at the call site turns it on: cl = obs.TraceClassifier(cl, name). The wrapper is a pure pass-through, and it never touches the Turn or the error. A wrapper that can alter a result is a bug you’ll eventually blame on the model.

The langfuse.* attribute prefix is the part you can’t guess from the OTel docs. Those are Langfuse’s own conventions: observation.type makes the span render as a generation with input/output panels instead of a bare timing bar, and any langfuse.trace.metadata.* key becomes a filterable field in the UI.

I stamp prompt_version on every single call. Without it a prompt edit is invisible, because old and new output land in one pile with no axis to split them.

This is what those attributes look like once they land. One classify_turn trace, 0.31s, from a Go program with no SDK:

Every field in that screenshot came from a SetAttributes call. Nothing was configured in the UI. And notice the left sidebar while you’re there, because it settles the misconception I’ll get to below: Evaluation and Prompt Management are two separate sections. Neither depends on the other.

Why traces, and not just numbers

Here’s what I underestimated. I went to Langfuse for run history and got something I hadn’t asked for.

A voice conversation isn’t one model call. It’s transcribe, classify, write the acknowledgment, synthesize, and around again, twenty turns deep. When a conversation feels wrong, the aggregate accuracy number is useless. It tells me how often something broke and never what happened.

A trace is the whole turn as one nested object: inputs, outputs, latency per step, in order, tagged with the session id. Scoring a conversation turns out to be much easier than scoring a sentence, because the sentence finally has context around it. The respondent’s previous answer sits right there, above the decision that misread it.

And latency stops hiding. In a voice call, TTS and transcription time is dead air, and it never shows up in the LLM spans. So those get traced too, with a tiny helper for steps that aren’t LLM calls:

op := obs.StartOp(ctx, "tts")
op.In(text).Bytes("audio.bytes", len(pcm))
defer op.End()

Those spans carry the session id, which is what turns twenty separate traces back into one readable conversation:

24 traces for one poll. That’s the number that convinced me a dashboard of averages was never going to be enough.

The thing I got wrong: evals live on top of traces

Now the misconception that kept me from starting, which is the real reason I wanted to write this post.

I walked into this assuming that to run evals in Langfuse I’d have to move my prompts into Langfuse first. Adopt their prompt management, version prompts in their UI, let their runner execute them. That’s a real migration, and it’s the kind of price that makes you close the tab and decide the terminal is fine actually.

It’s not true. Evaluation in Langfuse hangs off traces. The unit is a Score, and the score data model is the page that settled it for me: a name, a value, a data type, and exactly one subject - a trace, an observation, a session, or a dataset run. Trace is listed as the common case. Nothing about a score cares where your prompt lives. You emit the trace from Go, you attach scores to it, and you’re doing evals.

You don’t hand a restaurant critic your recipe. They eat the dish.

Prompt management only becomes a prerequisite for one specific thing: asking Langfuse to execute a prompt itself against a dataset. I never wanted that. My prompt lives in Go, next to the code that depends on it, and it stays there.

Door 2: scores over REST

A score is a small POST. The whole client is net/http:

// Score is one judgment. Exactly ONE subject must be set.
type Score struct {
    Name         string
    Value        float64
    DataType     string // NUMERIC or BOOLEAN
    Comment      string
    TraceID      string
    DatasetRunID string
    ID           string // supply it and the write is idempotent
}

Attaching a per-case verdict to the trace that produced it:

c.Score(ctx, obs.Score{
    ID:       obs.StableID(runName, itemID(cr.c), "intent_correct"),
    Name:     "intent_correct",
    Value:    correct,
    DataType: "BOOLEAN",
    TraceID:  cr.traceID,
    Comment:  fmt.Sprintf("want %s, got %s", cr.c.want, cr.got.Intent),
})

That single call is what turned my scrolling terminal into something usable. In the UI I filter to intent_correct = 0 and read only the misses, each one clickable straight through to the full conversation that produced it.

To capture the trace id, the tracing wrapper drops it into a ref carried on the context:

var ref obs.TraceRef
ctx = obs.WithTraceRef(ctx, &ref)
turn, err := cl.ClassifyTurn(ctx, c.q, c.reply)
outcomes[i] = caseResult{c: c, got: turn, err: err, traceID: ref.ID()}

And the offline side - my hand-written dataset - maps onto Langfuse’s dataset experiments the same way. The corpus becomes a dataset, each model’s pass becomes a run, each case becomes a run item that links the dataset item to the trace it produced. Aggregate metrics attach to the run instead of a trace:

c.Score(ctx, obs.Score{
    ID:           obs.StableID(runName, "intent_accuracy"),
    Name:         "intent_accuracy",
    Value:        r.acc(),
    DataType:     "NUMERIC",
    DatasetRunID: runID,
})

Which makes the point concrete: even an offline eval run goes through tracing, because a run item requires a trace id. There’s no path into Langfuse that skips the traces.

And this is the payoff for the whole exercise, the thing my terminal could never give me:

That’s the sign-off eval from earlier, the one UnsupportedWords belongs to. Four runs of the same 13 cases, each row tagged with the prompt hash that produced it, lined up so a change is something I read instead of something I remember. The baseline row and the row after the fix are both still there weeks later, which is the entire thing my terminal failed to do.

And that’s what finally answers the question I opened with, which storage alone never could. Is 94.6% worse than yesterday, or is it noise? is really two questions, and the one that matters is whether the prompt moved.

Two rows carrying the same fingerprint and different scores: that’s noise, and the honest response is to widen the dataset or stop reading that decimal place. Two rows with different fingerprints and different scores: that’s your change, and now you can argue about it.

The history is what let me ask the question next week. The fingerprint is what made it answerable at all.

Four gotchas that will bite you

Flush before you link. The OTLP batch exporter is asynchronous, and the run-item endpoint rejects a trace it hasn’t ingested yet. So force a flush first, and treat a 404 as retryable:

if err := obs.Flush(ctx); err != nil {
    return fmt.Errorf("flush traces: %w", err)
}

Retry the 404, and only the 404. A 404 there means nothing was created, so retrying is safe. That endpoint is otherwise not idempotent - the server mints the id - so no other status is ever worth retrying.

Learn which endpoints upsert. Datasets upsert by name, dataset items upsert by id, scores upsert by supplied id. So I derive ids from a content hash and re-push the whole corpus on every run without creating a single duplicate:

func itemID(c evalCase) string {
    return obs.StableID("turn-classifier", c.q, c.reply)
}

Edit dataset.go and only the cases that actually changed change upstream. Run items are the exception, so they’re posted exactly once and never blind-retried.

You now own prompt versioning. This is the bill for keeping prompts in Go, and it arrives quietly. Langfuse’s prompt management would have versioned them for me. My prompts are Go string constants, so nothing versions them but me, and a score I can’t attribute to a prompt is back to being a vibe.

My first instinct was a constant to bump by hand. Terrible idea: I would edit the prompt, forget the bump, and quietly attribute new output to the old version. Worse than no version at all, because it looks trustworthy.

So the version is derived from the prompt instead:

// ClassifyPromptVersion is a short, stable fingerprint of the classifier's
// instructions: the system prompt plus the few-shot anchors. Content-addressed,
// so it cannot drift out of sync the way a hand-bumped number would.
func ClassifyPromptVersion() string {
    _, shots := classifyPrompt("", "")
    h := sha256.New()
    h.Write([]byte(classifySystem))
    for _, m := range shots {
        h.Write([]byte(m.Role))
        h.Write([]byte(m.Content))
    }
    return hex.EncodeToString(h.Sum(nil))[:12]
}

Twelve hex characters, and that’s the 79bec7b42725 in the trace screenshot above. Change a single word of the prompt and the fingerprint changes with it, whether I remembered to think about it or not.

Two details worth stealing. Hash the few-shot examples too, not just the system prompt, because an edited example changes behavior every bit as much as an edited instruction. And the test asserts only that the value is stable across calls and 12 characters long, never what the value is - a test pinning the hash would fail on every legitimate prompt edit, which is the fastest way to teach yourself to ignore a red test.

The trade is real and I’d make it again: an opaque 79bec7b42725 is less readable than v3, and it cannot lie to me.

What I’d tell a friend starting this in Go

The absence of an SDK looked like a blocker and was a two-file inconvenience. internal/obs: one file for OTLP traces, one for the REST client. That’s the whole integration.

So, in order:

1. Write the dataset first. Twenty labeled cases in a []struct{} beats any tool you haven’t picked yet. The cases are the asset; everything else is downstream of them.
2. Score exactly, where you can. Push fuzzy outputs into small label sets and get your == back. Reach for similarity and judges only for what genuinely resists that, and don’t gate on them.
3. Point the standard OTel SDK at /api/public/otel/v1/traces. Basic auth, x-langfuse-ingestion-version: 4, NewSchemaless. You are done in 30 lines.
4. Instrument by wrapping interfaces, never by editing call sites. And keep the wrapper a pure pass-through.
5. Your prompts can stay in Go. Scores hang on traces. Fingerprint the prompt yourself and stamp it on every span, and you keep both the prompt and its history where the code is.

Ava’s numbers still print in the terminal, exactly like before. The difference is that they no longer vanish when I close it.

Thanks for reading!

Book recommendation: AI Engineering, by Chip Huyen - the evaluation chapters are the clearest treatment of this I’ve found, and the taxonomy in the first half of this post comes from there.