Saving Eve Eval Failures to Langfuse Datasets to Improve Evaluation Criteria
A validation note on sending a failed Eve weather-tool eval to Langfuse, comparing the failed run with a revised run, and improving the evaluation criteria.
Posts
Rubrics and Evals Are Not Correct from Day One. Grow Them with Failure Logs
A small verification of how coarse evaluation criteria can pass weak LLM output. This article shows how to treat rubrics and evals as operational assets that improve through failure logs.
Can Langfuse Assistant Serve as an Entry Point for Operations Investigation?
Testing the Langfuse Assistant public beta against existing Sakana Fugu observations and ground truth calculated through the Public API.
Making Your Blog Readable by AI Agents: AIO with llms.txt and MCP Server
Making a personal blog discoverable by AI agents as Perplexity and ChatGPT replace traditional search. Implementation of AIO (AI Optimization) with llms.txt, JSON API, and MCP Server design—decision criteria for a 46-post blog.
When You Build a Minimal API Loop, You Stop Designing Prompts and Start Designing Stop Conditions
I implemented a minimal generate-evaluate-feedback-regenerate loop in a verification script. This post organizes the stop conditions and evaluation units that actually matter when stabilizing AI output.
When Using OpenAI-Compatible APIs in Flue, Check the Model Specifier First
A note on getting stuck with 'Unknown model specifier' in Flue 1.0 Beta by mixing up the actual model ID and the provider-id/model-id format.
Before Growing Hermes Agent: Creating Synthetic Support Triage Scenarios
As a preparatory step before delegating support triage to Hermes Agent, I built three evaluation scenarios using synthetic data—fixing decision criteria and safety constraints in advance without relying on real customer data.
Observing the Sakana Fugu API with Langfuse: Understanding Hidden Costs in Multi-Agent Systems
A hands-on report instrumenting Sakana Fugu's OpenAI-compatible API with Langfuse, measuring how latency, token consumption, and TTFT change across Level 1–3 tasks.
Observing the Black Box of a Multi-Agent API with Sakana Fugu and Langfuse
I subscribed to Sakana Fugu to understand its nature as an OpenAI-compatible API and to plan how to observe its black-box cooperative reasoning from the outside.
Eve's TUI vs HTTP Event Streams: A Side-by-Side Look at Tool Calling
A hands-on log comparing how the same weather tool call looks in Vercel's Eve agent framework when observed from the TUI versus the HTTP API, separating the developer-friendly display from the integration-friendly event stream.
Streaming Flue Observe Events to Langfuse: Monitoring an Issue Triage Agent
An experiment log where I redact Flue 1.0 Beta observe events before sending them to Langfuse, tracking the issue triage workflow's runId, model, and results.
Observing Eve TUI Execution and Tool Calls with Langfuse
A hands-on comparison of two ways to send Eve tool-calling executions to Langfuse as trace/span/generation data.