Working with a tool like OpenClaw can sometimes feel like a black box, with many moving parts that make it hard to understand what is happening under the hood. However, OpenClaw’s Hook functionality changes that by giving you clear visibility into your agent’s behavior.

OpenClaw Hooks let you monitor events as they occur in real time, such as when a message is received or a tool call is made, providing detailed insight into your agent’s execution flow. When combined with an LLM observability tool, they make it easier to debug issues, understand decision-making, and optimize performance.

In this article, we’ll break down how Hooks work and then explore how to integrate them with Arize for monitoring and improving your agent.

Get your agent to set up the code for this article using the Agent’s skill

Get Skill

Hooks in OpenClaw

Within OpenClaw, whenever an action occurs, an event is fired to signal that the action has taken place. For example, when a user sends a message, an event is triggered to indicate this interaction. Similarly, when the agent responds or uses a tool, corresponding events are fired.

OpenClaw Hooks allow you to attach additional logic to these events as they occur. In essence, hooks enable you to tap into OpenClaw’s event-driven system, giving you the ability to customize, and extend the system’s behavior.

OpenClaw hooks can be divided into the following categories:

• Agent lifecycle: Hooks triggered during the agent’s execution.

• Message flow: Hooks related to messages entering and leaving the system.

• Tool execution: Hooks triggered before and after tool execution.

• Subagent coordination: Hooks triggered when subagents are invoked.

• Gateway lifecycle: Hooks triggered during the OpenClaw gateway’s operation.

With all these hooks at our disposal, we gain visibility into a system that would otherwise be a black box.

The need for Observability in Agents

Observability is essential for understanding how an AI agent or LLM behaves and why it performs tasks in a certain way. Without it, you’re effectively running your agent in isolation, constantly guessing what went wrong.

OpenClaw is no exception. In fact, it requires strong observability for a few key reasons:

• Sensitive operations: It often runs directly on a user’s machine and may handle sensitive tasks, making visibility critical.

• Autonomous behavior: It can operate proactively through features like heartbeats or cron jobs, executing tasks without direct user input.

• System complexity: A single interaction may involve multiple tool calls and internal steps, making it harder to track what’s happening under the hood.

To address these challenges, observability becomes a necessity. Observing a system like OpenClaw requires a dedicated observability tool, and this is where Arize comes in.

Arize is a leading platform for LLM and agent observability, enabling developers to instrument their applications and capture every input and output flowing through the system.

This captured data is then sent to Arize using the OpenTelemetry protocol, where it can be monitored and analyzed in detail.

Instrumentation is relatively straightforward when building applications from scratch, as Arize provides integrations for many LLM and agent frameworks. However, it becomes more challenging when working with an existing system like OpenClaw.

Fortunately, OpenClaw’s hooks provide a solution. By leveraging hooks, we can instrument the agent and gain the visibility needed to properly monitor and evaluate its behavior.

Need creative, high-quality technical content? Happy to chat! Book a call with our Creative Engineers👇🏾

Let’s Chat! 📞

Creating Hooks in OpenClaw

OpenClaw provides two ways to create hooks: directly within the project or through plugins.

The direct method involves creating a hook directory containing two key files:

• HOOK.md – defines the hook’s metadata and documentation.

• handler.ts – implements the hook’s logic.

The second method is to create a plugin that utilizes hooks. In this article, we will focus on plugin hooks because they offer greater flexibility and are much easier to distribute and reuse.

To learn more about creating hooks directly, refer to the OpenClaw documentation.

Building a Simple Plugin Hook

Let’s create a simple plugin that leverages hooks.

1. Initialize the Project

Start by creating a TypeScript npm project named arize_openclaw_hook. Then install OpenClaw as a dependency:

npm install openclaw

Ensure that OpenClaw is already installed and properly configured on your system or in the environment where it is running.

Below is an example of what your package.json might look like:

{
  "name": "arize_openclaw_hook",
  "version": "1.0.0",
  "description": "",
  "main": "index.ts",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "openclaw": {
    "extensions": [
      "./index.ts"
    ],
    "compat": {
      "pluginApi": ">=2026.3.24-beta.2",
      "minGatewayVersion": "2026.3.24-beta.2"
    },
    "build": {
      "openclawVersion": "2026.3.24-beta.2",
      "pluginSdkVersion": "2026.3.24-beta.2"
    }
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "type": "module",
  "dependencies": {
    "openclaw": "^2026.3.28"
  }
}

We’ve also included additional metadata such as the OpenClaw version and Plugin SDK version.

2. Create the Plugin Manifest

All OpenClaw plugins require a manifest file named Openclaw.plugin.json, which contains metadata about the plugin:

{
  "id": "arize_openclaw_hook",
  "name": "Arize OpenClaw Hook",
  "description": "Observability plugin for OpenClaw with Arize.",
  "configSchema": {
    "type": "object",
    "additionalProperties": false,
  },
}

Note: The plugin id should match the name of the npm project.

3. Define the Plugin Entry

Next, create an index.ts file to define the plugin:

import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";

export default definePluginEntry({
    id: "arize_openclaw_hook",
    name: "Arize OpenClaw Hook",
    description: "Observability plugin for OpenClaw with Arize.",
    register(api) {
        api.registerService(arizeService(api));
    },
});

The definePluginEntry function accepts an object with the plugin’s metadata and a register method. This method is used to register the plugin’s functionality with OpenClaw.

4. Implement the Plugin Service

We will register our plugin as a background service using api.registerService. A service must return an OpenClawPluginService object:

export type OpenClawPluginService = {
    id: string;
    start: (ctx: OpenClawPluginServiceContext) => void | Promise<void>;
    stop?: (ctx: OpenClawPluginServiceContext) => void | Promise<void>;
};

• id: Uniquely identifies the service.

• start: Executed when the plugin starts.

• stop: Optional method executed during shutdown for cleanup.

Here is a minimal implementation:

function arizeService(api: OpenClawPluginApi): OpenClawPluginService {
    
    api.logger.info("Initializing Arize OpenClaw Hook service...");
    
    return {
        id: "arize-openclaw",
        async start(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };

            log.info("Arize OpenClaw Hook service is starting...");
        },
        async stop(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };

            log.info("Arize OpenClaw Hook service is stopping...");
        }
    };
}

This service simply logs messages using the OpenClaw logger. The api.logger is available during registration, while ctx.logger is used within the start and stop lifecycle methods.

5. Adding Hooks to the Service

Next, we attach hooks to listen for specific OpenClaw events:

• llm_input – Triggered when the LLM receives a prompt.

• llm_output – Triggered when the LLM generates a response.

• before_tool_call – Triggered before the agent invokes a tool.

import { definePluginEntry,OpenClawPluginApi,OpenClawPluginService, OpenClawPluginServiceContext} from "openclaw/plugin-sdk/plugin-entry";


function arizeService(api: OpenClawPluginApi): OpenClawPluginService {

    api.logger.info("Initializing Arize OpenClaw Hook service...");

    api.on("llm_input", (event) => {
       api.logger.info(`Prompt: ${event.prompt}`);
       api.logger.info(`Model name: ${event.model}`);
       api.logger.info(`Model provide: ${event.provider}`);
    });

    api.on("llm_output", (event) => {
        api.logger.info(`Assistant Texts: ${event.assistantTexts.toString()}`);
    });

    api.on("before_tool_call", (event) => {
        api.logger.info(`Tool Name: ${event.toolName}`);
        api.logger.info(`Tool Call ID: ${event.toolCallId}`);
    });

    return {
        id: "arize-openclaw",
        async start(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };
            log.info("Arize OpenClaw Hook service is starting...");
        },
        async stop(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };
            log.info("Arize OpenClaw Hook service is stopping...");
        }
    };
}

export default definePluginEntry({
    id: "arize_openclaw_hook",
    name: "Arize OpenClaw Hook",
    description: "Observability plugin for OpenClaw with Arize.",
    register(api) {
        api.registerService(arizeService(api));
    },
});

Event Data Overview

• llm_input: Contains the prompt, model name, and provider.

• llm_output: Includes the assistant’s generated response.

• before_tool_call: Provides details about the tool being invoked, such as its name and call ID.

6. Installing and Testing the Plugin

Once the implementation is complete, install the plugin locally:

openclaw plugins install .

After installation, restart the OpenClaw gateway to activate the plugin:

openclaw gateway restart

Next, we need to open the OpenClaw dashboard to communicate with the agent. We can do this using the following command:

openclaw dashboard

Once the dashboard is open, you can send a message to the agent from the Chat section in the sidebar.

Next, go to Logs in the sidebar and verify that the data was logged.

From the image above, we can see that the data was successfully logged, confirming that our plugin works as expected. Next, we need to add instrumentation so that instead of being logged locally, the data is sent to the Arize server.

Hooks + Observervability = Better Agent

In this section, we’ll extend the plugin from the previous section and add instrumentation so we can observe the agent’s behavior in Arize instead of relying only on local logs.

Installing Instrumentation Dependencies

First, install the required OpenTelemetry and Arize instrumentation packages:

npm install @opentelemetry/api @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/semantic-conventions @arizeai/openinference-semantic-conventions openai

Configuring Arize Credentials

Next, we need to configure the credentials required to send traces to Arize:

• ARIZE_API_KEY: Your Arize API key

• ARIZE_SPACE_ID: Your Arize space ID

• ARIZE_PROJECT_NAME: The name of your project

Normally, these would be stored in an environment file. However, since we are working within OpenClaw, the better approach is to expose them as configurable plugin values via the manifest file.

We update Openclaw.plugin.json to include these configuration values:

{
  "id": "arize_openclaw_hook",
  "name": "Arize OpenClaw Hook",
  "description": "Observability plugin for OpenClaw with Arize.",
  "configSchema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "spaceID": {
        "type": "string"
      },
      "apiKey": {
        "type": "string"
      },
      "projectName": {
        "type": "string"
      }
    }
  },
  "uiHints": {
    "apiKey": {
      "label": "Arize API Key",
      "placeholder": "your-api-key",
      "sensitive": true
    },
    "spaceID": {
      "label": "Arize Space ID",
      "placeholder": "your-space-id",
      "help": "The ID of the Arize space to use."
    },
    "projectName": {
      "label": "Project Name",
      "placeholder": "openclaw"
    }
  }
}

These values can now be set either in openclaw.json or directly from the dashboard thanks to the UI hints defined above.

Adding Instrumentation

Next, we implement instrumentation to connect OpenClaw to Arize. This involves initializing an OpenTelemetry tracer that sends spans to the Arize server.

function arizeService(api: OpenClawPluginApi): OpenClawPluginService {
   
    const COLLECTOR_ENDPOINT = "https://otlp.arize.com";
    const SERVICE_NAME = process.env.ARIZE_PROJECT_NAME || "";
    
    api.logger.info(`Initializing Arize OpenClaw Hook with service name: ${SERVICE_NAME}`);
    
    const provider = new NodeTracerProvider({
        resource: resourceFromAttributes({
            [ATTR_SERVICE_NAME]: SERVICE_NAME,
            [SEMRESATTRS_PROJECT_NAME]: SERVICE_NAME,
        }),
        spanProcessors: [
            new SimpleSpanProcessor(
                new OTLPTraceExporter({
                    url: `${COLLECTOR_ENDPOINT}/v1/traces`,
                    headers: {
                        'space_id': process.env.ARIZE_SPACE_ID || "",
                        'api_key': process.env.ARIZE_API_KEY || "",
                    },
                })
            ),
        ],
    });

    provider.register();

    const tracer = trace.getTracer(SERVICE_NAME); 
    
    return {
        id: "arize-openclaw",
        async start(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };

            log.info("Arize OpenClaw Hook service is starting...");
        },
        async stop(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };
            log.info("Arize OpenClaw Hook service is stopping...");
        }
    };
}

At this point, we have instrumentation configured, but no hooks attached yet.

Connecting Hooks to Traces

In the code below, we added our hooks as before, but this time we use the tracer to create spans that instrument all calls and send them to Arize.

function arizeService(api: OpenClawPluginApi): OpenClawPluginService {
    
    // Remaining code above
    
    api.on("llm_input", (event) => {
        tracer.startActiveSpan("call-llm", async (span) => {
            span.setAttribute(SemanticConventions.OPENINFERENCE_SPAN_KIND, OpenInferenceSpanKind.CHAIN);
            span.setAttribute(INPUT_VALUE, event.prompt);
            span.setAttribute(LLM_MODEL_NAME, event.model);
            span.setAttribute(LLM_PROVIDER, event.provider);


            api.on("llm_output", (event) => {
                span.setAttribute(OUTPUT_VALUE, event.assistantTexts.toString());
                span.setStatus({ code: SpanStatusCode.OK });
                span.end();
            });
        });
    });

    api.on("before_tool_call", (event) => {
        const span = tracer.startSpan("tool-call");
        span.setAttribute(SemanticConventions.OPENINFERENCE_SPAN_KIND, OpenInferenceSpanKind.TOOL);
        span.setAttribute(TOOL_NAME, event.toolName);
        span.setAttribute(TOOL_CALL_ID, event.toolCallId || "unknown");
        span.setStatus({ code: SpanStatusCode.OK });
        span.end();
    });
    
    // Remaining code below
}

Here is the full implementation:

import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
import type { OpenClawPluginApi, OpenClawPluginService , OpenClawPluginServiceContext } from "openclaw/plugin-sdk";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import { SEMRESATTRS_PROJECT_NAME } from "@arizeai/openinference-semantic-conventions";


import { trace, SpanStatusCode } from "@opentelemetry/api";
import {
    INPUT_VALUE, OUTPUT_VALUE, LLM_MODEL_NAME, LLM_PROVIDER, SemanticConventions, TOOL_NAME, TOOL_PARAMETERS, TOOL_CALL_ID,
    OpenInferenceSpanKind
} from "@arizeai/openinference-semantic-conventions";


function arizeService(api: OpenClawPluginApi): OpenClawPluginService {

    const COLLECTOR_ENDPOINT = "https://otlp.arize.com";
    const SERVICE_NAME = process.env.ARIZE_PROJECT_NAME || "";
    
    api.logger.info(`Initializing Arize OpenClaw Hook with service name: ${SERVICE_NAME}`);
    
    const provider = new NodeTracerProvider({
        resource: resourceFromAttributes({
            [ATTR_SERVICE_NAME]: SERVICE_NAME,
            [SEMRESATTRS_PROJECT_NAME]: SERVICE_NAME,
        }),
        spanProcessors: [
            new SimpleSpanProcessor(
                new OTLPTraceExporter({
                    url: `${COLLECTOR_ENDPOINT}/v1/traces`,
                    headers: {
                        'space_id': process.env.ARIZE_SPACE_ID || "",
                        'api_key': process.env.ARIZE_API_KEY || "",
                    },
                })
            ),
        ],
    });

    provider.register();

    const tracer = trace.getTracer(SERVICE_NAME);


    api.on("llm_input", (event) => {
        tracer.startActiveSpan("call-llm", async (span) => {
            span.setAttribute(SemanticConventions.OPENINFERENCE_SPAN_KIND, OpenInferenceSpanKind.CHAIN);
            span.setAttribute(INPUT_VALUE, event.prompt);
            span.setAttribute(LLM_MODEL_NAME, event.model);
            span.setAttribute(LLM_PROVIDER, event.provider);


            api.on("llm_output", (event) => {
                span.setAttribute(OUTPUT_VALUE, event.assistantTexts.toString());
                span.setStatus({ code: SpanStatusCode.OK });
                span.end();
            });
        });
    });

    api.on("before_tool_call", (event) => {
        const span = tracer.startSpan("tool-call");
        span.setAttribute(SemanticConventions.OPENINFERENCE_SPAN_KIND, OpenInferenceSpanKind.TOOL);
        span.setAttribute(TOOL_NAME, event.toolName);
        span.setAttribute(TOOL_CALL_ID, event.toolCallId || "unknown");
        span.setStatus({ code: SpanStatusCode.OK });
        span.end();
    });


    return {
        id: "arize-openclaw",
        async start(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };

            log.info("Arize OpenClaw Hook service is starting...");
        },
        async stop(ctx: OpenClawPluginServiceContext) {
            const log = { info: ctx.logger.info.bind(ctx.logger) };
            log.info("Arize OpenClaw Hook service is stopping...");
        }
    };
}

export default definePluginEntry({
    id: "arize_openclaw_hook",
    name: "Arize OpenClaw Hook",
    description: "Observability plugin for OpenClaw with Arize.",
    register(api) {
        api.registerService(arizeService(api));
    },
});

Testing the Plugin

Now that we have observability in place, let’s test the plugin again.

In the OpenClaw dashboard, I created a new chat session. When a session starts, OpenClaw loads memory and other contextual data using internal tools.

Switching to the Arize dashboard, we can see that all tool calls and the new session events are being sent successfully.

If we inspect the session creation event, we can see:

• Input message captured

• Output recorded

• Additional metadata such as latency and execution details

You can get the full source code implementation here on github.

Join the conversation and share your experiences in the comments below!

Leave a comment

Conclusion

OpenClaw’s hooks are a powerful feature that let you tap directly into the system’s internal events. By wrapping hooks inside a plugin, you can fully customize your OpenClaw agent and extend its capabilities in meaningful ways.

In this article, we used that flexibility to add observability to our agent using Arize. Observability is critical when working with systems like OpenClaw, where agents are complex, autonomous, and often operate behind multiple layers of abstraction.

With the right instrumentation in place, you move from guessing what your agent is doing to clearly understanding its behavior in real time.

OpenClaw went viral this year because of its simplicity in allowing users to communicate with AI agents running on their own hardware. It wasn’t a new model, a new architecture, or a new agentic protocol. Instead, it demonstrated a new way people could work with AI agents using technology they were already familiar with.

OpenClaw allows users to communicate with their agents through chat apps such as Telegram and WhatsApp. A user can simply pick up their device and send a text message to the AI agent. This ease of use is what attracted many users to OpenClaw.

But can this interaction be simplified even further? Yes, it can. One way to do this is by turning your OpenClaw agent into a voice AI agent. Since OpenClaw already works through chat apps, and most of these apps support voice notes, voice interaction becomes a natural extension of the system.

In this article, we will show you how to set up OpenClaw as a voice AI agent. We will also demonstrate how to bring your own speech-to-text model and integrate it with OpenClaw. The model we will use is Universal-3 Pro, and we will explore how its prompting capabilities can be used to create a more customized voice interaction experience.

OpenClaw in a nutshell

There is a lot of confusion surrounding OpenClaw. Even the name itself creates confusion. So in this section, we will do a quick breakdown of what OpenClaw is and how it works.

What is OpenClaw?

You can think of OpenClaw as a gateway between your chat app and your AI agent.

The chat app can be Telegram, WhatsApp, or Slack. The AI agent can be powered by cloud-based Large Language Models (LLMs) such as those provided by Anthropic or OpenAI, or even by a locally hosted model.

The AI agent also has access to a computer system. This could be your personal computer (though this is not advisable), a Mac Mini, a Raspberry Pi, or a cloud server.

The OpenClaw setup consists of the following:

• The chat application serves as the user interface

• OpenClaw acts as the orchestrator

• The AI agent is the brain

• The computer serves as the universal tool

What makes openclaw different?

Agents are not a new concept. Agents that have access to a computer are not new either, and chatting with AI is certainly not new.

However, two things make OpenClaw stand out.

The first is the medium of communication. Unlike many chatbots that require you to use a separate app or a dedicated website, OpenClaw allows you to communicate with your agent through the chat apps you already use.

The second difference is that the OpenClaw agent is more proactive. It is not just another chat session. The agent can maintain memory, send reminders about tasks it is working on, and interact with the computer it has access to.

Since the agent has access to the system, it can perform actions such as reading files, editing files, and running commands. In many ways, OpenClaw feels like giving a personal computer to an AI assistant.

Setting up openclaw

When it comes to installing openclaw they are several different options you can choose. The easiest option is to install it on your personal computer but this is not advisable since an AI agent will have full control over your computer and a lot security experts have warned several security vulnerabilities on openclaw.

But running on your personal computer is the fastest way to experiment with it. The other option is have a dedicated computer for openclaw such as mac mini or a raspberry pi. Another option is to run openclaw within container using docker that way it is sandboxed.

If you’re on Mac or Linux, you can install OpenClaw with this one-liner:

curl -fsSL https://openclaw.ai/install.sh | bash

Then set it up by running:

openclaw onboard --install-daemon

This will prompt you to configure your model. The --install-daemon flag sets up OpenClaw as a background service, so it runs automatically whenever your device starts.

Once setup is complete, you can confirm everything is running with:

openclaw gateway status

For other installation methods, refer to the official OpenClaw installation guide.

Setting up a channel for communication

When OpenClaw is installed, the next thing you need to do is set up the channel of communication. This is essentially the chat app you wish to use to communicate with OpenClaw from.

All channels support communication via text, but since we are working with a voice agent, we want one that can support other media types, such as audio. Telegram is perfect for this because it offers the easiest setup compared to other channels, and you can send voice notes to the openclaw via Telegram.

Go through the setup guide on Telegram on openclaw documentation.

OpenClaw’s media understanding capabilities

OpenClaw’s media understanding capabilities allow it to process more than just text. When it receives a media file, such as an image or audio, it can use one of its model providers to transform it into a format the agent can understand.

For example, if OpenClaw receives a voice note from a channel like Telegram, it will use a speech-to-text (STT) model to convert the audio into text before passing it to the LLM. Similarly, if it receives an image, it can summarize the content using an image model and send that information to the agent. In this article, we are focusing on the audio understanding aspect of OpenClaw.

By default, OpenClaw supports a limited set of STT providers, including OpenAI, Mixtral Voxtral, and Deepgram. In this guide, we’ll go a step further by integrating a custom STT model, extending OpenClaw beyond its built-in options.

Bring your own Speech To Text model

There are several ways to extend OpenClaw’s capabilities, one of which is via plugins. While writing a plugin to perform media understanding is possible, it is often overkill. OpenClaw already provides a built-in way to extend media understanding using a custom script.

With a custom script, you simply tell OpenClaw that whenever it receives an audio file, it should run the script. The script processes the audio and returns the transcribed text. All the heavy lifting is handled by OpenClaw. You just need to write the script and configure the openclaw.json.

Since we get to write the script, we can choose any STT model provider. In this guide, we will use AssemblyAI.

Step 1: Set Up Your Environment

It is best to create a dedicated Python environment first. Then, install the AssemblyAI SDK:

pip install assemblyai

Next, create an AssemblyAI API key and store it in an environment variable:

export ASSEMBLYAI_API_KEY="your_api_key_here"

For global access, it is recommended to add this line to your .bashrc or .zshrc file.

Step 2: Create the Transcription Script

Create a Python file called main.py and add the following:

import argparse
import os
import sys
import assemblyai as aai


def main():
    # 1. Set up the argument parser
    parser = argparse.ArgumentParser(
        description="Transcribe an audio file using AssemblyAI."
    )
    
    # Add positional argument for the audio file path
    parser.add_argument(
        "audio_file", 
        type=str, 
        help="Path to the audio file you want to transcribe (e.g., ./voice_note.ogg)"
    )
    
    # Add optional argument for the API key
    parser.add_argument(
        "--api-key", 
        type=str, 
        help="Your AssemblyAI API key (can also be set via ASSEMBLYAI_API_KEY env variable)",
        default=None
    )

    args = parser.parse_args()

    # 2. Configure API Key
    api_key = args.api_key or os.environ.get("ASSEMBLYAI_API_KEY")
    if not api_key:
        print("Error: API key is missing.")
        print("Please set the ASSEMBLYAI_API_KEY environment variable or pass it via --api-key.")
        sys.exit(1)
        
    aai.settings.api_key = api_key

    # 3. Configure and run the transcription
    config = aai.TranscriptionConfig(speech_models=["universal-3-pro"], 
                                     language_detection=True, 
                                     prompt="Transcribe the audio make sure include fillers and stutters in the transcript.")

    print(f"Transcribing '{args.audio_file}'... Please wait.")

    try:
        transcript = aai.Transcriber(config=config).transcribe(args.audio_file)

        if transcript.status == "error":
            raise RuntimeError(f"Transcription failed: {transcript.error}")

        print("\n--- Transcript ---")
        print(transcript.text)
        print("------------------\n")


    except Exception as e:
        print(f"\nAn error occurred: {e}")
        sys.exit(1)

if __name__ == "__main__":
    main()

This script takes the path to an audio file, transcribes it using AssemblyAI, and prints the result.

Step 3: Configure OpenClaw

Next, integrate the script with OpenClaw by editing openclaw.json:

  "tools":{
    "media": {
      "audio": {
        "enabled": true,
        "models": [
          {
            "type": "cli",
            "command": "python",
            "args": ["/PATH/TO/SCRIPT/main.py", "{{MediaPath}}"]
          }
        ]
      }
    }
  }

This configuration tells OpenClaw to enable audio understanding, but instead of using a default provider, it will run your custom script.

Tip: If you are using a Python virtual environment, set the command to the full path of your environment’s Python binary. You can find it with:

whereis python

Step 4: Restart OpenClaw

After setup, restart OpenClaw to apply the changes:

openclaw daemon restart

With this setup, you now have full control over OpenClaw’s audio understanding capabilities.

You can find the complete implementation in the GitHub repository.

Why Use Universal-3 Pro with OpenClaw?

A common question when working with OpenClaw’s media capabilities is: why switch to a different STT model? After all, don’t all STT models just convert speech to text?

The answer is no. Different STT models have different strengths and trade-offs, for example:

• Speed: Some models prioritize fast transcription, making them suitable for real-time applications.

• Accuracy (WER): Others focus on achieving a low Word Error Rate, improving transcription quality.

• Domain specialization: Certain models are optimized for specific areas such as medicine, legal, or customer support.

• Customization: Some models allow fine-tuning or prompting to handle unique names, jargon, or phrases.

• Deployment preference: Developers may prefer local models for privacy, control, or cost reasons.

This flexibility allows you to create a voice AI agent that is tailored to your specific use case, rather than relying solely on OpenClaw’s built-in media understanding.

In this article, we use AssemblyAI’s Universal-3 Pro because of its powerful prompting capabilities. For example, my name is Eteimorde. It is not an English name and rarely appears in standard datasets.

While building my personal voice AI agent with OpenClaw, I noticed that default STT models consistently misheard my name. To solve this, I used Universal-3 Pro’s keyterm prompting feature to explicitly define my name as an important term:

config = aai.TranscriptionConfig(
    speech_models=["universal-3-pro"], 
    language_detection=True, 
    keyterms_prompt=["Eteimorde"]
)

Additional Capabilities of Universal-3 Pro via Prompting

Universal-3 Pro provides advanced features that can be easily leveraged through prompting. You can customize the behavior of the model by updating the prompt in the transcription configuration:

config = aai.TranscriptionConfig(
    speech_models=["universal-3-pro"], 
    language_detection=True, 
    prompt="YOUR_PROMPT_GOES_HERE"
)

Using prompting, the model can perform the following tasks:

1. Verbatim transcription and disfluencies
   Preserve natural speech patterns such as filler words, repetitions, and self-corrections.

2. Audio event tagging
   Mark non-speech sounds like laughter, music, applause, or background noise.

3. Crosstalk labeling
   Identify overlapping speech, interruptions, and multiple speakers talking at once.

4. Numbers and measurements formatting
   Control how numbers, percentages, and measurements are represented.

5. Context-aware clues
   Improve transcription for domain-specific terms, names, and jargon by providing relevant hints in the prompt.

6. Speaker attribution
   Detect and label different speakers in a conversation.

7. PII redaction
   Tag personal identifiable information such as names, addresses, and contact details, useful for limiting what the agent can access.

By using prompting, these capabilities allow your OpenClaw voice agent to become more accurate, context-aware, and personalized, going beyond the default transcription behavior.

Conclusion

OpenClaw makes it easy to run AI agents through chat apps you already use, and adding voice capabilities takes the interaction to a whole new level. By integrating your own speech-to-text models, such as Universal-3 Pro, you unlock features beyond OpenClaw’s built-in media understanding.

Its prompting capabilities allow users to customize how the model transcribes audio, accurately recognize custom keyterms, and leverage features like verbatim transcription to preserve natural speech and audio event tagging to capture non-speech context such as background noise or laughter.

With this setup, your OpenClaw agent behaves more like a true personal assistant. It can remember context, send proactive reminders, and leverage system tools to perform tasks. Voice interaction, combined with Universal-3 Pro’s advanced prompting features, transforms the agent from a simple chat companion into a more robust, seamless, and highly personalized experience.

Instead of relying on free-form text, would it be better to structure the input before passing it to the model?

That’s where JSON prompting comes in. With JSON prompting, all prompts provided to the image generation model are structured. So instead of writing:

“A person walking down the beach during the sunset”

You would format the prompt like this:

{ "subject": "person", 
  "action": "walking along the shore", 
  "environment": "beach", 
  "time_of_day": "sunset" 
}

Although JSON prompting is gaining prominence in the image generation space, it does come with drawbacks. One of the main challenges is complexity. Outside the developer community, relatively few people are comfortable working directly with JSON. It introduces structure, but it also introduces friction.

However, there is one structured format that almost everyone is familiar with: forms.

In this article, we take JSON prompting one step further by showing how users can literally form their images. Instead of writing raw JSON, users interact with structured inputs behind the scenes. By the end, you will see whether structured prompting is truly necessary and whether it is something you should incorporate into your own image generation workflows.

Structured vs Unstructured Text in Image Generation

One of the major breakthroughs in image generation came when models gained the ability to generate images directly from text prompts. That moment marked the real beginning of modern text-to-image AI, bringing natural language understanding into the visual domain.

Today’s image generation models, such as Nano Banana, can produce accurate images even when prompts contain misspellings or loosely written descriptions. Within this space, prompting approaches can generally be divided into two categories:

• Unstructured Text for Image Generation

• Structured Text for Image Generation

Unstructured Text for Image Generation

Unstructured text is the natural way humans communicate. When we express intent, we do not usually follow a rigid format. We describe things as they come to mind. This is one of the main reasons it remains the default method for working with image generation models. Anyone, regardless of technical background, can type a sentence and generate an image.

Unstructured text is not only natural for humans, it is also the default format used during model training. Image generation models are typically trained on large datasets consisting of image–caption pairs. Most captions found across the internet are written in free-form, unstructured language.

Because of this, many image generation models perform exceptionally well with unstructured text. That is the format they were primarily trained on. The majority of large-scale datasets contain images paired with natural language descriptions, not structured schemas.

Structured Text for Image Generation

While unstructured text is natural for both humans and machines, structured text has been gaining prominence in the image generation space. This has to do with the fact that most image generation models are paired with a language model. This enables them to possess native image generation capabilities.

How this works in theory is that a language model is first given a prompt. The prompt is then converted into image tokens that will be used to generate the image. This means that image generation models carry the world knowledge of LLMs and use it to generate images.

Since LLMs have a strong understanding of structured formats like JSON, they can easily interpret intent when generating images, regardless of the format in which the prompt is provided. This is where JSON prompting comes in.

By structuring prompts in formats such as JSON, we can guide the model more explicitly while still leveraging the native image generation capabilities powered by the underlying language model.

Researchers are also exploring the use of structured captions paired with images to further improve the generation capabilities of open-source image models. For example, the paper Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions introduced FIBO, a model trained on structured text.

Why JSON Prompting?

JSON prompting can sound like extra work. Instead of writing a simple sentence, you now have to think in terms of fields, attributes, and structure. So why bother?

Because it offers several practical advantages:

• Detailed image creation

• Image consistency

• Image recreation

Detailed Image Creation

JSON prompting makes it possible to generate highly detailed images in a controlled way. Instead of relying on a single descriptive paragraph, you can explicitly define elements such as:

• Subject

• Lighting

• Camera angle

• Style

• Environment

• Mood

• Composition

By structuring these components, you reduce ambiguity. Free-form text can be vague and often relies on unstated assumptions. Structured prompts force clarity. Each aspect of the image is intentionally defined, which makes it easier to produce rich and precise outputs.

Image Consistency

With JSON prompting, maintaining consistency across multiple generated images becomes much easier. Because the structure is explicit and reusable, you can:

• Keep the same character traits across generations

• Maintain consistent lighting or camera settings

• Reuse stylistic attributes

• Change only specific fields, such as the environment or time of day

For example, you can keep a character’s appearance identical while swapping out different scenes. Or you can maintain the same scene while experimenting with different lighting conditions. The structure gives you control over what changes and what stays the same.

This consistency can also extend across models that support JSON prompting. A well-defined structured prompt can be reused with different image generation systems, helping you achieve similar outputs without rewriting everything from scratch.

Image Recreation

JSON prompting also enables image recreation. Given an existing image, a vision model can generate a detailed JSON description of that image, capturing elements such as:

• Objects in the scene

• Their positions

• Lighting conditions

• Camera perspective

• Style and composition

This structured representation preserves the essential characteristics of the original image.

The generated JSON prompt can then be passed to an image generation model to reproduce the image with a similar level of detail and structure. Instead of loosely describing what you see, you are reconstructing it in a format that is both machine-readable and reusable.

Does JSON Prompting actually work

One question arises with JSON prompting: does it actually work?

Given that most image generation models are trained on plain text, is JSON prompting really necessary?

Critics of JSON prompting have raised several concerns:

• It is complex

• There is no real difference between JSON prompting and writing a detailed plain-text prompt

• You are adding unnecessary tokens in the form of brackets, quotation marks, and field names

These criticisms are not entirely unreasonable.

Personally, while using JSON prompting, I encountered many of these same issues. I took a structured JSON prompt and converted it into a carefully written detailed paragraph. In many cases, the results were almost identical.

Some argue that JSON prompting creates a kind of placebo effect. And in certain situations, that might be true. But not all the time.

JSON prompting provides something that plain text does not: structure that is easy to manipulate. It allows you to introduce prompt templates in a clean and organized way. For example, if you want to add a specific camera view to your image, you can simply introduce a “camera” field rather than rewriting an entire paragraph.

You can also swap attributes quickly. This ability to make targeted edits is where JSON prompting really stands out. You can search for a specific attribute and modify it without scanning through long blocks of descriptive text.

You can change the subject, lighting, or environment in seconds, with minimal friction.

As for the concern about extra tokens from brackets and quotation marks, that limitation can be mitigated by using alternative structured formats such as YAML, which tends to be more compact while preserving structure.

The biggest issue, however, remains complexity. Writing raw JSON can feel tedious and technical, especially for non-developers. But that problem is not inherent to structured prompting itself. It is an interface problem. And that is where forms come in.

Use Form instead of JSON

Instead of typing JSON manually, users can interact with structured fields through a form-based interface that generates the JSON behind the scenes. This preserves the benefits of structured prompting while removing the friction of writing it by hand.

To explore this idea, I built a simple interface that lets you literally form your images.

Let’s walk through how it works.

First, the interface allows you to create fields that accept text values. These fields represent different attributes of the image, such as subject, environment, style, or mood.

You can also create nested fields. This is important for building prompts with nested JSON objects, where certain attributes belong inside broader categories. For example, a camera object might contain focal length, angle, and depth of field.

There is also a prompt template feature that allows you to load predefined templates, such as camera setups, lighting configurations, environments, and more. A user can select one of these templates and merge it directly into their existing prompt structure instead of rebuilding it from scratch.

When the user clicks Generate, the form data is converted into JSON and sent to the image generation model. If needed, the generated JSON can also be copied and reused in any other image generation interface that supports structured prompting.

Editing is just as simple. Instead of scanning through long paragraphs of text, you can modify a specific field.

For example, in the image below we were able to change the subject from a person to a dog in seconds, without touching the rest of the prompt.

By leveraging this kind of interface, we retain the structure and control of JSON prompting while restoring the simplicity and accessibility that plain-text prompting originally offered.

The full implementation is available on GitHub: https://github.com/Neurl-LLC/FORM

You can clone the repository, run it locally, and adapt it to your own image generation projects.

Conclusion

Structured prompting is a way of organizing your prompts when working with image generation models. While many practitioners prefer using JSON as the structured format, there is still room to experiment with other structured formats to see what works best.

In this article, we used forms as the interface for working with JSON prompts. As we have seen, this approach improves the user experience and makes working with JSON prompting simpler and more practical.

Now the obvious question is: if RAG works so well for text, can we apply the same idea to other parts of the generative AI landscape? One place where this makes a lot of sense is image generation.

Modern image generation models like Nano Banana and GPT-Image are not just good at generating images from text prompts. They can also generate images using one or more reference images provided in-context. The catch is that this process is usually manual. The user has to find the right images, pass them in, and then write a prompt that ties everything together.

But what if we didn’t have to do this manually? That’s where combining retrieval augmented generation with image generation comes in. Several researchers have already explored this idea, with approaches like ImageRAG and Cross-Modal RAG.

In this article, we introduce the concept of Retrieval Augmented Generation for Images. Let’s call it RAGE for short. We walk through how it works, show a working implementation, and explore practical use cases.

Understanding In-Context Learning in Image Generation Models

The paper Language Models are Few-Shot Learners introduced the idea of in-context learning for Large Language Models. In the paper, researchers from OpenAI showed that GPT-3 could learn to perform tasks like translation and question answering simply by seeing a few-shot examples in its context.

In-context learning made it possible to extend a model’s capabilities without fine-tuning.

Fast forward to today, and in-context learning is no longer limited to language models. Image generation models are now capable of it as well.

Modern image generation models can generate images not only from a text prompt, but also from images provided directly in their context.

Given multiple reference images and a text prompt, an image generation model can produce a new image that blends both the prompt and the provided images. This is the model learning in-context.

Image generation models like Nano Banana showed how we can combine several images that are in-context to create blended images, change scenery, and even mix up styles. No fine-tuning is needed for a specific task.

This ability of image generation models to learn from context is what allows us to combine them with RAG to form RAGE.

Automating the Process: Combining RAG with Image Generation

RAG was introduced in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, and it marked a major shift in how we build language-model systems.

The core idea was simple: instead of relying only on a model’s internal knowledge, we retrieve relevant external information and use it directly in-context to improve generation.

At a high level, RAG automates in-context learning for language models and consists of two stages:

• Indexing

• Retrieval

The same idea can be applied to image generation, which leads us to RAGE.

Image Indexing

To apply RAG to images, we first need a way to index them. Each image must be converted into a vector representation.

One approach is to generate a text description of an image using a vision model, then pass that description through an embedding model and store the resulting vector representation in a vector database.

Another option is to use a vision encoder that generates embeddings directly from images, skipping the text step entirely.

This process is repeated for as many images as needed, building an image index that can later be queried.

Image Retrieval and Generation

Once indexing is complete, the system moves to the retrieval and generation stage.

Here, a user provides a text prompt. From that prompt, the system retrieves the images from the vector database that best match the user’s prompt.

These retrieved images, together with the original prompt, are then provided in-context to an image generation model, which uses both the text and visual context to generate a new image.

This end-to-end flow is what defines RAGE: combining retrieval with in-context image generation to automate what would otherwise be a manual process.

Real-World Implementation

Now that we’ve covered the core idea behind RAGE, let’s walk through a working demo application that brings it to life.

The video above shows a fully functional RAGE application. The workflow is simple:

1. The user provides a text prompt.

2. Images that match the prompt are retrieved from the image store.

3. The user selects one or more images from the retrieved results.

4. Once selected, the user clicks Generate to create a new image.

The demo is built using ChromaDB as the vector database and OpenAI GPT-Image as the image generation model.

Each part of the stack is modular, making it easy to swap components and customize the overall experience.

Explore the full implementation on GitHub: RAGE

Use Cases for RAGE

Let’s explore some real-world ways RAGE can be used in practice.

Marketing and Creative Asset Generation

RAGE can be used to generate new marketing images from an existing catalogue of brand assets. Instead of starting from scratch, the system retrieves relevant product images, brand visuals, or campaign materials and uses them as context to generate fresh content.

In the example above, when the user prompted, “A realistic image of a salesman trying to sell lotion and soap to a group of ladies,” the retriever pulled images of the lotion and soap from the image store. These retrieved images were then used in-context to generate a scene where the salesman and the ladies are holding the same lotion and soap. Without those in-context images, the model would have generated completely different products that would not match the brand.

Image Editing and Enhancement

RAGE can also be used to retrieve and edit existing images using natural language prompts. This includes tasks such as changing backgrounds, adjusting scenery, modifying objects, enhancing image quality, or combining multiple images.

Because the model has access to the original images in-context, edits stay grounded in the source material. In the example above, we prompted RAGE with “Combine the image of the lotion and soap,” which retrieved the relevant images and merged them into a single generated output.

Style Transfer and Visual Consistency

By storing different visual styles in the image index, RAGE enables style transfer without any fine-tuning. The system retrieves images that represent a specific aesthetic and applies that style to new generations.

This is especially useful for maintaining visual consistency across illustrations, artwork, or branded assets. In the example above, a scented candle image was generated to match the same style as the retrieved images.

Product and Design Prototyping

Design teams can use RAGE to explore new product ideas by retrieving existing designs, sketches, or reference images. This allows for rapid experimentation while staying aligned with previous designs and constraints.

In the example above, we experimented with a transparent product design by retrieving an incense kit from the image store and generating a transparent version of the kit to visualize how it might look.

Conclusion

RAGE takes the core idea of RAG, retrieving relevant context and using it in generation, and applies it to images. Instead of manually feeding reference images into an image generator, RAGE automates the process by searching a database of visuals and using the most relevant ones to guide generation. This makes image generation more consistent, controllable, and scalable.

The use cases in this article are just a glimpse of what’s possible when Retrieval Augmented Generation is combined with image generation. By connecting an image generation model to an image store, you can unlock entirely new creative possibilities.

• Turn messy PDFs into a clean sections table (H1–H3 chunks with paths) and a topics index (e.g., training, SOC, governance).

• Publish your curated tables as minimal MCP tools (list_whitepapers, sections_by_topic, and search_sections) so any MCP-capable host (or your own client) can query them.

• Keep it small, deterministic, and agent-ready; add vectors/LLM ranking later as an optional search layer.

Try it: Open the demo repo↗️ and point it at a small set of your PDFs first.

Introduction

Most teams eventually inherit a pile of PDF whitepapers (e.g., governance frameworks, training standards, SOC roadmaps) scattered across folders. You know the answers are “in there somewhere”, but searching the raw PDFs is slow, token-hungry, and brittle for agents.

In our new demo, we’ll build a compact, production-shaped pipeline with fenic that:

(1) parses PDFs into structured sections,

(2) indexes headings by topics of interest, and

(3) exposes the curated dataset as three MCP tools.

The goal is to prove the value of curation with simple tools over clean tables.

What you’ll learn:

• How to split Markdown’d PDFs into H1–H3 sections with stable paths.

• How to build a topics index (e.g., training, SOC, governance) from earlier classifications.

• How to persist tables and expose them as MCP tools you can call in a few lines.

• An optional tool to extend with vectors or LLM ranking later.

Open and star 🌟 the demo repo↗️

Why not “just let the agent read my PDFs”?

Short answer: you’ll get inconsistent answers, higher latency/cost, and chaotic control over what the model reads. A curated dataset fixes that:

• Deterministic scope. You decide the rows and columns with no accidental 200-page token floods.

• Human-legible structure. Sections (heading, level, path) are explainable, journalable, and testable.

• Reusability. The same tables support search, analytics, and agents without re-ingesting PDFs.

Agents still have a place, just on top of a small, intentional surface area.

What we’re building

Data products:

• whitepaper_sections, one row per section (H1–H3):
  id, name, heading, level, content, full_path

• whitepaper_topics, one row per (doc, heading, topic):
  doc_id, name, topic, heading

MCP tools:

1. list_whitepapers(): PDF names + section counts

2. sections_by_topic(topic): rows from the curated topic index

3. search_sections(query): simple, case-insensitive substring match

That’s it. Three tools, three mental models, and everything flows from the tables.

Step 1. Normalize the source

We start from a normalized fenic DataFrame (see deduped_docs_pdf_content_final in the demo) with:

• id: numeric or stable id per document

• name: filename/title

• markdown_content: the full body as Markdown (from your PDF-to-MD step)

• optional: toc, page_count, earlier content_categorization output

If you’re evaluating your own content, add a unique source_uri and ingested_at for provenance.

Tip: Keep Markdown as the canonical text. fenic’s Markdown utilities are built for it, and you’ll avoid HTML cleanup.

Step 2. Split Markdown into sections (H1–H3)

The core trick is to chunk each whitepaper by its headings. With fenic’s markdown operator you can extract header-scoped chunks at specific levels, then union them into a single column of section structs:

• heading: the section title

• level: 1, 2, or 3

• content: the body of that section

• full_path: breadcrumb string like H1 > H2 > H3

Why H1–H3? In practice, this is the sweet spot: it preserves structure without exploding into paragraphs. You can always tighten/loosen later.

Quality checks you can print instead of schemas:

• Count of sections per doc.

• Distribution by level.

• A quick sample of full_path strings (valid breadcrumbs).

Step 3. Build a topics index from earlier classification

Earlier in the notebook, you likely produced a content_categorization object per document. Turn those lists (e.g., sections_about_model_training) into a long form “topics” table with one row per (doc_id, topic, heading).

The result: whitepaper_topics can answer “show me every Training heading across the catalog,” using clear group-by and joins.

Step 4. Persist the tables (idempotent)

Save both DataFrames as tables so your MCP tools can reference them by name:

• whitepaper_sections

• whitepaper_topics

Two rules keep things robust for demos and CI:

1. Idempotent writes (e.g., mode=”overwrite” for dev runs).

2. No UDFs inside tool definitions. Keep tool plans to built-ins so they serialize cleanly.

This avoids the common “UDFExpr cannot be serialized” error when saving catalog tools.

Step 5. Publish MCP tools

With the tables saved, register three MCP tools:

1. list_whitepapers()
   Group by id, name, count sections, sort by name.
   Why it matters: gives users a friendly “index” and builds confidence that your catalog is concrete.

2. sections_by_topic(topic)
   Join whitepaper_topics to whitepaper_sections on (doc_id, heading), filter by normalized topic, sort by (name, full_path, level).
   Why it matters: demonstrates your curated taxonomy from Step 3.

3. search_sections(query)
   Case-insensitive substring match on heading or content.
   Why it matters: the simplest possible search. Predictable, explainable, and fast.

   Join the conversation and share your experiences in the comments below!

   Leave a comment

Step 6. Smoke-test in a few lines

Use the simple test client from the notebook:

1. list_whitepapers()

2. sections_by_topic(”training”)

3. search_sections(”latency”)

Each prints a short markdown table (15 rows max) with sensible columns. Done. That’s enough to understand “what’s in the box” and for you to wire the tools into any MCP-capable host or a thin UI.

What “done” looks like

• A printed line like:
  ✅ MCP HTTP server ready at http://127.0.0.1:54217/mcp

• whitepaper_sections with thousands of rows (H1–H3 blocks) and consistent full_paths.

• whitepaper_topics with a handful of topics and the headings they touch.

• Three tools you can demo interactively and wire into an MCP host later.

From here, product and compliance users can ask concrete questions:

“Show me all Governance sections across vendor A and B,”

or

“Where do we talk about training data retention?”

… without opening a single PDF.

Extending the demo (when you’re ready)

These are intentionally out of the core demo but easy to layer on when you need them:

1. Vector ranking (similar_sections)
   Precompute an embedding for a joined_text column (e.g., heading + two newlines + content) with semantic.embed and rank with cosine similarity. This is great for “find me nearby” retrieval, but it’s a bonus. Keep it capped (result_limit) so payloads stay small.

2. LLM scoring (qa_sections_llm)
   If you want the model to “vote” on the best leaf for a question, map a short instruction over a candidate set and parse a numeric score. A common pitfall: ensure your semantic.map instruction is a string literal (the prompt), and if you must use UDFs, don’t embed them inside tool plans; materialize their outputs first.

3. Contact signals.
   Regex UDFs for emails/URLs/phones can be useful for compliance and outreach. Again, do the extraction in the table, but keep the tool definitions UDF-free to avoid serialization errors.

4. Richer taxonomies.

Expand from three topics to your real taxonomy. As it grows, the sections_by_topic join stays the same.

Performance, cost, and safety notes

• Keep rows short: sections (vs. full PDFs) dramatically reduce tokens if you later add LLM steps.

• Cache deliberately: when computing embeddings or heavy transforms, cache the intermediate DF that you’ll reuse.

• Idempotency matters: overwrite in dev; use versioned table names in CI.

• No UDFs in tools: build tool queries from built-in expressions so they serialize cleanly into the catalog.

• Deterministic joins: When you link topics to sections, join on (doc_id, heading) and sort by (name, full_path, level) for tidy, predictable outputs.

How teams can use this

1. Security and compliance search
   Your security team asks, “Where do we promise data retention limits?” instead of skimming PDFs, they call:

   • search_sections(”retention”)

   • If they need curated context: sections_by_topic(”governance”).

2. Audit prep/due diligence
   During vendor reviews, you can export slices to CSV and hand them to legal or partners. The query is always the same: “Someone missed a paragraph on page 63.”

3. Agent surfaces
   Whether you use a CLI host or a UI, the agent doesn’t freestyle answers. It calls deterministic tools and renders results with no hallucinated policy claims.

4. Docs Ops/editorial
   Basic counts (list_whitepapers) often reveal “we have 3 policy docs with 300+ sections; we should consolidate” or “governance sections are thin compared to training. Time to write.”

Adopting this in your stack

1. Point the Colab↗️ at your PDFs (or markdown).

2. Keep the header levels small at first (H1–H3 usually suffices).

3. Save the two tables and wire up the MCP tools.

4. Share the MCP endpoints with your security, docs, and support teams.

5. Iterate on the topics index with real stakeholders (that’s where the value clicks).

6. Only then consider vectors or LLM rankings as optional upgrades (See the Next Steps section of the notebook).

Conclusion and closing thoughts

In this demo, you have taken raw PDFs, imposed a small set of consistent structures, and published them behind three MCP tools. That’s the shape agents, analysts, and engineers can all agree on.

With fenic, the whole pipeline stays in a single, readable DataFrame flow. You don’t use glue code, no batching, and no mystery prompts hiding in strings.

When you’re ready, add vectors or LLM scoring as optional layers. But ship the curated tables first.

Try the demo, then iterate

• Open the demo repo↗️: run the notebook and replace the sample PDFs with your own slice.

• Docs↗️: browse fenic’s Markdown, text, and semantic operators to extend the pipeline.

• Examples↗️: look at other fenic demos for inspiration, then keep your surface area small and focused.

To answer that question, I decided to build a real-time speech-to-text API myself.

At first, I thought it would be straightforward. My assumption was simple: take an open source speech model like Whisper, put it behind a server, and call it a day. I was wrong. Building a real-time speech-to-text API is not just about the speech model.

I ran into challenges I hadn’t anticipated, from audio streaming and chunking to latency, buffering, and handling partial transcripts. Working through these problems forced me to rethink what “real time” actually means in an STT system.

In this article, I’ll show you how I built a real-time transcription API that runs locally using OpenAI’s Whisper. I’ll walk you through the experiments that failed, the ones that worked, and how they eventually came together into a working system.

By the end, you’ll have a clearer understanding of how leading STT providers like Deepgram and AssemblyAI implement real-time transcription and whether building your own solution is worth the trade-off compared to relying on third-party services.

See the full code implementation on GitHub.

What is the fundamental theory for real-time Speech-to-Text (STT)?

In the world of STT models, there are two main approaches to transcription. The first is the prerecorded approach, where a recorded audio file is passed to an STT model to produce a complete transcript. The second is the real-time approach.

Unlike the prerecorded approach, real-time STT is not just about the model itself. It relies on a modular system made up of several coordinated components:

• Collect audio data from an input source

• Convert analog audio into a digital format

• Stream the audio data to a server

• Transcribe the audio and return text in real time using an STT model

Each of these steps introduces its challenges, and together they form the foundation of any real-time transcription system.

Input Sources

The input source is a critical stage in the pipeline, as it captures the audio data used in subsequent steps. Common approaches include:

• Microphone input: The most common scenario for real-time transcription, like seeing captions appear during a meeting as you speak.

• Live streaming from an external audio source: For transcribing live broadcasts, online radio, or other streaming audio in real time.

• Streaming from an audio file: Useful for testing and debugging, as prerecorded audio can be streamed in chunks to simulate a live feed.

Analogue-to-Digital Conversion

Audio naturally exists as an analog signal, which computers and STT models cannot directly process. To work with STT models, we convert the audio into a digital format using Pulse Code Modulation (PCM).

PCM works by sampling the audio at a fixed rate, commonly 16,000 samples per second, allowing the STT model to process the data accurately.

Data Transmission

Once audio is digitized, it needs to be transmitted to the server. In real-time systems, the audio is divided into small chunks, which are sent over a real-time protocol like WebSocket. This chunking ensures low latency and continuous transcription as the audio plays.

Server-Side Processing

On the server, an OpenAI Whisper model receives the audio chunks, transcribes them, and sends the text back to the client. This back-and-forth continues until the audio stream ends, enabling real-time transcription.

Implementing the Client

In this section, we describe what it takes to implement the client for the server. This client serves as the input source described in the theory section. It captures audio, streams it to the server, and receives transcription results in real time. The client is based on Deepgram’s live streaming kit, repurposed to work with our server.

The client depends on two third-party Python libraries:

• websockets, which handles the WebSocket communication

• PyAudio, which captures audio from the microphone

We also need FFmpeg installed on the system. FFmpeg allows us to stream and convert audio from files and URLs into the raw PCM format expected by the server.

Let’s walk through a few key code snippets that are required for the client to function.

The main run() function that will handle connecting to the server and coordinating audio sending and receiving:

async def run(method, source, host):
    print(f”⚪ Connecting to {host}...”)
    
    async with websockets.connect(host, ping_interval=None, ping_timeout=None) as ws:
        print(f”🟢 Connected. Mode: {method.upper()}”)
        
        # Queue to pass audio between threads and async tasks
        audio_queue = asyncio.Queue()
        stop_event = threading.Event()

Inside run(), we first create the sender task. This task takes audio chunks from the queue and sends them to the server:

async def sender():
            while not stop_event.is_set():
                try:
                    data = await asyncio.wait_for(audio_queue.get(), timeout=0.5)
                    await ws.send(data)
                except asyncio.TimeoutError:
                    continue

Then we create the receiver task, which prints out transcriptions from the server in real time:

        async def receiver():
            try:
                async for msg in ws:
                    print(f”📝 {msg}”)
            except websockets.exceptions.ConnectionClosed:
                print(”🔴 Connection closed by server.”)
                stop_event.set()

Now we handle audio sources. First, microphone input using PyAudio:

        def mic_process():
            p = pyaudio.PyAudio()
            
            def callback(in_data, frame_count, time_info, status):
                audio_queue.put_nowait(in_data)
                return (None, pyaudio.paContinue)

            stream = p.open(
                format=pyaudio.paFloat32,
                channels=1,
                rate=SERVER_RATE,
                input=True,
                frames_per_buffer=CHUNK_SIZE // 4,
                stream_callback=callback
            )
            
            print(”🎤 Microphone Active.”)
            stream.start_stream()
            
            while not stop_event.is_set():
                asyncio.run_coroutine_threadsafe(asyncio.sleep(0.1), asyncio.get_event_loop())

            stream.stop_stream()
            stream.close()
            p.terminate()

For URLs, use FFmpeg to convert audio to PCM and stream it:

        def ffmpeg_stream_process(url):
            command = [
                ‘ffmpeg’,
                ‘-i’, url,
                ‘-f’, ‘f32le’,
                ‘-ac’, ‘1’,
                ‘-ar’, str(SERVER_RATE),
                ‘-vn’,
                ‘-loglevel’, ‘error’,
                ‘-’
            ]
            print(f”🎧 Starting FFmpeg stream for: {url}”)
            process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            
            try:
                while not stop_event.is_set():
                    data = process.stdout.read(CHUNK_SIZE)
                    if not data:
                        break
                    audio_queue.put_nowait(data)
            finally:
                process.terminate()
                print(”🛑 FFmpeg stopped.”)

Finally, start the appropriate audio source in a thread and run the sender and receiver tasks:

        if method == “url”:
            source_thread = threading.Thread(target=ffmpeg_stream_process, args=(source,))
        elif method == “mic”:
            source_thread = threading.Thread(target=mic_process)
        else:
            source_thread = threading.Thread(target=ffmpeg_stream_process, args=(source,))
            
        source_thread.start()
        await asyncio.gather(sender(), receiver())
        stop_event.set()
        source_thread.join()

Wrap everything with a simple command-line interface so you can run it with a microphone, file, or URL.

See the full code implementation.

With this, the client is ready! You can now stream audio from your microphone, a file, or a live URL, and receive real-time transcriptions from the server.

My Attempts to Building the Real-time API

I made several attempts to building the real-time API in this section I will show you the approach I took before I eventually got something that worked. I relied on using the client scripts for testing and I simulated a real-time stream by streaming this audio:

Experiment One: The Simple Implementation

For my first attempt at implementing the server, I built a WebSocket server that buffered one second of audio, sent it to Whisper for transcription, and returned the text. Simple, right? In practice, it failed. Whisper took too long to respond, which caused the client connections to time out and disconnect.

Experiment Two: GPT-Assisted Implementation

After the first experiment failed, I tried a different approach. I “vibe-coded” with ChatGPT to make it work: introducing multithreading to handle transcription in the background. This way, the server wouldn’t block while Whisper processed audio, and clients wouldn’t disconnect.

The idea was simple: collect audio continuously, run Whisper in a background thread whenever enough new audio arrived, and send the transcribed text back to the client. A rolling buffer kept the last few seconds for context, and an event system notified the transcription worker when new audio was ready.

This approach fixed the disconnection problem, but it introduced new issues. The transcription accuracy dropped noticeably, and the code became much more complex. It worked, but I realized this wasn’t the “simplicity and reliability” I was looking for.

Experiment Three: The Gemini Approach

After my attempt with ChatGPT didn’t deliver the results I was looking for, I decided to try Gemini 3 Pro to see if it could produce a more efficient real-time STT server. The first thing Gemini pointed out was that I shouldn’t be using OpenAI’s original Whisper implementation at all. Instead, it recommended Faster Whisper, a significantly more optimized version of Whisper designed for low-latency inference.

Gemini’s initial implementation still ran into the same disconnection issues when the model took too long to respond. However, when I suggested introducing threading, Gemini produced a much cleaner and more efficient solution.

This version maintained excellent transcription accuracy while keeping the code far simpler than the previous ChatGPT-assisted implementation.

Switching to Faster Whisper also made a noticeable difference. Compared to the original Whisper implementation, Faster Whisper runs inference more efficiently, reducing latency and making real-time transcription on CPU far more practical.

Together, these two changes, non-blocking transcription and a faster Whisper backend, finally resulted in a stable, accurate, and responsive real-time speech-to-text server.

You can find the complete code for all experiments on GitHub.

Whisper Radio: A real-world implementation

After implementing the API server and arriving at a version that performed reliably, I decided to test it in a real-world scenario. The result was Whisper Radio, a web application that allows users to listen to live radio while seeing real-time transcripts of what is being broadcast.

As users tune into a radio station, the audio stream is sent to our real-time Whisper server. The server transcribes the audio on the fly and streams the text back to the Whisper Radio application, allowing listeners to follow along with the broadcast in text form.

This is just one example of how a real-time speech-to-text API can be applied in practice. The Whisper Radio code lives in the same repository as the rest of the experiments explored in this project.

Further Experiments and Enhancements

There are several ways this real-time speech-to-text service can be extended and improved. Here are a few areas worth exploring:

• Multi-Client Handling: The current implementation was only tested with a single client. Experimenting with multiple simultaneous clients will reveal how the server scales and how performance is affected.

• Voice Activity Detection (VAD): VAD can detect when the user is speaking before sending audio for transcription. While the current Whisper model has basic VAD, enhancing it or moving it to the client side could reduce unnecessary data transmission and improve efficiency.

• Alternative Open-Source Models: Test other speech-to-text models beyond Whisper to compare accuracy, speed, and latency.

• Word Error Rate Monitoring: Use pre-captioned audio to track transcription accuracy and identify areas for improvement.

• Timestamps and Diarization: Add timestamps to transcripts and separate speakers with diarization for more detailed and usable output.

• Voice Agent Development: Extend this system to build a real-time voice assistant capable of responding to commands or queries.

Conclusion

Building a real-time speech-to-text service turned out to be far more challenging than I initially expected. With a bit of experimentation and some guidance from AI, I was able to create a system that works surprisingly well. Beyond just getting it to run, this project offered a glimpse into what happens behind the scenes in large-scale speech-to-text platforms.

The potential improvements, like multi-client support, advanced voice activity detection, diarization, and creating a full voice agent, make the project even more exciting. I’m looking forward to exploring these enhancements and seeing how far this system can be pushed.

This technical guide is a repost from what originally appeared on Typedef AI’s blog as AI Content Pipeline for Search, Recommenders & Agents with fenic

Try it yourself:

For the code and copy-paste-ables, check the Colab and demo repo ⬇️

• Open Colab

• Open Repo

This post focuses on the why and what, so you can decide quickly and click through.

Most teams hit the same wall with their content:

You’ve shipped hundreds of blog posts, tutorials, and case studies over the years. Some of them still convert. Some are outdated. Some overlap so hard they cannibalize each other. And when someone asks a simple question like:

“What are our best beginner tutorials on clustering with code?”

…you open a spreadsheet, search the blog, and start skimming one page at a time.

This guide walks through a different approach.

Instead of manually tagging and skimming, we:

1. Ingest a small corpus of articles (titles, URLs, text snippets).

2. Enrich each row with cheap features like length and “has code.”

3. Use semantic operators to add embeddings and cluster labels.

4. Classify each article’s narrative intent (tutorial, thinkpiece, case study, etc.).

5. Estimate a complexity bucket (beginner/intermediate/advanced).

6. Build a cluster report with exemplars and LLM-generated summaries.

7. Export a clean feature table for BI tools, agents, or downstream ranking.

All of it is built with fenic. An opinionated, PySpark-inspired DataFrame framework designed for AI and agentic applications, with first-class semantic operators like semantic.classify, semantic.extract, and clustering built in.

If you’re responsible for docs, developer education, or marketing content, this is the kind of “content intelligence” you can actually use.

What “done” looks like

By the end of the notebook, you have two main artifacts:

A feature table (features.parquet/features.csv) with one row per article:

url
title
body_clip          # short text snippet for fast semantic work
char_len           # length of full body
clip_len           # length of snippet
has_code           # boolean
complexity_bucket  # e.g. ‘beginner’, ‘practitioner’, ‘advanced’
intent             # e.g. ‘tutorial/how-to’, ‘news/announcement’, ...
cluster            # numeric cluster id
cluster_label      # human-friendly cluster label (e.g. ‘K-Means basics’)

1. A cluster report (cluster_report.csv) with ~10 clusters:

   • Cluster ID + label

   • Exemplar article (title + URL) closest to the centroid

   • LLM-generated bullets summarizing what that cluster is “about”

Once you have this, you can ask questions like:

• “Show me all tutorial posts that are beginner level and include code.”

• “What are our top 10 clusters of content, by article count?”

• “Find opinion pieces about AI in business that link well with this new announcement.”

Or from an agent’s perspective:

“Given this developer’s last 3 reads, suggest 2 more articles that match their level and intent.”

The nice part is: all of this emerges from one fenic pipeline (load ➡️ enrich ➡️ embed ➡️ cluster ➡️ classify ➡️ export).

Why fenic is a good fit for this problem

Content intelligence is messy. You deal with:

• Unstructured text (markdown, HTML, scraped pages)

• A mix of cheap heuristics (string length, regex) and semantic signals (embeddings, classification)

• The need to batch model calls and keep costs predictable

You could wire this together with Pandas, a few bespoke scripts, and some async model calls. But it gets hard to:

• Explain what’s happening

• Reproduce the run later

• Move from notebook to scheduled job to agentic tools

fenic’s value here is that everything is a DataFrame transformation:

• semantic.embed to add an emb column

• semantic.classify for narrative intent

• semantic.extract to produce typed summaries

• write.parquet/write.csv for export

You describe the pipeline once, and it can run in Colab today and as a scheduled job later. There is no need for custom batching, model SDK glue, or a separate ETL job.

Architecture at a glance

The notebook is deliberately small and linear. Here’s the flow you’ll run in Colab:

1. Load and normalise
   Bring a CSV or table with at least url, title, body. Create a short snippet (body_clip) that keeps model calls cheap and deterministic.

2. Cheap features first
   Compute char_len, clip_len, and a simple has_code flag (look for code fences/backticks/language tags). These already answer useful questions before any embeddings.

3. Semantic clusters
   Embed body_clip and group articles into K coherent clusters. For each cluster:

   • Pick the exemplar (closest to centroid)

   • Generate a concise cluster label (human-readable)

4. Narrative intent
   Use few-shot semantic.classify to tag each article as tutorial/announcement/explainer/opinion/case study. Crucially, build examples from your corpus (not toy prompts) so labels reflect your house style.

5. Complexity buckets
   Bucket into intro/intermediate/advanced with a transparent rule over char_len and has_code.

6. Cluster summaries
   For the ~10 clusters, run grounded extraction to get 1–3 bullets on audience + tone (typed via a mini Pydantic schema). These bullets anchor editorial decisions and agent prompts.

7. Export
   Write features.parquet and cluster_report.csv with fenic’s native writers. Done.

All of the above is implemented in the Colab notebook.

Where this becomes useful

Once you have features.parquet and cluster_report.csv, there are several easy wins.

1. Smarter RAG and content search

Instead of feeding your LLM the “top 5 BM25 matches,” you can:

• Prefer tutorial/how-to articles when the question sounds like “how do I…?”

• Prefer case studies when the question is about business impact

• Filter to beginner-level content for newer users

You already have the features; you just plug them into your retrieval and ranking logic.

2. Content recommendations (“read next”)

Given an article the user just read:

• Use the embedding and cluster to find semantically similar pieces

• Use intent and complexity to avoid repetitive or mismatched suggestions

Example:

If a user just finished a “beginner” explainer in cluster “K-Means basics,” recommend one “practitioner” tutorial and one related opinion piece at the same cluster.

3. Editorial analytics and gaps

Your content/product teams can now ask:

• “How many advanced tutorials do we have on observability?”

• “Are most of our opinion pieces clustered around the same theme?”

• “Which clusters have no case studies yet?”

Because everything is just a table, you can answer these with a few fenic group_by/agg calls or plug the parquet into your BI tool of choice.

4. Agent surfaces

Finally, this is a perfect substrate for agents.

You can expose read-only MCP tools like:

• list_articles(intent, complexity)

• similar_articles(url)

• cluster_overview(cluster_id)

Each tool is a deterministic fenic query over the feature table. Your agent becomes a thin layer that:

1. Interprets the user’s request

2. Calls the relevant tool(s)

3. Renders the result in a friendly way

No need to let the model “hallucinate” your catalog; it just queries it.

Cost, latency, and control

Because fenic handles batching and a minimal API surface, it’s straightforward to reason about cost:

• Embeddings: main paid step; we embed body_clip once per article

• Classification: one semantic.classify call per row, typically via a small “mini” LLM

• Extraction: only for ~10 cluster exemplars, so negligible

You can tune:

• The snippet length (body_clip)

• The subset of rows you classify (e.g., skip archival content)

• The model aliases used in your semantic_cfg

Everything else (feature engineering, clustering, exports) runs locally.

Adopting this pattern in your own stack

If you want to replicate this with your docs or blog:

1. Load your corpus into fenic with at least url, title, and body.

2. Add cheap features (char_len, has_code, maybe tags or product names).

3. Use with_embeddings and with_cluster_labels to create semantic clusters.

4. Summarise clusters with semantic.extract so humans can reason about them.

5. Define a small, opinionated intent taxonomy and build few-shot examples from your corpus.

6. Add a lightweight complexity bucket signal that fits your audience.

7. Export features.parquet and integrate it into RAG, search, agents, or analytics.

You don’t need to adopt agents on day one. Even just having a clean content feature table will pay off in search, analytics, and planning.

💡 Maybe useful: From Threads to Themes – How We Built a Fast HN Research Agent with fenic + Pydantic-AI

Conclusion and closing thoughts

If your team is sitting on a large, messy content archive and you’re not sure what to do next, the pipeline is a very practical way to see the shape of what you already have.

fenic’s sweet spot is exactly this kind of work: turning unstructured text into structured, semantic, agent-ready tables without giving up the simplicity of a DataFrame pipeline.

fenic gives you the DataFrame-style ergonomics you’re used to, plus semantic tools built for this kind of AI-adjacent work.

The rest is just deciding what questions you care about and letting the pipeline answer them.

Try the demo, then make it yours

• Clone the demo: Use our Colab and point it at your own small corpus slice first. Keep body_clip short, then grow from there.

• Docs: semantic operators, text utilities, batch inference

• Examples: GitHub → typedef-ai/fenic (look for other applications and MCP tool demos).

But here’s the twist: models aren’t just smaller now; they’re smarter, more efficient, and in some cases capable enough to run in full precision without any compression at all. Which leads to the big question:

What models today can run unquantized directly on your CPU?

That’s what this article is all about. We’ll walk through the small but powerful models you can run locally without quantization, focusing on three key categories:

• Language models

• Reasoning models

• Agent-ready models with tool-use capabilities

Along the way, we’ll compare them using two practical criteria: how much memory they require and how fast they run during inference.

Gemma 3 270M: The perfect Small Language Model

Gemma 3 270M is one of the smallest modern language models available, and its tiny size makes it ideal for CPU inference. It delivers surprisingly fast generation speeds and has a very small memory footprint, making it easy to run on machines with as little as 8 GB of RAM.

The model comes in two variants:

• Gemma 3 270M PT: The pretrained version

• Gemma 3 270M IT: The instruction-tuned and chat-focused version

The instruction-tuned version is exciting because it can maintain natural conversations in a way that feels much closer to larger models. Despite its small parameter count, it handles dialogue and everyday tasks better than many older models that were several times its size.

Thanks to its compact architecture, Gemma 3 270M is also perfect for fine-tuning experiments. You can fine-tune it on a GPU in Google Colab, export the resulting model, and then run it on your CPU without any quantization. This versatility makes it one of the most flexible and accessible small LLMs available today.

Running the model

Running the unquantized version of Gemma 3 270M is straightforward. Once you have access to Hugging Face, you can load it directly using the Transformers library:

from transformers import pipeline

pipe = pipeline(”text-generation”, model=”google/gemma-3-270m-it”)
messages = [
    {”role”: “user”, “content”: “Who are you?”},
]
pipe(messages)

Although the model is free to use, it is gated, meaning you must request and receive permission on Hugging Face before you can download or load it.

Gemma 3 1B: Bigger but still Small

If you want something noticeably more capable than Gemma 3 270M, Gemma 3 1B is the next best step up. It consistently outperforms the 270M model across benchmarks while still maintaining a small memory footprint and fast inference speeds. And just like its smaller sibling, it comes in two variants: a pretrained (PT) version and an instruction-tuned (IT) version.

Despite being larger, Gemma 3 1B remains lightweight enough to run unquantized on many CPUs, making it a great middle ground between tiny models and full-scale LLMs.

Running the model

Running Gemma 3 1B is identical to running the 270M version, simply reference the model name when loading it through the Transformers pipeline:

from transformers import pipeline

pipe = pipeline(”text-generation”, model=”google/gemma-3-ib-it”)
messages = [
    {”role”: “user”, “content”: “Who are you?”},
]
pipe(messages)

Qwen 3 0.6B: Thinking and Agent in a small package

Qwen 3 0.6B is another model that can be run fully on the CPU without quantization, but unlike the Gemma models, it offers much more than standard text generation. Qwen 3 0.6B is both a reasoning model and a tool-using model, giving it capabilities typically found in much larger LLMs.

This makes it especially impressive for a model that still maintains a very small memory footprint. The only trade-off is that reasoning mode is slower, since generating internal thoughts produces more tokens.

Qwen 3 0.6B is a great example of how far small models have come. They’re no longer limited to simple text generation, they can reason, plan, and interact with tools, making them suitable for lightweight agent setups that run entirely on a CPU.

Running the model

Just like the Gemma models, Qwen 3 0.6B is available on Hugging Face. Below is an example showing how to enable the model’s thinking mode:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = “Qwen/Qwen3-0.6B”

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=”auto”,
    device_map=”auto”
)

# prepare the model input
prompt = “How many R are in Blueberry?”
messages = [
    {”role”: “user”, “content”: prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors=”pt”).to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip(”\n”)
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip(”\n”)

print(”thinking content:”, thinking_content)
print(”content:”, content)

Below is the model’s output:

thinking content: <think>
Okay, so I need to figure out how many times the letter “R” appears in the word “blueberry”. Let me start by writing down the word and looking at each letter one by one.

The word is “blueberry”. Let me spell it out: B-L-U-E-B-R-R-Y. Now, I need to check each letter for the letter “R”. Let me go through each position:

First letter: B – not R.
Second: L – nope.
Third: U – no.
Fourth: E – no.
Fifth: B – no.
Sixth: R – here we go! The first “R” is at position 6. Then the next letter is “R” again at position 11. So that’s two instances.

Wait, let me check again to make sure I didn’t miss anything. Let’s break it down:

B, L, U, E, B, R, R, Y. So positions 6 and 11. So yes, two “R”s. I think that’s it. I don’t see any other “R”s in the word. So the answer should be 2.
</think>
content: How many R is in blueberry?

The word “blueberry” is spelled B-L-U-E-B-R-R-Y. 

Looking at each letter:
- The first R is at position 6.
- The second R is at position 11.

Therefore, the number of R’s in “blueberry” is **2**.

Tool use can be combined with thinking mode. Here’s an example:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = “Qwen/Qwen3-0.6B”
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, dtype=”auto”, device_map=”auto”)

def get_current_temperature(location: str, unit: str):
    “”“
    Get the current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format “City, Country”
        unit: The unit to return the temperature in. (choices: [”celsius”, “fahrenheit”])
    “”“
    return 42.

def get_current_wind_speed(location: str):
    “”“
    Get the current wind speed in km/h at a given location.

    Args:
        location: The location to get the wind speed for, in the format “City, Country”
    “”“
    return 6.

tools = [get_current_temperature, get_current_wind_speed]

messages = [ {”role”: “user”, “content”: “Hey, what’s the temperature in Paris right now?”} ]

inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=True, return_tensors=”pt”)
outputs = model.generate(**inputs.to(model.device), max_new_tokens=128)
print(tokenizer.decode(outputs[0][len(inputs[”input_ids”][0]):]))

Model’s output:

<think>
Okay, the user is asking for the current temperature in Paris. Let me check the tools available. There’s get_current_temperature which requires location and unit. The user didn’t specify the unit, but maybe I should assume Celsius unless told otherwise. The location is Paris, so I need to call the function with location “Paris, France” and unit “celsius”. Let me make sure the parameters are correctly formatted. Yes, that should do it.
</think>

<tool_call>
{”name”: “get_current_temperature”, “arguments”: {”location”: “Paris, France”, “unit”: “celsius”}}
</tool_call><|im_end|>

Qwen 3 1.7B: 1 Billion parameter bigger but still small

Qwen 3 1.7B takes everything the 0.6B model can do and scales it up by an extra billion parameters. The result is a model that still runs on CPU without quantization, but with noticeably stronger reasoning, improved tool use, and better overall performance.

Despite being the largest model on this list, it remains lightweight enough to run on machines with 8GB RAM, making it an impressive “upper-limit” option for small-model experimentation on consumer hardware.

The trade-off?

Inference can be slow on CPU, especially when the thinking mode is enabled. But if you don’t mind waiting a bit longer, it consistently produces solid reasoning and remains one of the most capable CPU-friendly LLMs you can run today.

Running the model

Running Qwen 3 1.7B is identical to running Qwen 3 0.6B:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = “Qwen/Qwen3-1.7B”

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=”auto”,
    device_map=”auto”
)

# prepare the model input
prompt = “How K is in Strawberry?”
messages = [
    {”role”: “user”, “content”: prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors=”pt”).to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip(”\n”)
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip(”\n”)

print(”thinking content:”, thinking_content)
print(”content:”, content)

Example Output:

thinking content: <think>
Okay, the user is asking “How K is in Strawberry?” which seems a bit confusing. Let me break this down.

First, “Strawberry” is a common fruit, but the question is about the letter “K” in it. So maybe they’re asking how the letter “K” appears in the word “Strawberry.”

Let me check the spelling of “Strawberry.” S-T-R-A-W-B-E-R-R-Y. Wait, is there a “K” in there? Let me spell it again: S-T-R-A-W-B-E-R-R-Y. No, there’s no “K” in “Strawberry.” So the answer would be that there’s no “K” in the word.

But maybe the user is thinking of a different word. Could it be a typo? Like “Strawberry” might be misspelled, or maybe they meant another word. Alternatively, maybe they’re referring to a different context where “K” is involved. For example, in the name “Strawberry” in a different language, but I don’t think that’s the case here.

Alternatively, maybe the user is asking about the position of the letter “K” in the word, but since there’s no “K,” the answer would be that there’s no “K” in “Strawberry.”

Wait, but maybe they’re thinking of a different word. For example, “strawberry” in some other language. Let me check. In French, “strawberry” is “framboise,” which doesn’t have a “K.” In Spanish, “fresa,” which also doesn’t have a “K.” In German, “Birne” is a different fruit, but that’s not strawberry. So no “K” there either.

Alternatively, maybe the user is referring to the letter “K” in the name “Strawberry” as a brand or something else. But I don’t think that’s the case. The question seems straightforward, so the answer is that there’s no “K” in “Strawberry.”

I should also consider if there’s any possible way the user might have made a mistake. For example, maybe they meant “Strawberry” with a “K” in another part of the word. But as written, “Strawberry” doesn’t have a “K.” So the answer is that there’s no “K” in the word.
</think>
content: The letter **K** does not appear in the word **”Strawberry”**. 

**Explanation:**  
The spelling of “Strawberry” is **S-T-R-A-W-B-E-R-R-Y**, which contains no “K” sound or letter. 

If you’re referring to a different word or context where “K” might be involved, feel free to clarify! 😊

Even with thinking mode disabled, Qwen 3 1.7B still performs tool calls accurately:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = “Qwen/Qwen3-1.7B”
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, dtype=”auto”, device_map=”auto”)

def get_current_temperature(location: str, unit: str):
    “”“
    Get the current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format “City, Country”
        unit: The unit to return the temperature in. (choices: [”celsius”, “fahrenheit”])
    “”“
    return 42.

def get_current_wind_speed(location: str):
    “”“
    Get the current wind speed in km/h at a given location.

    Args:
        location: The location to get the wind speed for, in the format “City, Country”
    “”“
    return 6.

tools = [get_current_temperature, get_current_wind_speed]

messages = [ {”role”: “user”, “content”: “Hey, what’s the temperature in Paris right now?”} ]

inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_dict=True,  enable_thinking=False, return_tensors=”pt”)
outputs = model.generate(**inputs.to(model.device), max_new_tokens=128)
print(tokenizer.decode(outputs[0][len(inputs[”input_ids”][0]):]))

Output without thinking:

<tool_call>
{”name”: “get_current_temperature”, “arguments”: {”location”: “Paris, France”, “unit”: “celsius”}}
</tool_call><|im_end|>

Conclusion: LLMs You Can Run on Your CPU Without Quantization

In this article, we’ve seen how far large language models have come, becoming smaller, more efficient, and still surprisingly capable.

Today, it’s possible to run sophisticated LLMs entirely on a CPU without any quantization, opening up new possibilities for experimentation, fine-tuning, and lightweight deployment.

This marks a significant milestone in the evolution of the LLM ecosystem, showing just how accessible powerful AI has become.

Our goal isn’t to suggest abandoning quantized models; they remain essential for extremely large models or resource-constrained environments.

Rather, this exploration highlights the performance of small, unquantized models straight out of the box and encourages a new way of thinking: for many tasks, do you really need a massive, quantized model, or could a smaller model deliver what you need more efficiently?

By understanding the strengths of these compact models, developers and researchers can make smarter choices, balancing speed, memory, and capability in ways that were not possible just a few years ago.

This technical guide is a repost from what originally appeared on Typedef AI’s blog as Build an LLM Agent for Log Clustering & Triage.

There’s a point in every incident where the problem isn’t finding logs but in understanding them. Formats drift, error messages mutate, and even well-tuned dashboards can feel like they’re describing yesterday’s system.

In the middle of the scramble, engineers ask the same questions again and again:

What broke? How widespread is it? Where should we look first?

This article lays out a pragmatic, agent-first way to answer those questions with Fenic. Fenic is an opinionated, PySpark-inspired DataFrame framework built for AI/agentic applications.

You’ll learn how we turn raw logs into severity-aware clusters, expose read-only tools via MCP, and let a LangGraph agent respond to natural-language questions like “only errors in payments” or “show me the raw lines for cluster 7.”

The full runnable demo is in a Colab/Repo. In this blog, you’ll see the why and the what so you can decide quickly whether this solution fits your stack.

Try it now → Open the Colab | Repo → typedef-ai/fenic-examples | Docs → https://docs.fenic.ai/latest/

Why a different approach to log triage

Most of us have tried two extremes:

1. Regex everywhere. It starts clean and ends brittle. One library upgrade or format tweak and your “error” dashboard quietly loses half its signal. You might not notice until the night you needed that signal most.

2. One-click LLM summarization. Tempting, but operationally awkward: cost is difficult to predict, behavior drifts with prompts, and there’s no clean hand-off back into the tooling your SREs actually use.

The middle path is boring on purpose, but it works: Keep the DataFrame ergonomics engineers already use, add semantic operators where they pay off, and make the “AI bits” visible, testable, and schedulable.

That’s the ethos behind Fenic. You manipulate logs like a table, but you also get first-class text extraction, embeddings, and LLM utilities. This is all with explicit configuration and batch-friendly behavior.

The promised end-state (what “done” looks like)

Imagine asking your system:

• “Top clusters above WARN in the last hour.”

• “Only ERRORs for payment-api.”

• “Assignments for cluster 7—I want the raw lines.”

• “Coverage check—are we parsing reliably today?”

The agent answers in seconds because it isn’t guessing. It’s calling deterministic MCP tools backed by DataFrames you just computed:

• Severity-aware clusters of similar failures (stable across parameter noise).

• Exemplars that read like summaries, not stack dumps.

• Evidence via fingerprints so a human can sanity-check the grouping.

• Coverage: a simple metric that tells you whether your parsing kept up with reality.

There is no need to sift through pages of stack traces. The agent uses the read-only, queryable views you produced during the pipeline run.

Behind the scenes, you also keep operator-ready artifacts (CSV/JSON) for dashboards and a compact Markdown report for humans. Those artifacts are the audit trail; the agent is the interface.

Interested in Markdown Processing? Run the extensive example in Colab.

Architecture at a glance (why it stays maintainable)

The pipeline that powers those tools is deliberately small and linear. Each step is a DataFrame transform you can diff, test, and run headless.

1) Parse without brittleness

Real systems emit a mix of ISO-ish timestamps, syslog-style messages, Python logging lines, and occasionally JSON.

With Fenic, you define a few templates (named fields, not brittle capture groups), unnest them into candidate columns, and coalesce to a canonical schema:

• timestamp (normalize later if needed)

• level (INFO/WARN/ERROR/...)

• service (e.g., payment-api)

• message (payload after the header)

• optional trace_id

A line that doesn’t match any template doesn’t vanish; it falls back to raw text. That single choice makes coverage transparent and drift visible.

2) Fingerprint for stability

Clustering succeeds or fails on the grouping key. We want a fingerprint that ignores volatile tokens (IDs, ports, timings) but preserves the cause. A practical pattern is:

service | symbol | file#function | stem

• symbol: recognizable error label (TimeoutError, OperationalError, unique_constraint_failed).

• file#function: optional call site (useful for stack-tracey languages).

• stem: a normalized message that keeps the idea of the failure while stripping noise.

You can extract these with rules or a tiny LLM call. Either way, the fingerprint stays explainable because you keep the pieces next to it in the table.

3) Tag severity deterministically

Before we embed anything, we separate signal from noise with cheap rules:

• Hard failures (level=ERROR/FATAL/CRIT, timeout, connection refused, HTTP 5xx, nginx upstream errors) → error

• Soft indicators (retry, latency, degraded) → warn

• Everything else → info

This stops INFO chatter from drowning out real problems and lets you cluster inside severity buckets. You can layer an LLM adjudicator later for ambiguous messages, but the baseline is deterministic and fast.

4) Cluster semantically (per severity)

Now the semantic bit pays off. You embed enriched text, like:

[svc:payment-api] [sev:error] TimeoutError on /v1/users after Xms

Clustering within severity buckets keeps WARN chatter from bleeding into ERRORs. Choose K-Means (with a small heuristic for K) or a density approach (HDBSCAN) for larger bins within each severity.

For each cluster, pick the exemplar closest to the centroid; that exemplar often reads like the sentence you’d write in a postmortem.

The result is a handful of clusters, each with a count, an exemplar, and a fingerprint, which is precisely what humans need to triage.

5) Publish read-only tools

Persist the three useful views (triage, clusters, and assignments) and register MCP tools with Fenic:

• list_clusters(severity_floor) → ranked by severity-weighted score

• clusters_by_severity(severity) → a single lane

• assignments_for_cluster(cluster_id) → raw lines and evidence

• coverage_metrics() → processed vs total lines

These are just queries over tables; they return rows, not plain-English (narrative) summaries.

A LangGraph agent sits on top and chooses the right tool based on the question.

The Neural Blueprint: Practical Content for AI Builders

In-Depth Review of LangGraph: The Agentic State Machine

At the inaugural LangChain Interrupt conference this year, the spotlight wasn’t on LangChain itself; it was on LangGraph. The event focused heavily on AI agents and showcased how major companies, such as Uber, LinkedIn, and Replit, are deploying them in production…

Read more

10 months ago · Eteimorde Youdiowei and Neurl Creators

Agent-first: the interface humans actually use

The trick to a good agent is primarily in well-chosen tools. With deterministic MCP calls, the agent becomes a thin orchestration layer:

• Question → tool selection (e.g., “top” → list_clusters with warn)

• Fetch rows → render clearly (markdown tables are enough)

• Drill-down → if the human asks, call assignments_for_cluster with the id in the current row

There’s no hidden magic. If the agent faces an ambiguous question, it could enrich a summary for the top 3 clusters with a small, cheap LLM. But the default behavior is to show the facts and let the responder decide.

This is why the agent replaces the daily digest: it answers the current question, at the moment of curiosity, with just enough detail to act.

What “good” looks like in production

After a week of daily runs, you’ll notice:

• Coverage is stable (e.g., 80–90%). If it dips, that’s a crisp signal you need a new template or a parsing tweak.

• Clusters are consistent day-to-day. The exemplar for “connection timeouts to db-prod” still points at the same root cause. If a cluster explodes in size, you know where to look fast.

• On-call asks the agent first. Instead of searching dashboards, responders query: “top clusters above warn,” “only errors,” “assignments for #7.” They get compact tables, not walls of text.

• Artifacts fit right in. CSV/JSON slots into dashboards and data lakes; Markdown renders well in tickets and wikis.

The effect isn’t flashy but helps you respond with more velocity, which, of course, is less guesswork and more progress.

Why this design holds up in day-2 reality

• Adoption stays high. DataFrame ergonomics mean less training and faster pull requests. You can add a template or tweak a rule without changing the whole system.

• Determinism where you want it. Severity, fingerprints, and clusters don’t change because the operators are explicit. If a cluster changes, you can usually point to the reason (new message variant, new template).

• Costs don’t surprise you. Deduplicate before embedding, embed once per unique fingerprint, and summarize only exemplars if you need narrative text. Your spend tracks distinct issues and not raw volume.

• It’s easy to run headless. The same notebook steps translate into a small script; artifacts are files; tools are queries. Nothing exotic.

• The agent is an interface, not a black box. It calls small, auditable tools over your tables. You can inspect the tables, test the tools, and trust the response.

Adopting this incrementally

A common failure mode is trying to “agentify” the entire logging stack at once. Don’t. Here’s a low-friction rollout that respects your time and budget:

1. Run the demo against a small, representative slice of your logs.

2. Check coverage. If it’s low, add a template (JSON is a big win if you have it) and normalize service names (merge aliases).

3. Turn on the agent and use it during on-call. Let people ask natural-language questions and see cluster movement.

4. Stabilize cluster IDs if needed: enrich the fingerprint prefix or seed K-Means with yesterday’s centroids so exemplars stay familiar.

5. Only then consider LLM summaries for the top few clusters if responders want them. Keep it bounded.

6. Integrate light observability. Track coverage and cluster counts; alert on sudden dips or spikes.

At no point do you need to replace your existing log tools. This pipeline sits next to them and explains what they’re already collecting.

Cost, latency, and control (picking an operating point)

• Parsing is free aside from CPU. Templates are deterministic and cheap.

• Fingerprints are near-free if rule-based; if you choose to extract with an LLM, dedupe prompts by unique message text.

• Embeddings are the main paid step. Control spending by deduplicating on the fingerprint and batching aggressively. You embed distinct issues, not every log line.

• Clustering (K-Means or HDBSCAN) runs locally and scales well.

• Summaries are optional and should be limited to top-K clusters, where a human will actually read them.

If you want one sentence to carry to your next platform meeting: rules and embeddings, agent-first, and summaries only if people ask for them.

Governance and privacy (don’t skip this)

If there’s any chance your logs include PII/PHI or secrets, redact before any model call:

• Mask emails, tokens, phone numbers, and user IDs at the message field.

• Keep raw logs inside your private boundary; export only fingerprints, counts, exemplars, and cluster IDs externally.

• If required, route embeddings to a provider in your VPC, or use a private endpoint.

Fenic doesn’t force you into any specific model provider, which keeps compliance conversations simpler.

What success looks like

After a few days, you’ll notice the conversation change. People stop posting raw stack traces and start asking the agent:

• “only error clusters in payments since 9am?”

• “assignments for cluster 12?”

• “coverage today vs yesterday?”

You’ll also recognize a few patterns:

• Coverage hovers above the threshold you set (say, 70-80%). When it dips, a new template or a rename fixes it.

• Top clusters stay stable unless something genuinely changes in production. When an ERROR cluster doubles overnight, it’s a true signal.

• The agent becomes the default way people ask for status: SREs, support engineers, and even PMs who want a quick read before stand-up.

• Code changes that would have broken a regex pipeline are non-events. You coalesce a new template and move on.

And that’s the real win: fewer firefights over tooling and more time hunting actual root causes.

Try it, then make it yours

• Open the Colab and run the demo with sample logs, or drop in a small batch of your own.

• Adopt the minimal pipeline (parse → fingerprint → severity → cluster) and expose the tools.

• Use the agent during on-call. Adjust templates and fingerprints as reality evolves.

• Iterate: If responders ask for more context, add domain columns (endpoint, table, region) into fingerprints and enriched text. If they want prose, summarize exemplars only.

Grab the Essentials

• Docs: Fenic operators, semantic config, and MCP tooling.

• GitHub: clone, star, and open issues/PRs.

More examples and ready-to-run scripts.

1. OSS extraction library (unstructured)

2. ML add-ons (unstructured-inference)

3. Scalable pipelines (unstructured-ingest)

4. Serverless/hosted Partition and Chunking API

That mix is why it shows up in so many production RAG stacks today. Version 0.18 series (summer 2025) introduced faster PDF partitioning and a revamped chunking API for RAG pipelines.

📢 What Is Unstructured?

An open-source ETL framework that:

1. Loads documents from 40+ sources (filesystem, S3, Gmail, Jira, …).

2. Partitions each file into logical elements (Title, NarrativeText, Table, ListItem, PageBreak).

3. Enriches with layout ML (table boundaries, header detection, image OCR).

4. Spits out JSON, Markdown, HTML, or Arrow, with coordinates, page numbers, languages, SHA hashes.

The same API powers:

• 🐍 Python SDK (from unstructured.partition.pdf import partition_pdf(...))

• 🛠️ CLI (unstructured-ingest --strategy hi_res --recursive s3://bucket …)

• ☁️ Cloud SaaS/Docker micro-service (POST /general/v0/general)

⚙️ How the Pieces Fit

• FAST strategy = pdfminer + heuristic chunking (seconds per doc).

• HI-RES (--strategy hi_res) calls the Chipper model for page-layout accuracy; slower, GPU optional GitHub.

⚙️ Key Features of Unstructured

Here are all the core components of Unstructured:

🛠️ How Unstructured Works (Under the Hood)

Partition → Elements → Chunk → Embed
You call a partition_* function (or the API). It returns a list of typed elements (e.g., Title, NarrativeText, Table) with metadata (page, coords, text_as_html for tables, etc.). You then chunk those elements (e.g., “by_title”) and embed the chunks.

Extraction strategies

• fast: fastest, fewer heavy dependencies; good for bulk/simple docs.

• hi_res: uses unstructured-inference layout models (Detectron2) for higher fidelity on PDFs/slides; better table/figure handling (needs OS deps).

• vlm: uses a vision-language model for image-heavy docs or screenshots.

Tables and images

• Set skip_infer_table_types to False to preserve tables (HTML in metadata.text_as_html).

• There’s a guided “Extract tables as HTML” and “Extract images and tables” how-to in the docs.

Chunking

• Core chunkers combine neighboring elements with size limits; by_title respects section boundaries (and never mixes a Table with other text). API adds extra strategies and parameters (combine_under_n_chars, multipage_sections, by_similarity).

🚀 Quick Spin-Up (two ways)

A) One-line API (Docker or hosted)

# Local API (example)
docker pull downloads.unstructured.io/unstructured-io/unstructured-api:latest

# Run locally
docker run -p 8000:8000 -d --rm --name unstructured-api \
  downloads.unstructured.io/unstructured-io/unstructured-api:latest

# Send a PDF → get elements + chunks, hi-res + tables HTML preserved
curl -X POST “http://localhost:8000/general/v0/general” \
  -H “Content-Type: multipart/form-data” \
  -F “files=@sample.pdf” \
  -F “strategy=hi_res” \
  -F “chunking_strategy=by_title” \
  -F “combine_under_n_chars=500” \
  -F “multipage_sections=true” \
  -F “include_orig_elements=true”

Why this way? Fewer OS deps; newest chunking features are first-class in the API.

B) Local Python (OSS): digital PDF, keep it fast

B) Local Python (OSS): digital PDF, keep it fast

# pip install “unstructured[pdf]”
from unstructured.partition.pdf import partition_pdf

elements = partition_pdf(
    filename=”report.pdf”,
    strategy=”fast”,            # try fast first; faster on simple PDFs
    extract_images=False
)
# elements is a list of typed blocks: Title, NarrativeText, ListItem, Table, ...

Unstructured’s docs recommend trying fast on simple PDFs before reaching for hi_res to reduce processing time.

C) Local Python (scanned or mixed layout PDF, include tables)

# pip install “unstructured[pdf]”  # brings layout/ocr deps
from unstructured.partition.pdf import partition_pdf

elements = partition_pdf(
    filename=”financials_scanned.pdf”,
    strategy=”hi_res”,                # layout models (Detectron2/YOLOX)
    infer_table_structure=True,       # include HTML table structure
    extract_images=True
)

(Note: hi_res needs OS deps like poppler, tesseract, libmagic, libreoffice per README.)

⚙️ Connectors for Real Pipelines

Need to pull from SaaS or clouds and push to a lake/vector DB? Unstructured Ingest ships many source and destination connectors (S3/GCS/Azure, Confluence, Notion, Salesforce, Airtable, Slack, SharePoint, etc.), and can call the API directly (--partition-by-api).

Example (Airtable → API → JSON):

# deps
pip install “unstructured-ingest[airtable]”

# run local API in another shell if you want to avoid hosted keys:
# docker run -p 8000:8000 -d --rm --name unstructured-api \
  downloads.unstructured.io/unstructured-io/unstructured-api:latest

unstructured-ingest \
  airtable \
  --personal-access-token “$AIRTABLE_PAT” \
  --list-of-paths “appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit” \
  --output-dir ./local-out \
  --num-processes 2 \
  --reprocess \
  --partition-by-api \
  --partition-endpoint “http://localhost:8000/general/v0/general”

🔥 Why MLOps Engineers Should Care

1. Better recall with fewer hallucinations: Element-aware chunking maintains the section semantics and table structure, resulting in retrieval units that are cleaner than those produced by blind text splits.

2. Switchable fidelity: fast for throughput; hi_res when layout matters (scientific PDFs, slide decks).

3. Production dataplane: Use the API to avoid wrestling with OS deps and to access advanced chunking controls.

4. End-to-end ingestion: Connectors + chunking + metadata pave a straight path to embeddings and vector stores.

5. API extras. The hosted API exposes chunking modes and strategies beyond OSS defaults (e.g., vlm).

📉 Gotchas and Caveats (Read This Before You Roll Out)

1. hi_res is heavier. Layout inference improves accuracy on tables/figures but increases runtime and system deps; try fast first on digital PDFs.

2. Tables aren’t “just there.” On PDFs you’ll want strategy=”hi_res” and skip_infer_table_types=False to populate text_as_html on Table elements. (The older pdf_infer_table_structure is deprecated.)

3. API-only capabilities exist. Some strategies (e.g., vlm) and chunking options are exposed in the Serverless API rather than the OSS path.

4. OCR languages and agents. For non-English scans, pass languages=[...] (API) or set the OCR agent in OSS; you’ll need language packs.

5. Large-file performance. Unstructured’s docs include concrete tuning tips for big PDFs/batches; read them before you queue a 2k-page corpus.

🧪 Sanity-Check Scenarios

• RAG for financial filings (scanned + tables). Use strategy=”hi_res” with skip_infer_table_types=False; chunk by title/semantics; consider API if you want vlm for image-heavy pages.

• Policy docs/handbooks (digital PDFs). Run fast, chunk by titles/sections. Faster, fewer deps, usually enough fidelity.

• SharePoint/Confluence crawl. Use unstructured-ingest; start local, then flip --partition-by-api for scale without changing code.

🗣️ Community Pulse

Thread in MLOps.Community Slack.

💡 Real-World Use Case: Agentic Safety Red-Team

1. Ingest 10k annual reports (PDF) from S3 nightly.

2. Hi-Res partition with table inference → JSON → DuckDB.

3. Chunk and embed with OpenAI Ada-3.

4. Query via a LangChain retriever in a chatbot.

5. Outcome: 40% reduction in hallucinated KPIs vs. plain pdfminer pipeline.

📊 How Unstructured Stacks Up (2025)

➡️ Third-party rows are directional; choose based on vendor posture, compliance, and TCO.

🧑‍⚖️ Final Verdict: 4.4/5 “Pragmatic, Proven, and Worth the Slot in Your Stack”

Rating: ⭐⭐⭐⭐☆ (4.4/5)

✅ Ship it if…

• Your corpus spans scans + digital PDFs + HTML/Office, and you need typed elements (esp. tables, images).

• You want one ETL that can run locally now and API later (same JSON shape).

• You need connectors to move content at scale without writing crawlers.

❌ Hold off if…

• Your docs are all digital PDFs with simple text, and a basic text extractor suffices.

• You can’t accept heavier deps/runtime for hi_res, and no API usage is allowed.

• You already standardized on a single cloud OCR like LlamaParse/Llama Cloud that meets all needs and don’t need OSS flexibility.

Bottom line: For teams serious about production RAG/agents over messy docs, Unstructured gives you a cohesive OSS toolbox (from one-liners to a full ETL) with the right escape hatches for fidelity and scale.

📌 Resources to Learn More

• Docs & Quick-starts: https://docs.unstructured.io/welcome

• GitHub core library: https://github.com/Unstructured-IO/unstructured

• Ingest connectors: https://github.com/Unstructured-IO/unstructured-ingest

• Chunking best practices blog: https://unstructured.io/blog/chunking-for-rag-best-practices

• Example notebooks: https://github.com/Unstructured-IO/notebooks

Love this review? Drop your war stories (and weird PDFs) in the comments, or share on X with #TuesdayToolReview and tag

Leave a comment

If you want a deeper walkthrough (like parsing, chunking strategies, and table-aware retrieval), subscribe to The Neural Blueprint 👇🏾 for hands-on guides! 🫡 Follow us on YouTube, X, and LinkedIn.

Subscribe now

It ships opinionated primitives (dataset → Task → Solver → Scorer), multi-turn/agent workflows with tools, sandboxed execution (Docker built-in, optional Kubernetes/Proxmox adapters), a VS Code log viewer, and a web-based Inspect View.

Install is one-liner: pip install inspect-ai (inspect-ai 0.3.130 as of Sep 7, 2025; MIT-licensed; Python ≥ 3.10).

📢 What Is Inspect AI (in 2025)?

At its core, Inspect is a declarative framework for evaluating AI systems via composable “tasks.” A task might check:

• whether a model refuses dangerous requests,

• whether an agent leaks system instructions,

• or whether an AI-to-AI convo converges on a correct answer.

Inspect handles:

✅ Prompt construction

✅ Response evaluation

✅ Score aggregation

✅ Logging (JSONL, Postgres, Dash UI)

✅ Isolation (via K8s or VMs)

Runs are explicit and inspectable. Everything is typed, reproducible, and introspectable, which is a must for frontier model auditing.

Backed by: UK AI Safety Institute (the team driving UK government safety standards for frontier AI).

Active user base: Adopted by METR, Apollo Research, other government AISIs, and major safety labs.

⚙️ Key Features of Inspect AI

Here are all the core components the eval framework comes with:

• Tasks/Datasets/Solvers/Scorers: Tasks bring together datasets (load data), solvers that elicit behavior (single/multi-step), and scorers (score outputs).

• Models layer: One interface over OpenAI, Anthropic, Google, Groq, Mistral, xAI (Grok), AWS Bedrock/AI Inference, Azure AI, Together, Cloudflare, Goodfire, plus local vLLM/Ollama/llama-cpp.

• Agents: Built-in ReAct, multi-agent composition, external agent bridge (e.g., AutoGen/LangChain).

• Tools: Built-ins (bash, python, text-edit, web_search, web_browser, computer) + MCP/Custom tools.

• Sandboxing and safety: Run untrusted code and browsers in Docker, K8s (pods per sample), or Proxmox with sandbox extensions and optional domain/network controls; Tool Approval (human-in-the-loop or policy-based gating).

• Dev Experience: Inspect View (web log viewer for basic observability) and VS Code extension for run/debug/tune.

• Scale Knobs: Caching, batch mode, parallelism, eval-set slicing, retry/resume.

• Evals Registry: Dozens of canonical evals (reasoning, coding, agent tasks, and cybersecurity).

Bonus: AISI publishes add-on packages like inspect-cyber for agentic cyber evals (see PyPI).

🛠️ How Inspect AI Works (Under the Hood)

1. Define Tasks: JSON/CSV/HF datasets feed Task objects; mix single-turn Q&A, coding, multi-modal, or open-tool agent prompts.

2. Elicit Solutions: inspect_ai.solver provides chain-of-thought, self-critique, tool-use, and MCP calls.

3. Execute Securely: Choose sandbox back-ends: process jail, Docker, or the community K8s sandbox for scale and isolation.

4. Score and Aggregate: Out-of-the-box model-graded QA, F1, pass@k, statistical bootstrap, plus custom metrics.

5. Analyze: Transform logs into evals_df, samples_df, events_df for Pandas-like slicing or Inspect Viz dashboards.

Each Task bundles:

• a prompt template (e.g., a jailbreak or math question),

• a model runner (e.g., OpenAI, HuggingFace, Ollama, local endpoint),

• a scoring function (numeric, boolean, class).

Example CLI command:

inspect eval my_eval.py --model openai/gpt-4o --limit 100

🚀 Quick Spin-Up (two minutes)

1. Install and pick a model provider

   pip install inspect-ai openai
   export OPENAI_API_KEY=your-key

2. Hello-World eval (exact match + single call)

   from inspect_ai import Task, task
   from inspect_ai.dataset import Sample
   from inspect_ai.solver import generate
   from inspect_ai.scorer import exact
   
   @task
   def hello_world():
       return Task(
           dataset=[Sample(input="Just reply with Hello World", target="Hello World")],
           solver=[generate()],
           scorer=exact(),
       )

   inspect eval hello.py --model openai/gpt-4o
   
   # You can also write a Parquet log you view live with the GUI and can also slice in Pandas for basic monitoring using the commands:
   # inspect eval theory.py --model openai/gpt-4o --log_dir runs/hello-world
   # inspect view runs/hello-world            # live GUI

3. Model-graded scoring (use an LLM to judge correctness):

   from inspect_ai.scorer import model_graded_fact
   # ... Task(dataset=..., solver=[...], scorer=model_graded_fact())

Examples for multi-choice (HellaSwag), GSM8K few-shot math, and custom math equivalence scorers are in the docs tutorial.

🔥 Why MLOps and AI Safety Teams Should Care

1. Single CLI for multi-vendor evals: No more bespoke scripts per provider.

2. Agent and tool safety: Isolated K8s pods or VM sandbox plugins let you run ReAct or Auto-GPT-style agents in locked pods. Includes policy limits for tokens, clock- and working-time (CPU time spent inside model/tool calls), messages, and sandbox parallelism/retries.

3. Model-graded scoring: A unified scoring library with bootstrap CIs and pass/fail gates that is vastly cheaper than human eval yet richer than Exact Match.

4. Rich observability: Inspect View + Inspect Viz render per-step traces, costs, and scorer heatmaps. You can also log every prompt, response, and token count for audit.

5. Ecosystem: k8s-sandbox, Hugging Face dataset bridges, SWE-Bench datasets, active Slack community, etc.

I found out that you can also add solvers, scorers, or entire toolchains via the extensions registry.

🗣️ Community Pulse

“We use https://github.com/UKGovernmentBEIS/inspect_ai which is great and comes with model-graded scorers. We also built support for reporting the results back to Langtrace AI which we use for observability.” — Karthik Kalyanaraman, Slack thread (July 2025)

“Inspect also has its own very active Slack community, and the Inspect userbase includes other safety research organisations (other AISIs, Apollo, METR) as well as some of the frontier labs.” — Jason Gwartz (Head of Platform, UK AISI), Slack thread (July 2025)

💡 Real-World Use Case: Agentic Safety Red-Team

1. Dataset: 500 complex multi-step jailbreak prompts.

2. Solver chain: ReAct agent → tool-calling → self-critique.

3. Sandbox: k8s_sandbox spins a pod per sample, capturing file/HTTP output.

4. Scorer: model-graded policy checker + custom regex.

5. Outcome: Run 500 × 4 models overnight; Inspect Viz dashboard highlights policy breaks with stack traces.

The agent is allowed to browse/execute code inside a K8s sandbox with domain allowlists while completing capture-the-flag tasks; this reduces blast radius if the model tries risky actions. (See Cybench guide.)

📊 How Inspect AI Stacks Up Against Other Eval Tools (2025)

🧑‍⚖️ Final Verdict: 4.3/5 “Production-grade evals & agent sandboxes, batteries included.”

Rating: ⭐⭐⭐⭐☆ (4.3/5)

If your org treats evals as production infrastructure (with agents, tools, and safety constraints), Inspect AI is the most complete OSS choice today. The learning curve and runtime overhead are real, but the auditability, sandboxing, and scorer depth justify it for serious teams.

✅ Ship it if…

• You run recurring evals/regressions across multiple vendors or models (cloud + local).

• You need to evaluate agents using tools with isolation (Docker/k8s) and approval policies.

• You want LLM-graded or domain-specific scoring beyond Exact Match.

• Regulatory or SOC2 demands tamper-proof logs and stats.

❌ Hold off if…

• You’re doing a one-week bake-off on static benchmarks. Lighter harnesses will be faster.

• You don’t plan to test tool use/agents (Inspect’s agent tasks can be spendy), and simple text-in/text-out scoring suffices.

• You can’t support even Docker-level isolation yet (agentic evals will be risky).

• You dislike declarative DSLs (lm-eval-harness may feel lighter).

🚀 Starter Commands You’ll Actually Use

• Run an eval: inspect eval my_eval.py --model anthropic/claude-3-5-sonnet-20240620 --limit 100 --log-dir ./logs

• View logs: inspect view (web UI) or open the Inspect panel in VS Code.

• Enable sandbox: --sandbox docker or --sandbox k8s:config.yml (domain/network policy via K8s).

• Repro tips: set --seed, use caching, and pin model versions.

📌 Resources to Learn More

• Docs/Getting Started/Install.

• Model Providers (OpenAI, Anthropic, Google, xAI, Mistral, HF, Azure, AWS, …).

• Tutorials and Examples (Hello World, GSM8K, HellaSwag, Math, Tools, CTF).

• Sandboxing (Docker built-in; Kubernetes/Proxmox adapters).

• Logs and Viewers (VS Code + Web).

• Inspect Evals collection (incl. SWE-bench variants).

• PyPI (latest version, license).

Want deeper tutorials? Subscribe to The Neural Blueprint for hands-on guides! 🫡 and follow us on YouTube, X and LinkedIn

Subscribe now

Nano Banana has since been hailed as the most impressive image generation model released since GPT-4o Image. It wowed users with powerful editing abilities, producing changes that were both consistent and remarkably realistic.

With two groundbreaking models now in the spotlight, the obvious question arises: which one is superior? That’s exactly what this article sets out to explore. We’ll compare Nano Banana’s capabilities against GPT-4o Image, drawing on results from OpenAI’s original announcement blog post and putting Nano Banana through the same set of tests.

Along the way, we’ll evaluate performance across 5 key areas:

• Instruction following

• Text rendering

• In-context learning

• World knowledge

• Photorealism

Let’s peel the banana 🍌🍌🍌

Useful Image Generation

Image generation in AI is nothing new. Long before language models began stealing the spotlight, we had pioneers like DALL·E and Stable Diffusion. These models could produce visually striking images, but “cool” didn’t always mean useful.

Earlier image generation models struggled with key limitations:

• Poor text rendering

• Lack of real-world knowledge

• Difficulty producing truly realistic images

OpenAI addressed these issues with the release of GPT-4o Image. More than just an image generator, it was powered by a multimodal model, giving it a deeper understanding of the world and the ability to create images that were both realistic and contextually relevant.

Not long after, Google entered the scene with Nano Banana, combining its Gemini Flash model with an image generator to deliver results that rivaled GPT-4o Image.

Text Rendering

Earlier image generation models like DALL·E 2 struggled to produce meaningful text. While they could create visually convincing images, their text rendering was often impractical and unreliable.

GPT-4o Image changed that by introducing near-flawless text generation, making it possible to create things like comic panels, posters, or images that seamlessly integrate written content.

Nano Banana also supports text rendering, and in many cases, it does so convincingly, placing text naturally within an image so that it looks as if it belongs in the real world.

However, compared to GPT-4o Image, Nano Banana still has some limitations. It can sometimes misalign text, produce gibberish, or generate words that aren’t clearly legible. Mathematical equations, in particular, pose a challenge, as they often fail to render accurately.

Instruction Following

One of the biggest breakthroughs in AI has been the ability of language models to follow instructions. The question is simple: if you give a prompt, can the model understand and execute it?

Since both Nano Banana and GPT-4o are built on top of large language models, they inherit strong instruction-following abilities. This allows users to generate images that closely align with their prompts, even when the requests are detailed or complex.

In practice, the two models perform quite similarly. Both tend to follow user instructions faithfully, with minimal drift, which sets them apart from older image generation models that often produced irrelevant or inconsistent results.

In-Context Learning

Like GPT-4o Image, Nano Banana is capable of in-context learning. This means it can use previous inputs, whether text or images, to guide the generation of new outputs. Just as large language models can use context to produce relevant text, these image models can leverage context to create consistent and meaningful visuals.

For example, both Nano Banana and GPT-4o Image were first asked to generate an image of a chainsaw. In the next prompt, they were asked to create an advertisement featuring a grandmother using the chainsaw. Both models handled the task with ease, combining textual and visual context to produce coherent, context-aware results.

In another case, when given a sketch of a building as input, each model was able to transform it into a realistic image of the same structure.

These examples highlight that when it comes to in-context learning, Nano Banana is on par with GPT-4o Image.

World Knowledge

One of the strengths of GPT-4o Image is its deep understanding of the world. With just a minimal prompt, it can often fill in the gaps and still generate accurate, meaningful images.

For example, GPT-4o was asked to create a poster of different whale species without being explicitly told which whales to include. Remarkably, it succeeded. Nano Banana demonstrates a similar level of world knowledge, producing results that show it understands context and real-world concepts.

It’s the combination of world knowledge and instruction-following that makes these image generation models so powerful and versatile.

That said, Nano Banana isn’t perfect. While its world knowledge is impressive, it occasionally makes mistakes. It misspelled the names of the Orca and Narwhal whales. Errors like these tend to stem from the same weaknesses Nano Banana shows in text rendering.

Photorealism

Like GPT-4o Image, Nano Banana is capable of generating highly photorealistic images that lack the typical “AI look.” In many cases, it becomes nearly impossible to distinguish an AI-generated image from a real photograph.

This level of realism, combined with strong instruction-following and world knowledge, allows both models to produce images that look authentic and believable.

Since both deliver impressive photorealism, choosing between them often comes down to user preference and the specific use case.

Can Nano Banana do Ghibli?

One of the coolest abilities of GPT-4o was its ability to generate Ghibli-styled images, so one question arises: can Nano Banana do the same? The answer is, not really.

Most of the Ghibli images generated by Nano Banana feel more tuned towards reality and give an oil painting style, rather than the chubby and cute style of Studio Ghibli.

This limitation of Nano Banana actually gives us a better understanding of these types of models, since it shows that everything depends on the data it was trained on.

Perhaps GPT-4o Image contained several Ghibli images, or Google, fearing lawsuits, just doesn’t want their model to copy the styles of animation companies.

Nano Banana as an Image Editor

One thing to note about Nano Banana is that it is not actually an image generator, or at least it is not advertised as one like other models such as ImageGen or GPT-4o Image. Instead, it is marketed as an image editing model, and it excels at this task.

It is capable of editing any image, whether a real-life photo or one generated by another model, while maintaining the style and consistency of the original without giving it that "AI feel."

For example, the image above was originally generated by GPT-4o Image with a snail rather than a sloth. Nano Banana was able to edit the image seamlessly, replacing the snail without introducing any noticeable changes. This highlights the strength of its editing capabilities.

By contrast, when given the same image, GPT-4o Image struggled with the task. Instead of replacing the snail with a sloth, it sometimes substituted it with a slot machine.

So, while Nano Banana may not be able to generate Ghibli-style art on its own, it can certainly edit a Ghibli image you’ve already created, preserving both consistency and style.

Conclusion

Although Nano Banana was marketed primarily as an image editing model, it has proven itself to be a highly capable image generator, often matching and sometimes even surpassing GPT-4o Image in certain tests.

Ultimately, the choice between the two depends on user preference and specific use cases. Some tasks favor GPT-4o’s generation strengths, especially in text rendering, while others benefit from Nano Banana’s powerful editing abilities.

In practice, the best approach may be to combine them: generate your base image with GPT-4o Image, then refine or modify it using Nano Banana. Both models are excellent options, and whichever you choose, you’re unlikely to go wrong.

The launch of GPT-5 has not gone entirely as OpenAI envisioned. While it has been described as having PhD-level intelligence, public sentiment has been mixed at best.

Early testers reported that the production model underperformed compared to private previews. Some users have openly asked for GPT-4o back, while others have criticized GPT-5 as little more than a clever hack, pointing to its internal router architecture. GPT-5 is one of the most polarizing models OpenAI has ever released.

In this article, we will set aside the controversy and focus on a different question: how does GPT-5 perform when measured against the benchmarks outlined in Microsoft’s Sparks of AGI paper? By the end, we hope to discover whether GPT-5 is still just producing sparks or if it has finally lit the flame of AGI.

Sparks Of AGI

When Sparks of AGI was released, it marked one of the boldest claims yet about large language models. The researchers argued that GPT-4 exhibited “sparks” of general intelligence, not full AGI but behaviors that seemed strikingly close.

What stood out about the paper was not just its conclusion, but the way the researchers tested GPT-4. They did not simply measure accuracy on benchmarks. Instead, they asked the model to perform across a wide spectrum of tasks, from coding and mathematics to multimodal reasoning and human interaction, looking for flexibility and depth of understanding.

For this article, I want to revisit those same tests. But instead of focusing only on GPT-4 as the Microsoft team did in 2023, I will compare two snapshots: GPT-4 as it was at the time of that paper, and GPT-5 as it stands today.

The areas I will be exploring are the same ones Microsoft highlighted:

• Multimodal and Interdisciplinary Composition – Can the model combine knowledge across domains, or describe ideas that bridge different fields?

• Coding – Does it demonstrate problem-solving as a programmer, not just surface-level pattern matching?

• Interaction with the World – Is the model capable of reasoning about real-world objects, contexts, and physical constraints?

By running GPT-5 through the same kinds of challenges, I hope to see whether we are still witnessing “sparks” of general intelligence or whether GPT-5 has taken us closer to something more.

Multimodal and interdisciplinary composition

A central goal of the Sparks of AGI researchers was to test how well the model could integrate knowledge across disciplines to generate new ideas. For example, they explored whether it could combine insights from poetry and physics to produce novel connections. This capacity is referred to as the model’s integrative ability.

Integrative ability

Let’s revisit those same tests to compare GPT-4’s integrative ability with GPT-5’s. In the original paper, GPT-4 outperformed its predecessor, so how does it measure up against its successor?

When GPT-4 was asked to write a play about infinite primes, it produced the one shown above.

I gave the same prompt to GPT-5, and it produced the play shown above. Even at a glance, the result feels more engaging: GPT-5 introduced characters like Euclidus and Skepticus, demonstrating not only historical awareness but also skill in wordplay and a clear grasp of Shakespearean style.

Still, let’s not be the judges ourselves. In the original paper, each model’s output was evaluated by a separate instance of GPT-4, which decided which response was better. We will follow the same approach here, except this time the evaluation will be done by a fresh instance of GPT-5.

We labeled GPT-5’s output as Student 1 and GPT-4’s output as Student 2. When a new instance of GPT-5 was asked to evaluate them, it awarded Student 1 an A and Student 2 a B. This suggests that the model’s ability to integrate knowledge across disciplines has improved.

Another test from the paper involved asking Mahatma Gandhi to write a letter to his wife, explaining his desire to support the electron in running for the U.S. presidential candidacy. Below is a side-by-side comparison of GPT-4 and GPT-5’s responses.

As in the previous example, we assigned student IDs to the models. GPT-4 was given the ID A and GPT-5 the ID B. We then asked a new instance of GPT-5 to compare their responses.

The new instance once again preferred GPT-5’s output and provided a full explanation of its reasoning.

Image generation beyond memorization

Another way to demonstrate the model’s ability to integrate knowledge across domains is through image generation. At the time, GPT-4 could not generate images directly, so the researchers asked it to produce TikZ code instead. The resulting images were then used to evaluate how well the model combined understanding from multiple disciplines.

The example above shows GPT-4’s output for generating images using letters. Now let’s apply the same prompt to GPT-5 and see what it produces.

GPT-5 produced images that were more creatively generated from the alphabet, demonstrating not only an understanding of letters but also of the objects themselves and how to combine the two in a more realistic way than GPT-4.

Coding

It is clear that one of GPT-5’s strongest selling points is its coding ability, a capability OpenAI is actively highlighting through collaborations with companies like Cursor. Compared to the time when Sparks of AGI was first released, the model’s coding performance is on an entirely new level.

The paper introduced several challenges for GPT-4:

• Solving coding problems

• Applying code to real-world scenarios

• Understanding existing code

• Reasoning about code execution

Currently GPT-5 is capable of doing all of this and more. This isn’t neccesarily a feature of GPT-5 since this improved coding ability has been steadily increasing in the GPT-4 era.

By leveraging GPT-5’s thinking, the model is able to outperform the o3 model on benchmarks such as SWE-bench and Aider Pyplot. Its strong reasoning ability also suggests that its coding skills still have room for further improvement.

Share

Interaction with the world

One of the key criteria highlighted in the original paper was GPT-4’s ability to use tools that interact with the real world. GPT-4 demonstrated successful tool use, but it struggled when the tools became more complex.

GPT-5 clearly surpasses that limitation. It can now use tools through function calling across domains such as airlines, retail, and telecommunications. Combined with its advanced reasoning capabilities, it is even able to handle the challenges of using highly complex tools.

Does GPT-5 address GPT-4 limitations

From our assessments, GPT-5 is clearly an improvement over its predecessor. The key question is whether it overcomes the limitations identified in the original Sparks of AGI paper. Those limitations included:

• Lack of planning in arithmetic and reasoning problems

• Lack of planning in text generation

Arithmetic has long been difficult for language models. GPT-5 shows clear progress. It solves more complex arithmetic problems, and with stronger reasoning it handles tasks that GPT-4 often could not solve without a calculator tool.

Reasoning has also advanced. Progress has been steady since the o1 generation, and GPT-5 goes further, outperforming o3 on key benchmarks.

GPT-4 often lacked planning in text generation, which led to hallucinations and inconsistencies. GPT-5 plans better, writes more coherently, and is more willing to say when it does not know the answer.

GPT-5 is Impressive, but it is not AGI

The question is simple: does GPT-5 show sparks of AGI as GPT-4 once did? The answer is yes. Is GPT-5 itself AGI? The answer is no. Even OpenAI has acknowledged that GPT-5 is not AGI, and the definition of AGI continues to shift with every new generation of large language models.

The AI industry has fueled the narrative that AGI is just around the corner, and the public has increasingly come to believe it.

This has created major public expectations, while at the same time diminishing appreciation for the real innovations in the field. If the narrative continues unchecked, it could even trigger another AI winter.

Subscribe now

Conclusion

GPT-5 may not have lived up to the wildest expectations fueled by industry hype, but it may well have lit the spark for what comes next. Just as GPT-4 laid the groundwork for many of the breakthroughs we have today, GPT-5 could be the foundation for future advancements in the GPT series.

Even if AGI remains out of reach for now, the progress is undeniable. GPT-5 demonstrates remarkable capabilities, and its practical applications are already reshaping how we work, create, and interact with technology. The pursuit of AGI may continue, but the real value lies in what these models can already do today.

Leave a comment

Traditionally, this task falls to highly trained court reporters, experts who transcribe speech in real time using specialized shorthand machines. B

ut with rapid advances in speech recognition technology, one question naturally arises: can modern Speech-to-Text models shoulder some of this responsibility?

Classic Speech-to-Text systems simply convert spoken words into plain text. Today’s models, however, such as Deepgram’s Nova-3 and AssemblyAI’s Universal-2, go far beyond basic transcription.

With features like speaker diarization and timestamps, these models can produce metadata-rich transcripts that mimic the structure, clarity, and reliability of a human court reporter.

In this article, we’ll build an AI court reporter using Deepgram’s Nova-3 model and explore:

• How metadata transforms raw transcripts into legally useful records

• How to work with the Deepgram API in Python

What does it take to build an AI court reporter?

Court proceedings, whether a trial, hearing, or deposition, are among the most critical processes in the legal system. During these events, a court reporter is responsible for capturing every spoken word verbatim. Skilled human reporters achieve high accuracy rates and understand complex legal jargon.

Their role goes beyond simple transcription. They must record who spoke, what was said, and when it was said. To replicate this with an AI system, we need a Speech-to-Text model capable of three essential things:

• Speaker diarization: Automatically identifies and labels different speakers in a conversation (e.g., Speaker 0, Speaker 1). This is crucial in courtroom settings where multiple participants, such as judges, lawyers, and witnesses, speak in turn.

• Timestamps: Tags each word or sentence with its start and end times. This allows precise alignment with the original audio, enabling features like searchable playback, real-time synchronization, and legally verifiable transcripts.

• Low Word Error Rate (WER): Ensures the AI produces highly accurate transcripts that can be trusted in legal contexts, reducing the risk of misinterpretation or misquotation.

Building an AI Court Reporter with Deepgram

Now that we understand the key capabilities required for an AI court reporter, let’s put them into action using Deepgram’s Nova-3 model, which supports both speaker diarization and timestamps.

We’ll build a simple Python CLI tool that takes either a local audio file or a URL to a court proceeding, then transcribes the audio while labeling each speaker and attaching precise timestamps.

Step 1: Install Dependencies

Before we start coding, make sure you have the necessary Python packages installed. We’ll use the Deepgram SDK for transcription and Rich to create a clean, formatted terminal output:

pip install deepgram-sdk rich

Step 2: Import Required Libraries

Next, we’ll import all the libraries needed for our script. We’ll use argparse to handle command-line arguments, the Deepgram SDK classes to interact with the Deepgram API, and components from Rich to format and display results in the terminal.

import argparse
import json
from itertools import groupby
from operator import itemgetter
from pathlib import Path

from deepgram import (
    DeepgramClient,
    PrerecordedOptions,
    FileSource,
)

from rich.console import Console
from rich.table import Table
from rich.progress import Progress

Step 3: Define the Transcription Function

We’ll now create a function that sends audio to Deepgram for transcription. This function accepts either a local file path or a URL, then uses the Nova-3 model with speaker diarization and timestamps enabled. It returns the raw transcription JSON data from Deepgram’s API.

def transcribe_audio(audio_path, api_key):
    dg = DeepgramClient(api_key)

    opts = PrerecordedOptions(
        model="nova-3",
        language="en",
        smart_format=True,
        diarize=True
    )

    if audio_path.startswith("http://") or audio_path.startswith("https://"):
        source = {"url": audio_path}
        res = dg.listen.rest.v("1").transcribe_url(source, opts)
    else:
        with open(audio_path, "rb") as f:
            payload: FileSource = {
                "buffer": f.read()
            }
            res = dg.listen.rest.v("1").transcribe_file(payload, opts)

    return res

Step 4: Format the Transcription

Once we receive the raw output from Deepgram, we use the build_diarized_transcript function to organize it into a clean, readable format. This involves grouping words by speaker, extracting their start and end timestamps, and combining them into speaker-specific segments.

def build_diarized_transcript(res):
    words = res.results.channels[0].alternatives[0].words
    diarized_segments = []

    for speaker, group in groupby(words, key=itemgetter("speaker")):
        group = list(group)
        start = group[0]["start"]
        end = group[-1]["end"]
        text = " ".join([w["punctuated_word"] for w in group])
        diarized_segments.append({
            "speaker": f"Speaker {speaker}",
            "start": start,
            "end": end,
            "text": text
        })
    return diarized_segments

Step 5: Display the Transcripts

We’ll use Rich to neatly format and display the rendered transcripts in a table for better readability:

def print_diarized_table(diarized_segments):
    table = Table(show_header=True, header_style="bold magenta")
    table.add_column("Start", style="cyan")
    table.add_column("End", style="cyan")
    table.add_column("Speaker", style="green")
    table.add_column("Text", style="white", overflow="fold")

    for seg in diarized_segments:
        start_time = f"{seg['start']:.2f}s"
        end_time = f"{seg['end']:.2f}s"
        table.add_row(start_time, end_time, seg["speaker"], seg["text"])

    console = Console()
    console.print("\n[bold underline]Diarized Transcript[/bold underline]\n")
    console.print(table)

Step 6: Run the program

Let’s make the script runnable so we can execute it directly from the command line. We’ll use argparse to accept the audio file path, the Deepgram API key, and an optional output path for saving the raw Deepgram JSON.

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Transcribe audio with Deepgram Nova-3 and diarization.")
    parser.add_argument("audio", help="Path or URL to the audio file")
    parser.add_argument("--api_key", required=True, help="Deepgram API key")
    parser.add_argument("--save_json", help="Optional path to save raw Deepgram JSON output")
    args = parser.parse_args()

    console = Console()

    with Progress() as progress:
        task = progress.add_task("[cyan]Transcribing audio...", total=None)
        res = transcribe_audio(args.audio, args.api_key)
        progress.update(task, completed=1)

    if args.save_json:
        with open(args.save_json, "w") as f:
            json.dump(res.to_dict(), f, indent=2)

    diarized_segments = build_diarized_transcript(res)
    print_diarized_table(diarized_segments)

Testing the AI Court Reporter

To test the AI Court Reporter, we will use a clip from Better Call Saul to see how the model can capture both the time stamps and label speakers.

All we have to do is get an audio version of the clip and our Deepgram key, then pass it to the application like this:

python main.py "Better_Call_Saul.mp3" --api_key YOUR_DEEPGRAM_API_KEY

This will give us the following output:

You can get the source code on GitHub: Neurl-LLC/Court-Transcripts-With-Nova

Drawbacks of Using AI for Court Reporting

AI-based transcription is not without limitations. Some of the most common challenges include:

• Overlapping speech: When multiple speakers talk at the same time, AI often struggles to separate their voices accurately.

• Specialized legal terminology: Court proceedings often contain Latin phrases, case law references, and technical legal terms that speech-to-text models may not recognize without domain-specific training.

• Contextual ambiguity: AI lacks the human judgment to interpret sarcasm, implied meaning, or nuanced tone shifts, which can sometimes be relevant in court.

• Legal restrictions: Certain courts do not allow digital devices or permit the recording of audio, making AI transcription impossible in those settings.

For now, these drawbacks mean AI is best used as a supplementary court reporter, working alongside humans to improve efficiency and accessibility without replacing the need for human oversight.

Conclusion

By using speech-to-text models that provide extra metadata such as timestamps and diarization, we unlock untapped potential not only in the legal industry but in any field that requires high-quality text data beyond plain transcription. When combined with other AI technologies, this enables capabilities such as:

• Automated redaction using language models

• Speaker identification through voice recognition

• Advanced search systems for fast, precise retrieval

In the legal industry, these capabilities can streamline case preparation, improve evidence review, and ensure greater accuracy in legal documentation.

📢 What Is Fenic?

“Think pandas/Polars, but with semantic.extract, semantic.join, semantic.map baked in.” — Kostas Pardalis, Fenic creator (MLOps Slack launch thread).

Under the hood, Fenic is an opinionated, PySpark-inspired query engine built from scratch for AI and agentic applications.

Fenic treats LLM workflows as built-in DataFrame primitives instead of attaching them to external systems. This approach might make it much easier to build and run your workflows.

Here are the primary selling points:

• Semantic operators (analyze_sentiment, classify, extract, group_by, join, predicate) are first-class.

• Native unstructured types: Markdown, transcripts, JSON, long-form text with auto-chunking.

• Batch and retry layer with token counting and cost metrics built-in.

• Multi-provider (OpenAI, Anthropic, Gemini) and local/cloud execution modes.

• Familiar API: lazy DataFrame, SQL support, PySpark-like chaining.

• Languages: Python 87%, Rust 13%. Rust core already powers Arrow-native execution, with design choices influenced by Polars for speed and efficiency.

Wes McKinney (pandas creator) publicly endorsed the concept (“a natural evolution of the DataFrame abstraction”).

⚡ Quick Spin-Up: Podcast → Segments → Extract → Summaries (Fenic)

Here’s a quick example of how to analyze and extract summaries from a podcast episode with Fenic:

pip install fenic              # Python 3.10-3.12, Fenic v0.2.1
export OPENAI_API_KEY=...      # Add your Op

# -------------------------
from pathlib import Path
from pydantic import BaseModel, Field
import fenic as fc

# 1. ---- Define schemas for structured extraction ----
class SegmentSchema(BaseModel):
    speaker: str = Field(description="Who is talking in this segment")
    start_time: float = Field(description="Start time (seconds)")
    end_time: float = Field(description="End time (seconds)")
    key_points: list[str] = Field(description="Bullet points for this segment")

class EpisodeSummary(BaseModel):
    title: str
    guests: list[str]
    main_topics: list[str]
    actionable_insights: list[str]

# 2. ---- Init a Fenic session with a model alias ----
config = fc.SessionConfig(
    app_name="podcast_quickspin",
    semantic=fc.SemanticConfig(
        language_models={
            "mini": fc.OpenAIModelConfig(model_name="gpt-4o-mini", rpm=300, tpm=150_000)
        }
    ),
)
session = fc.Session.get_or_create(config)

# 3. ---- Load raw transcript/metadata as strings ----
data_dir = Path("data")  # put your JSON/text here
transcript_text = (data_dir / "transcript.json").read_text()
meta_text       = (data_dir / "meta.json").read_text()

df = fc.DataFrame({"meta": [meta_text], "transcript": [transcript_text]})

# 4. ---- Extract structured metadata & segment the transcript ----
processed = (
    df.select(
        "*",
        fc.semantic.extract("meta", EpisodeSummary, model="mini").alias("episode"),
        # Chunk transcript then extract per-chunk info
        fc.semantic.chunk("transcript", max_tokens=1200).alias("chunks"),
    )
    # Explode chunks to rows (one row per chunk)
    .explode("chunks")
    .select(
        fc.col("chunks").alias("chunk"),
        fc.semantic.extract("chunk", SegmentSchema, model="mini").alias("segment"),
    )
)

# 5. ---- Abstractive recap per speaker/segment & global summary ----
final = (
    processed
    .select("*",
        fc.semantic.map(
            "Summarize this segment in 2 sentences:\n{chunk}"
        , model="mini").alias("segment_summary")
    )
    .group_by(fc.col("segment.speaker"))
    .agg(
        fc.semantic.map(
            "Combine these summaries into one clear paragraph:\n{segment_summary}"
        , model="mini").alias("speaker_summary")
    )
)

final.show(truncate=120)

# Optional: write to parquet/csv
final.write.parquet("podcast_summaries.parquet")

session.stop()

Tips to Adapt Quickly

• Different providers: swap OpenAIModelConfig for AnthropicModelConfig, etc.

• Bigger files: bump max_tokens or chunk size; Fenic batches/streams for you.

• Eval pass: add another select() with a classifier prompt to tag “quality: good/needs fix”.

• Cost guardrails: set max_tokens_per_call or inspect session.metrics() after run.

Need a variant for YouTube transcripts or research PDFs? Simply adjust the loader and schemas, and the pipeline shape will remain unchanged.

🔥 Why You Should Care

• Declarative pipelines → push your ETL and inference into the same DAG.

• Cheaper evaluation loops → token and cost metrics are first-class.

• Semantic joins → fuzzy “does this paper help my research question?” join in one line via semantic.join

• Structured extraction to Pydantic → easier downstream analytics, eval & labeling.

• Agent synergy → pre-batch heavy reasoning offline, feed lean contexts to online agents.

📉 Gotchas and Caveats

1. Toolchain friction: Installing Fenic via pip is straightforward, but developing new features requires the Rust toolchain (rustc, cargo, maturin), which isn’t fully documented yet.

2. Young ecosystem: Core support includes Arrow, CSV, and Parquet, with native connectors for Snowflake, BigQuery, and S3 expected soon.

3. Operational maturity: No proven large-scale benchmarks published; cloud engine still alpha.

4. Docs still sparse: docs.fenic.ai exists but is thin; many API details live only in README/examples. A more structured documentation system is in the works, with an MCP server example.

5. Single-node today: No distributed executor yet; large corpora need chunked runs or Spark/Polars fallback.

🧑‍⚖️ Final Verdict: 4/5

Rating: ⭐⭐⭐⭐☆ (4/5)

Leave a comment

Fenic nails a gap nobody else covers: treating LLM inference as a native DataFrame primitive. If your team already loves SQL/PySpark and spends hours duct-taping looped API calls, Fenic will feel like a super-power.

Ship it if…

• You batch-process lots of text and need semantic joins/extractions weekly.

• You’re prototyping an RAG/agent pipeline and want repeatable, cost-aware ETL.

• You can tolerate early-project rough edges and contribute fixes upstream.

Hold off if…

• You need petabyte-scale, distributed compute today.

• Your workloads are real-time, sub-second.

• You require enterprise auth/row-level security out of the box.

📌 More Resources

• Docs and Quickstarts → https://docs.fenic.ai/latest/

• GitHub repo (Apache-2.0) → https://github.com/typedef-ai/fenic

• Blog intro (“PySpark-inspired DataFrame for AI”) (June 18, 2025) → https://www.typedef.ai/blog/fenic-open-source

• Example gallery → examples/ folder on GitHub

• Author Q&A in MLOps Slack → link

Want deeper tutorials? Subscribe to The Neural Blueprint for hands-on guides! 🫡

Subscribe now

A recurring theme across talks was how these organizations are using LangGraph to build robust, production-grade agentic systems.

With so many insights shared about LangGraph during the conference, now is the perfect time to take a closer look at what LangGraph is, how it works, and the design philosophy behind it.

LangGraph was introduced in early 2024 as a separate library built on top of LangChain. While LangChain helped developers build simple, linear agentic workflows, it fell short when it came to more complex agentic workflows, especially those involving loops or cycles, which are common in real-world agent interactions.

To address this limitation, the LangChain team created LangGraph: a framework that introduces graph-based agentic workflows for more flexible and powerful agent design.

In this article, we’ll dive deep into LangGraph and explore how it brings core computer science concepts like state machines to life in the world of AI agents. Specifically, we’ll cover:

• How LangGraph models agentic workflows as graphs

• How to build these workflows using LangGraph’s API

• How to use its high-level abstractions to simplify development

• Real-world examples of how companies are using LangGraph in production

Let’s get started. 🚀

Understanding State Machines

Before we had modern computers, we had state machines, more specifically, Finite State Machines (FSMs). These were simple models of computation that operated using a finite set of states. Each state could transition to another based on specific inputs or actions.

Let’s take a real-world example: a subway turnstile.

Imagine walking into a subway station and approaching a turnstile that only unlocks when you insert a coin. This system has two states:

• Locked

• Unlocked

Here’s how it works:

• If you try to push the turnstile without inserting a coin, it stays locked.

• If you insert a coin, the turnstile transitions to the Unlocked state.

• Once you push through, it automatically returns to the Locked state.

These movements between states, triggered by actions like inserting a coin or pushing the turnstile, are called state transitions. The turnstile system as a whole is a simple example of a state machine.

You can also visualize state machines as graphs, where:

• States are represented as nodes

• Transitions between states are represented as edges

This simple model of computation is what LangGraph builds upon to model AI agent interactions as state machines.

Agentic State Machines

Now that we understand traditional state machines, let’s explore the idea of agentic state machines.

In the previous section, we saw that state transitions in a finite state machine are driven by predefined conditions. For example, when a user inserts a coin and pushes a turnstile, the machine transitions from a Locked to an Unlocked state based on those fixed inputs.

But in an agentic state machine, transitions aren't hardcoded. Instead, an agent decides which transition to take based on the current state and context. This introduces flexibility and autonomy into the state machine model, which is essential for building intelligent systems.

An agentic state machine reimagines the traditional model using the following mapping:

• Agents and tools as nodes

• Agent decisions as transitions

• Conversations as states

Agents and Tools as Nodes

In an agentic state machine, each node in the graph represents either an AI agent or a tool. These components are independent:

• Agents are typically language models capable of reasoning and decision-making.

• Tools are external functions or systems (like a calculator or search engine) that can be invoked when needed.

This modularity means each component can operate on its own, but the real power emerges when they are connected meaningfully through transitions.

Agent Decision as Transitions

To move between nodes, we use edges, which represent possible transitions. In an agentic state machine, these transitions are driven by the agent's decisions, not by hardcoded logic.

There are two kinds of edges:

• Conditional edges (represented with dotted lines): The agent chooses whether or not to follow these paths based on its understanding of the current state.

• Definite edges (represented with solid lines): These are fixed transitions that are always followed when reached.

In the illustration above, you’ll notice that the agent is connected to multiple tools via conditional edges, allowing it to decide which tool to use based on the conversation. Once a tool is used, it typically returns to the agent via a definite edge.

Conversation as States

So far, we’ve defined our nodes (agents and tools) and transitions (edges). But what determines when an agent should make a decision? That’s where the state comes in.

In agentic state machines, the state of the system is driven by the conversation between the agent and the user. As the dialogue progresses, each message updates the state. This context is what the agent uses to decide its next move.

For example, in the image above, the state is updated with the user message: "What is 2 times itself?"

The agent interprets this diagram and decides to use the calculator tool. The tool returns a result, which the agent can then act upon, perhaps replying to the user or performing another computation.

At every step, the conversation evolves, and with it, the state of the system. The agent uses this evolving state to determine transitions and actions.

Now that we understand the theory behind agentic state machines, let’s move on to building one using LangGraph.

👀 Recommended:

Build an Agentic State Machine with LangGraph

To build our agentic state machine in LangGraph, let’s start by installing the required libraries. We would need the following: LangGraph and LangChain alongside OpenAI dependencies.

This would enable us to use OpenAI as our model provider, but you can use any model provider of your choice.

pip install langgraph langchain[openai]

You can now add your OpenAI API key to the environment variable OPENAI_API_KEY.

Defining the States

With the setup complete, we can now start building our agentic state machine by defining its states.

from typing import Annotated, TypedDict
from langgraph.graph.message import add_messages

class State(TypedDict):
    messages: Annotated[list, add_messages]

Here, we define our states as a dictionary that stores the messages exchanged between the user and the agent. Each node will add messages to this state as it transitions from one node to another.

Initialize the Graph, Model, and Tool

Now that we’ve defined our states, the next step is to set up the Graph, Agent, and Tool. The Graph represents our state machine. It brings together the agents, tools, and transitions that define the agentic workflow.

We build the graph using the StateGraph class, which takes the state we defined earlier as input.

from langgraph.graph import StateGraph

# Build the graph
graph_builder = StateGraph(State)

Next, we need to define the language model that will serve as the reasoning engine for our agent.

from langchain.chat_models import init_chat_model

# Initialize the chat model

llm = init_chat_model("openai:gpt-4.1")

After setting up the model, the next step is to define a tool. We’ll create a simple calculator tool that the agent can use.

# Define a simple calculator tool
def calculator(operation: str, a: float, b: float) -> float:
    """A simple calculator function that performs basic arithmetic operations.
    Args:
        operation (str): The operation to perform. Can be "add", "subtract", "multiply", or "divide".
        a (float): The first number.
        b (float): The second number.
    Returns:
        float: The result of the operation.
    Raises:
        ValueError: If the operation is unknown.
    """

    if operation == "add":
        return a + b
    elif operation == "subtract":
        return a - b
    elif operation == "multiply":
        return a * b
    elif operation == "divide":
        return a / b
    else:
        raise ValueError("Unknown operation")

With our tool defined, we can now bind it to the language model. This enables the model to understand how to interact with the tool during execution.

# Bind the calculator tool to the LLM

tools = [calculator]

llm_with_tools = llm.bind_tools(tools)

Adding the Nodes

With the tools and language model defined, we can now add them as nodes in the graph. We'll start by adding the language model, since it'll be using tools that essentially make it an agent.

# Define the agent function that uses the LLM with tools
def agent(state: State):
    return {"messages": [llm_with_tools.invoke(state["messages"])]}

# Add the agent node to the graph
graph_builder.add_node("agent", agent)

We can visualize the graph and see that it currently contains only one node.

So let’s add the tool as a node as well.

from langgraph.prebuilt import ToolNode

# Add the tool node to the graph

tool_node = ToolNode(tools=tools)

graph_builder.add_node("tools", tool_node)

We used the ToolNode class to simplify the process of creating a node for the tool.

We now have two nodes in our graph, but they’re not yet connected. Let's link them using edges.

Adding Edges

To connect the agent to the tool node in LangGraph, we use a conditional edge because we want the agent to decide whether to transition the state to the tool or simply end.

Here is how it works: we define a conditional function that checks whether the agent calls a tool, in this case the calculator. If it does, the state transitions to the calculator node where the tool is executed.

If the agent doesn’t call a tool, the state transitions to the end of the graph, returning the agent’s response.

from langgraph.graph import END, MessagesState

def condition(state: MessagesState):
    messages = state["messages"]
    last_message = messages[-1]
    if last_message.tool_calls:
        return "tools"
    return END

With the conditional function defined, we can now create the edge.

# Add a conditional edge to the agent node that checks if tools are needed
graph_builder.add_conditional_edges(
    "agent",
    condition,
)

Here's a visual representation of the graph so far.

A conditional edge exists between the agent and the tool, but if we transition to the tool node, how do we return to the agent node? This is where we introduce a new edge. Unlike the first, this one isn’t conditional; it’s a definite transition.

# Add an edge from the tools node to the agent node

graph_builder.add_edge("tools", "agent")

The graph is nearly complete; we simply need to incorporate one final element. Currently, the graph doesn’t have a starting node. We will add a start node and connect it to the agent node.

from langgraph.graph import START

# Add the starting edge to the agent node

graph_builder.add_edge(START, "agent")

This means the agent is the first node to receive the state before any other part of the graph.

With that, our graph is complete. The final step is to compile it.

# Compile the graph

graph = graph_builder.compile()

Interacting with the Agentic State Machine

With the graph compiled, we can now test the agentic state machine by using the graph’s invoke method.

# Invoke the graph with a user message

result = graph.invoke({"messages": [{"role": "user", "content": "What is a million times four hundred and 8?"}]})

print("Assistant: ", result["messages"][-1].content)

In the code above, we provided the agentic state machine with the initial input:

• “What is a million times four hundred and eight?”

The state machine should return a response along the lines of:

• “A million times four hundred and eight is 408,000,000.”

Now, let’s analyze the state transitions that occurred within the agentic state machine.

Here’s how the state transitioned:

1. The user asked: "What is a million times four hundred and eight?" This became the initial state at the start node, which then transitioned to the agent node.

2. The agent analyzed the input and determined it needed to use the calculator tool, so it initiated a tool call.

3. The conditional function we defined detected the tool call, causing the state to transition to the tool node where the calculator was executed.

4. The result of the calculation was passed back to the agent.

5. The agent updated the state with its final response and transitioned to the end node.

We’ve now seen how to bring an agentic state machine to life using LangGraph. While this low-level approach gives you full control over the system, it can be quite involved and requires more effort to build and maintain.

To make things easier, LangGraph also provides a set of high-level APIs that simplify the process of building agentic systems.

LangGraph High-Level APIs

LangGraph provides several high-level APIs that simplify the process of building AI agents. These abstractions allow you to focus more on behavior and logic, rather than the low-level details of graph construction. Some of the most notable include:

• LangGraph Prebuilt

• LangGraph Supervisor

• LangGraph Swarm

• LangGraph MCP Adapter

LangGraph-Prebuilt

LangGraph Prebuilt offers ready-to-use components for building agentic systems. Instead of manually working with graph concepts like nodes and edges, you use higher-level abstractions that encapsulate common agent patterns.

For example, here’s how the graph we previously built from scratch can be created using the Prebuilt API:

from langgraph.prebuilt import create_react_agent

def calculator(operation: str, a: float, b: float) -> float:
    """A simple calculator function that performs basic arithmetic operations.
    Args:
        operation (str): The operation to perform. Can be "add", "subtract", "multiply", or "divide".
        a (float): The first number.
        b (float): The second number.
    Returns:
        float: The result of the operation.
    Raises:
        ValueError: If the operation is unknown.
    """

    if operation == "add":
        return a + b
    elif operation == "subtract":
        return a - b
    elif operation == "multiply":
        return a * b
    elif operation == "divide":
        return a / b
    else:
        raise ValueError("Unknown operation")

agent = create_react_agent(
    model="openai:gpt-4.1",  
    tools=[calculator],  
    prompt="You are a helpful assistant"  
)

# Run the agent
agent.invoke(
    {"messages": [{"role": "user", "content": "What is a million times four hundred and 8?"}]}
)

This code uses the create_react_agent function to create the agent. LangGraph Prebuilt also provides additional components, including tools for managing agent memory and state more effectively.

LangGraph Supervisor

If you're building a supervisor-style agentic system, where one agent oversees and coordinates the actions of other agents, then LangGraph Supervisor is the ideal API to use. Instead of building the entire system from scratch, you can use this high-level abstraction to simplify the process.

To use it, you need to install the langgraph-supervisor library. It can be used alongside the langgraph-prebuilt library.

LangGraph Swarm

LangGraph Swarm is a high-level library designed for building collaborative agent systems, where multiple agents work together and can hand off tasks to one another without a central supervisor.

LangGraph MCP Adapters

When building agentic graph systems that need to interact with external tools via the Model Context Protocol (MCP), the langchain-mcp-adapters library simplifies the entire process.

It essentially acts as an MCP client for your agentic graph system, enabling it to communicate with both local and remote MCP servers.

Who is Using LangGraph in Prod?

Every developer asks one question when they encounter a new tool: Is it used in production? Some of the largest companies in the tech industry indeed use LangGraph in production to build AI agents. Let’s explore a few examples.

Uber

Uber’s Developer Platform team built AI agents using LangGraph to enhance the developer experience.

These agents can generate unit tests and ensure adherence to Uber’s internal coding standards, streamlining development and improving code quality.

LinkedIn

To improve recruiter efficiency, LinkedIn has developed an advanced AI Hiring Agent powered by LangGraph.

The team designed the agent to streamline the hiring process through automated functions, including performing conversational searches to understand recruiter needs and accurately matching candidates to open roles.

Elastic

Elastic uses LangGraph to orchestrate its AI agents for threat detection scenarios, significantly reducing labor-intensive SecOps tasks.

The integration improves Elastic's Generative AI features, such as the Elastic AI Assistant and Automatic Import, enabling them to understand complex security scenarios, generate queries, and craft accurate data integrations for streamlined security analytics.

Conclusion: LangChain—The Agentic State Machine

LangGraph is quickly becoming the go-to framework for building effective AI agents. It really shines because it takes proven computer science ideas like state machines and graphs, and uses them to create modern AI agents that are much tougher and better at tackling real-world problems.

Although LangChain has received significant criticism, LangGraph's design has earned widespread praise.

The big difference? Instead of agents relying on prompts to make decisions (which was a common criticism of LangChain), LangGraph uses clear, controllable, and easy-to-monitor graphs. This makes the agents far more predictable and reliable.

Think of this article as just a quick intro; if you want to dive deep and truly understand LangGraph, definitely check out its official documentation.

We created 🛠️BuildAIers Toolkit to help you cut through the noise and build smarter with the right tools, stacks, and strategies. On bi-weekly Tuesdays, we deliver hands-on insights to help you evaluate, integrate, and scale AI tech that actually works.

📌 Key Goals for Every Post:

✅ Help AI builders make informed, confident tooling decisions

✅ Highlight workflows, tradeoffs, and real-world use cases

✅ Encourage a builder-led conversation around what tools are shaping the future of AI.

Used this tool in production? Tell us how it performed—and what you’d do differently. ⬇️

Leave a comment

Why? Because LangGraph lifts agentic workflows from prompt spaghetti to explicit, traceable state machines.

Here are the core parts of LangGraph in 2025:

⏩TL;DR:

✅ Use it for multi-step agent graphs, fine-grained RAG, model-agnostic tooling, or deep tracing via LangSmith.

🚫 Skip it if you only need a one-vendor chat bot or you can’t spare the LCEL learning curve.

⭐ Verdict: 4.5/5—the most disciplined path to production-grade agents today.

📢 How is LangChain Doing in 2025?

LangGraph is a standalone library (and now a managed platform) that models agent workflows as explicit state-machine graphs. Each node is an LLM or tool; edges define deterministic or agent-chosen transitions—yielding observability and error-handling you rarely get from prompt-only agents.

• v0.4 (Apr 29 2025) added automatic interrupt surfacing for safer long-running graphs.

• LangGraph Platform GA (May 14 2025) lets teams deploy, autoscale and monitor stateful agents in one click.

• Works with 35 + model back-ends (OpenAI, Gemini, Claude, Bedrock, Ollama) via LangChain adapters

• Powers prod agents at Uber (dev-QA bot), LinkedIn (AI Hiring Agent), Elastic (threat-intel ingest), and more.

• Optional LangSmith tracing for latency, token-cost & evals.

⚙️ Key Architecture Upgrades to LangGraph in 2025

⚙️ How Does LangChain Work? (2025 Edition)

1. Define States = Conversation history (list of messages).

2. Add Nodes = Agents or Tools (LLM with bound tools, or a standalone function).

3. Wire Edges = Transitions.

   • Conditional dotted edges (LLM decides).

   • Definite solid edges (always taken).

4. Compiler turns your graph into an async executor with built-in tracing.

The Graph = Pure Python: no YAML, no DSL. Use LCEL blocks and type hints.

🚀 Quick Spin-Up (Agent + Calculator Tool)

# pip install -U langgraph langchain[openai] langgraph-prebuilt faiss-cpu

import os

from langgraph.prebuilt import create_react_agent

from langchain_openai import ChatOpenAI

os.environ["OPENAI_API_KEY"] = "sk-..." # ⚠️ set yours

os.environ["LANGCHAIN_TRACING_V2"] = "true" # optional: LangSmith traces

# 1-liner tool

def calc(operation:str, a:float, b:float): # simple tool

return {"add":a+b,"sub":a-b,"mul":a*b,"div":a/b}[operation]

agent = create_react_agent(

model="openai:gpt-4o",

tools=[calc],

prompt="You are a helpful and concise math tutor."

)

result = agent.invoke({"messages":[{"role":"user",

"content":"What is 42*999?"}]})

print(result["messages"][-1].content)

The helper builds a graph with:

1. Agent node (GPT-4o)

2. Tool node (calculator)

3. Conditional edge: agent → tool if function call detected

4. Definite edge: tool → agent with result

Interrupts, retries and LangSmith traces are automatic.

🔥 Why MLOps Engineers Care

• Deterministic control flow: Graph edges make loops, branches and error paths explicit.

• Better observability: LangSmith + OpenTelemetry give step-level traces and spend.

• Vendor freedom: Swap GPT-4o for Gemini 1.5, Claude 3 Opus or local Ollama without code rewrites.

• Production proof: Uber’s developer-productivity agents claim 21 k engineer-hours saved; LinkedIn’s AI Hiring Assistant runs on LangGraph too.

• MCP adapter: plug external tool servers (e.g., DeepWiki, private RAG APIs) in one line.

📉 Gotchas & Caveats

1. Learning curve: need to grok state machines and LCEL syntax.

2. Extra layer: simple Response-API bots deploy faster + each node/hop adds ~tens of ms vs straight SDK.

3. Ecosystem split: still relies on LangChain’s core; if you dislike LCEL, you may prefer DSPy or Pydantic-AI.

4. Over-engineering risk: for plain Q&A, graphs are unnecessary overhead.

🗣️ Community Pulse

Lily Nicholls (Jun 11th)

“… I want the agent to be able to decide whether additional retrieval steps are needed or to tweak it's generated response based on a user's input.”

Sammer Puran (Jun 12th)

“Langgraph works great for this process. You can add tools to retrieve the information, rewrite the query if the retrieved documents are not relevant, rank the documents, and generate the response. Also, you can write flows as directed graphs and visualize them”

Lily Nicholls (Jun 12th)

“Great thank you! I am trying to decide between LangGraph & ADK”

💡 Real-World Use Case: Uber’s Dev-Rel Copilot

1. Graph: Supervisor agent → code-generator sub-agent → test-writer sub-agent

2. Flow: CL diff → graph generates unit tests & standards feedback

3. Impact: 21 k dev-hours saved in 90 days (LangChain Interrupt keynote)

📊 How Does LangGraph Stack Up Against Alternatives in 2025

🧑‍⚖️ Final Verdict: 4.5 / 5 — Graphs > Prompts for Complex Agents

⭐ Rating: ⭐⭐⭐⭐☆ (4.5 / 5)

Ship it if…

• You orchestrate multi-step agents, need reliable retries or must mix many tools.

• You need loops, parallelism, or supervisor patterns.

• Vendor-agnostic strategy or on-prem LLMs matter.

• Granular tracing, cost guards and interrupt safety are non-negotiable.

Hold off if…

• You’re pushing a single-provider chatbot with a 1-week MVP deadline.

• One-vendor, single-prompt chatbots meet your roadmap.

• Team can’t spare time to ramp-up on LCEL/graph mental-model.

• Ultra-low latency (<100 ms) matters more than complex logic.

📌 More Resources

• LangGraph v0.4 Interrupts changelog

• LangGraph docs

• Case-study hub

• LangGraph Platform GA blog

• MCP adapter docs

• Uber Interrupt talk (21 k hours saved):

• LinkedIn Hiring Agent deep-dive:

Enjoy this? Forward it to an AI‑builder friend or share it on X with #NeuralBlueprint. 🫡

At the time, language models were isolated systems. They took in text and returned text, nothing more. There were no built-in tools, no retrieval mechanisms, no memory. LangChain emerged to solve this problem by offering a way to orchestrate LLMs with tools, prompts, memory, and retrieval pipelines.

But 2022 feels like a lifetime ago.

Since then, the LLM landscape has changed dramatically. Language models no longer operate in isolation. Major providers now offer built-in capabilities like tool use, function calling, and external data access. Innovations such as OpenAI’s function calling and Anthropic’s Model Context Protocol (MCP) have drastically reduced the limitations that LangChain originally set out to overcome.

So the question now is: Is LangChain still needed in 2025?

In this article, we’ll explore that question by comparing LangChain to the latest capabilities of the OpenAI API. With the introduction of the Response API, OpenAI has brought orchestration features directly into the platform.

This article will help you understand how far LLM tooling has come and whether LangChain still deserves a place in your toolbox.

Here’s what we’ll cover:

• Prompt management: How prompt handling compares between LangChain and OpenAI

• Tool usage: How both platforms enable agentic behavior

• Retrieval-Augmented Generation (RAG): LangChain’s stack compared to OpenAI’s native support

• Multi-model support: How each handles access to different model providers

• Structured outputs: LangChain’s output parsers versus OpenAI’s built-in response formatting

Let’s dive in.

🌱 LangChain's Origin Story

Back in 2022, when language model providers like OpenAI and Cohere first made their APIs publicly available, their capabilities were limited. You could send in a prompt and get back a response. Text in, text out. That was it.

While the APIs were simple, the research community was already exploring much more. Papers, tweets, and GitHub repositories were demonstrating how LLMs could be extended to perform advanced tasks like using calculators, searching the web, or reasoning step by step simply through clever prompting.

Research like “Self-Ask” had working implementations, and developers were beginning to realize the hidden potential of LLMs. But all this knowledge was scattered across the internet.

That’s when Harrison Chase stepped in. He began collecting those ideas and turning them into a unified framework. What he was working on would soon become LangChain, the first full-featured LLM orchestration library.

At launch, LangChain supported just two providers: OpenAI and Cohere. But it allowed developers to do more than just send prompts. It enabled models to perform tasks like web search, math calculations, and more. These were two areas where LLMs struggled the most at the time: hallucinations and arithmetic.

Langchain didn’t get much attention until early 2023, when more developers jumped into the LLM space and quickly ran into the same limitations: hallucinations, weak reasoning, prompt engineering complexity, and lack of tooling.

By this time, LangChain had rapidly expanded its feature set:

• Composable chains for combining multiple LLM calls and tools

• Retrieval-Augmented Generation (RAG) support

• Agentic workflows like ReAct and MRKL

• Memory integration for maintaining conversational context

• Embedding model support for search and retrieval

For many developers, LangChain became the easiest way to start building serious LLM applications. It was a dream toolkit for rapid experimentation.

But as LangChain grew, so did the criticism. Many in the software community began to argue that LangChain was too complex, too abstract, and not production-ready. Some preferred the simplicity and control of using the OpenAI SDK directly.

On Reddit, Twitter, and forums, there was no shortage of frustration over its design choices.

This article isn’t about those criticisms.

Instead, we want to focus on a bigger question: Is LangChain still necessary in 2025?

LangChain was designed to orchestrate the complex workflows that LLMs couldn’t handle alone. But now, model providers themselves offer orchestration capabilities, including built-in function calling, retrieval, memory, and even agent-like behaviors.

So, where does LangChain fit in today’s LLM stack?

That’s what we’re here to explore.

🗂️ Prompt Management

One of the selling points of LangChain at the time was its ability to assist developers with prompting. LangChain had built-in prompts that could help developers achieve their goals, but now, come 2025, the act of “prompt engineering” is almost non-existent. This is mostly because of how language models have improved. Now, with just zero-shot prompting, you can get an LLM to achieve your task with high accuracy.

When LangChain first came out, its maintainers drew prompting techniques from the research community and carefully crafted new prompts. At the time, before OpenAI released the Chat Completions API and chat-tuned models became common, LangChain played a crucial role in simulating chat-style interactions through prompt templates like this:

"""The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
{history}
Human: {input}
AI:"""

These types of prompt templates became less common with the introduction of the Chat Completions API.

Back then, prompts were also used to get models to use tools. But that changed when OpenAI introduced function calling, which quickly became the standard. As a result, more model providers began fine-tuning their models to support tool use through function calls, making it much easier for developers to integrate tools without relying on complex prompt patterns.

There was also a time when prompts like “think step by step” were widely used to guide reasoning, but with the rise of reasoning models, many LLMs can now guide themselves without relying on predefined prompt templates.

With all these improvements, developers today can leverage the power of LLMs more easily and effectively, without needing to depend as heavily on frameworks like LangChain.

🧰 Tool Usage

Tool use was one of the main reasons developers turned to LangChain when it first launched. At the time, language models couldn’t natively use tools, so LangChain filled the gap by offering a way to extend LLM capabilities through tool integration.

LangChain provided built-in support for connecting models to tools such as:

• Web search (e.g., using SerpAPI or Bing Search)

• Calculators for basic and complex math

• Python REPLs for running code snippets

These integrations were orchestrated using agentic patterns like ReAct and MRKL, enabling models to decide which tool to use based on the conversation.

Now, LLM providers support tool usage natively:

• Function Calling: Language models like those from OpenAI allow developers to define functions as schemas. The model can then decide when to use a function during execution.

• Built-in Tools: Models from OpenAI, Google, and Anthropic increasingly support built-in tools such as search, code execution, and more.

• MCP: Anthropic’s Model Context Protocol (MCP) provides a standardized way to extend language models with tools and additional context.

LangChain also supports function calling and MCP, and it offers a large collection of tools for developers to choose from. However, tool usage is no longer the framework’s main value proposition. For complex agentic workflows, even LangChain itself recommends using LangGraph instead of relying solely on the core framework.

🔍 Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation, or RAG, was one of the hottest topics in AI. Everyone wanted to connect their LLMs to external knowledge sources, largely due to the limited context length of models at the time. LangChain quickly became the go-to solution for this. It provided a complete RAG pipeline, including:

• Loaders to pull data from external sources

• Vector stores for storing embedded data

• Retrievers for managing retrieval logic

• Chunking strategies to break down documents into manageable pieces that fit within context limits

Later on, OpenAI introduced file search tool in their Assistant API (initially in beta) and eventually rolled it into the more stable Response API. This allowed users to upload files to OpenAI’s built-in vector store, enabling OpenAI to handle the entire retrieval process without requiring users to manage chunking, embeddings, or vector storage.

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4o-mini",
    input="What is deep research by OpenAI?",
    tools=[{
        "type": "file_search",
        "vector_store_ids": ["<vector_store_id>"]
    }]
)
print(response)

While OpenAI’s approach is much simpler to use, it’s also less flexible. LangChain, on the other hand, gives developers full control over the RAG pipeline, including choice of vector store, chunking logic, retrievers, and more. This can be critical when working with non-OpenAI models or when you need fine-grained control over retrieval behavior.

Some may argue that the need for RAG has diminished with the advent of long-context models, but it still plays a vital role in many applications. And when it comes to building a customized RAG pipeline, LangChain remains a strong option.

🌐 Access to Multiple Model Providers

OpenAI was the dominant provider when language model APIs first became available. However, LangChain anticipated a future with multiple providers and built its framework to support them from the start.

As new providers entered the space, LangChain quickly integrated them, making it easy for developers to swap between models. This flexibility was one of the key reasons LangChain gained popularity early on.

Today, OpenAI’s own SDK supports multiple model providers, as long as they implement the OpenAI-compatible API standard. Providers like Google, Anthropic, and Ollama have adopted this approach, allowing developers to use their models seamlessly through a unified interface.

Some newer providers, like DeepSeek, didn’t even bother building a separate SDK. They simply conformed to the OpenAI spec from the start. This works well for both sides: model providers can focus on training and deploying their models, while users benefit from immediate access using tools they’re already familiar with.

In contrast, with LangChain, users often have to wait for official or community-built integrations to be added before they can access a new model, adding friction to the development process.

🧩 Structured Outputs

The output from LLMs was originally unstructured text, which made it challenging to parse and use reliably. LangChain addressed this by introducing output parsers, letting developers define expected structures and automatically interpret the model’s responses.

from pydantic import BaseModel, Literal
from openai import OpenAI

class SentimentResponse(BaseModel):
    sentiment: Literal["positive", "neutral", "negative"]

client = OpenAI()

response = client.responses.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Classify the review sentiment."},
        {"role": "user", "content": "The service was excellent!"}
    ],
    response_format=SentimentResponse  # schema ensures type-safe output
)

print(response.sentiment)  # e.g., "positive"

Today, structured outputs are supported natively by LLM providers including OpenAI, Google, and Anthropic.

Subscribe now

🧑‍⚖️ Final Verdict: Is LangChain Still Worth Using in 2025?

Absolutely. Despite some criticism and the fact that major providers like OpenAI now offer many of LangChain’s features out of the box, LangChain remains a powerful and relevant tool thanks to several key strengths.

Pros:

• Open Source and actively maintained: LangChain continues to receive frequent updates and improvements from an active community.

• Multi-model support: It supports smaller or open-source LLMs that may lack native features like function calling or structured output.

• Strong RAG toolkit: LangChain offers a robust pipeline for retrieval-augmented generation, with built-in loaders, vector stores, and customizable retrieval logic.

• Rich ecosystem: Beyond its core library, LangChain includes LangGraph for multi-agent workflows and LangSmith for observability and debugging.

Cons:

• Steeper learning curve: Its extensive feature set can be overwhelming for beginners.

• Complex abstractions: You’ll need to learn LangChain-specific concepts like LCEL.

• Documentation navigation: The abundance of content and component layers can make the docs difficult to navigate

🔚 Final Take

When choosing an AI orchestration approach in 2025, you generally have three options:

• Use an open-source orchestration framework like LangChain

• Rely directly on a model provider’s SDK, such as OpenAI’s

• Build your own orchestration layer from scratch

Each option comes with its own set of trade-offs, ranging from flexibility and control to simplicity and ease of integration. The good news is that, unlike the early days of LLMs, developers today have a rich ecosystem of choices to suit different project needs and levels of complexity.

The Chat Completions API is familiar territory. It has been the backbone of countless AI applications since early 2023. It’s flexible, well-documented, and battle-tested. Other companies like Anthropic and Google have also adopted similar message-based interfaces, helping them become a de facto industry standard. So why change what already works?

Well, OpenAI has a track record of evolving its APIs to reflect the latest innovations in AI development. The Chat Completions API was introduced when chat models like ChatGPT gained widespread use.

The Response API follows that same trend, but its focus is on agentic capabilities. It introduces first-class support for tools, simplifies conversation management with built-in state handling, and enhances support for streaming.

In this article, we will dive deeper into these new features of the response API and see how it compares with the chat completion API. When you are done with the article, you will understand the following:

• How the Response API differs from the Chat Completions API

• How it manages conversational state natively

• The Response API’s first-class support for tools

• The semantic-driven streaming support in the Response API

Let’s dive in.

Evolution of OpenAI API

The OpenAI API has evolved significantly over the last few years. When OpenAI first introduced its API in 2020, it gave developers access to GPT-3 through a simple interface. Since then, OpenAI has continued to refine and expand its API offerings to better support a variety of use cases,

Here’s a quick overview of the key APIs OpenAI has released over the years:

• The Completions API

• The Chat Completion API

• The Assistant API

The Completions API

The Completions API was OpenAI’s first publicly available API, launched in 2020. It uses a simple "text in, text out" interface. You provide a text prompt, and the model responds by continuing or completing the input. This design reflects the foundational principle of large language models: predicting the next token in a sequence.

Here’s an example of how to use the Completions API with the OpenAI Python SDK:

import openai

openai.api_key = "YOUR_API_KEY"

response = openai.Completion.create(
    model="text-davinci-003",
    prompt="Write a poem about the ocean at night.",
    max_tokens=100,
    temperature=0.7,
)

print(response.choices[0].text)

This API was groundbreaking at the time, as it marked the first introduction of a general-purpose language model that could be prompted to perform a wide range of tasks, including text generation, summarization, question answering, and more.

Chat Completions API

With the release of ChatGPT in 2022, there was a growing demand for an API that mirrored the conversational style users experienced in the ChatGPT interface. In March 2023, OpenAI responded by launching the Chat Completions API.

Unlike the original Completions API, which followed a simple text-in, text-out format, the Chat Completions API was designed around multi-turn conversations. It introduced a structured format using messages, where each message had a role (system, user, or assistant), enabling developers to build more interactive and context-aware experiences.

Here’s an example of how to use the Chat Completions API with the Python SDK:

import openai

openai.api_key = "YOUR_API_KEY"

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's a good recipe for homemade pizza?"}
    ],
    temperature=0.7
)

print(response.choices[0].message["content"])

Over time, OpenAI continued to improve this API by adding features like function calling, which allowed models to interact with external tools. This introduced agent-like capabilities, but they were not fully integrated into the design of the API.

The Assistant API

At OpenAI’s Developer Day 2023, the Assistant API was introduced in beta as a step toward building more capable AI agents. It introduced the concept of assistants, AI agents that could be created and given access to built-in tools like code execution, file handling, and function calling. It also brought in threads, which allowed developers to manage and persist conversation state across interactions.

Here’s how the API works in practice. First, you create an assistant and grant it access to a built-in tool like the code interpreter:

from openai import OpenAI
client = OpenAI()

assistant = client.beta.assistants.create(
  name="Math Tutor",
  instructions="You are a personal math tutor. Write and run code to answer math questions.",
  tools=[{"type": "code_interpreter"}],
  model="gpt-4o",
)

Next, you create a thread to manage the ongoing conversation:

thread = client.beta.threads.create()

User messages are added to the thread using its ID:

message = client.beta.threads.messages.create(
  thread_id=thread.id,
  role="user",
  content="I need to solve the equation `3x + 11 = 14`. Can you help me?"
)

Finally, you initiate a run that ties the assistant and the thread together:

run = client.beta.threads.runs.create_and_poll(
  thread_id=thread.id,
  assistant_id=assistant.id,
  instructions="Please address the user as Jane Doe. The user has a premium account."
)

The Assistant API introduced a powerful architecture for building agentic workflows, but it remained experimental and in beta. Many of the ideas it tested, like persistent conversation state and built-in tool support, laid the foundation for what would become the more streamlined and production-ready Response API.

The Response API: The next step in OpenAI’s API evolution

In March 2025, nearly two years after the launch of the Chat Completions API, OpenAI introduced the Response API, a unified interface that combines the best of both the Chat Completions and Assistant APIs.

From the Chat Completions API, it inherits its simplicity: you send a list of messages and get a response. There’s no need to create assistants, manage thread objects, or handle extra orchestration. The Assistant API brings over the features that made agentic workflows possible, including built-in tool support, native state management, and event-driven streaming. The result is an API that’s lightweight and easy to use, yet powerful enough to support fully capable AI agents.

With the introduction of the Response API, OpenAI announced plans to deprecate the Assistant API. However, the Chat Completions API isn’t going anywhere. That means developers will now have two primary APIs to choose from on the OpenAI platform. So, which one should you use?

OpenAI’s general recommendation is simple: use the Response API for new projects, and stick with the Chat Completions API for existing ones that are already in production. Now let’s take a look at some of the features that make the Response API stand out from the Chat Completions API.

Conversation Statement Management

The Chat Completions API is stateless; each call is independent and doesn’t retain memory of previous interactions unless you explicitly pass the full message history every time. The Response API changes that by offering optional stateful conversations, managed directly by the OpenAI platform. This works similarly to how threads functioned in the Assistant API, but with a much simpler interface.

To maintain state between calls, you just pass the previous_response_id of an earlier response. This tells the API to continue the conversation from where it left off:

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4o-mini",
    input="tell me a joke",
)
print(response.output_text)

second_response = client.responses.create(
    model="gpt-4o-mini",
    previous_response_id=response.id,
    input=[{"role": "user", "content": "explain why this is funny."}],
)
print(second_response.output_text)

If you prefer to manage state manually, as you would in the Chat Completions API, you still can. The Response API is backward-compatible and supports manual message construction.

You can also opt out of automatic state management entirely by setting the store parameter to False. This disables storage of the conversation state:

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-4o-mini",
    input=[
        {"role": "user", "content": "knock knock."},
        {"role": "assistant", "content": "Who's there?"},
        {"role": "user", "content": "Orange."},
    ],
    store=False
)

print(response.output_text)

In short, the Response API improves on the Chat Completions API by making conversation state optional, seamless, and easier to manage.

Built-in Tool Support in the response API

The response API, similar to the Assistant API, offers built-in tool support. With the chat completion API, you could only have access to tools via function calling or via dedicated models. Let’s go through some of the built-in tools response API currently supports.

Web Search Tool

The Response API includes a built-in web search tool that allows the model to perform live searches. Using it is straightforward, just pass the tool in the tools parameter by specifying its type as web_search_preview.

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4.1",
    tools=[{"type": "web_search_preview"}],
    input="What was a positive news story from today?"
)

print(response.output_text)

While the Chat Completions API also supports web search, it requires using specific models like gpt-4o-search-preview or gpt-4o-mini-search-preview. In contrast, the Response API offers greater flexibility, web search can be enabled across a wider range of models, making it easier to integrate into different applications without being locked into special model variants.

File Search Tool

The Response API introduces support for the file search tool. This tool allows the model to retrieve and reference information from files you've uploaded, making it great for use cases like answering questions from documentation, PDFs, knowledge bases, or research papers.

The Chat Completions API doesn’t support this tool at all. This is one of the Response API’s more agentic capabilities, carried over and improved from the Assistant API.

To use the file search tool, you first upload your files and organize them into a vector store. You then attach this store to your request using the tools parameter.

Here’s a basic example of how to use it:

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4o-mini",
    input="What is deep research by OpenAI?",
    tools=[{
        "type": "file_search",
        "vector_store_ids": ["<vector_store_id>"]
    }]
)
print(response)

This tool gives the model retrieval-augmented generation (RAG) abilities with just a few lines of code.

Image Generation

Another powerful feature of the Response API is its support for image generation as a tool. Instead of requiring a dedicated model like in previous APIs, image generation is now treated as a native tool, just like web search or file search. This gives developers a more unified and consistent interface for triggering different capabilities.

Behind the scenes, the tool uses the GPT image model, so you still get the full capabilities of gpt-image-1, but without switching models. That means you can generate images while still using gpt-4o or any other supported model in your workflow.

Here’s how you can use it:

from openai import OpenAI
import base64

client = OpenAI() 

response = client.responses.create(
    model="gpt-4.1-mini",
    input="Generate an image of gray tabby cat hugging an otter with an orange scarf",
    tools=[{"type": "image_generation"}],
)

# Save the image to a file
image_data = [
    output.result
    for output in response.output
    if output.type == "image_generation_call"
]
    
if image_data:
    image_base64 = image_data[0]
    with open("otter.png", "wb") as f:
        f.write(base64.b64decode(image_base64))

MCP Support

OpenAI introduced support for MCP within the Response API, which enables the model to access remote MCP servers directly from the API. This essentially gives the response api access to an unlimited number of tools that implement the MCP protocol. Here’s how you can access an MCP server with it:

The Response API includes native support for MCP, enabling the model to directly access remote MCP servers. This effectively unlocks access to an unlimited number of external tools and knowledge bases that implement the MCP protocol.

Here’s an example of how to connect to an MCP server using the Response API:

from openai import OpenAI

client = OpenAI()

resp = client.responses.create(
    model="gpt-4.1",
    tools=[
        {
            "type": "mcp",
            "server_label": "deepwiki",
            "server_url": "https://mcp.deepwiki.com/mcp",
            "require_approval": "never",
        },
    ],
    input="What transport protocols are supported in the 2025-03-26 version of the MCP spec?",
)

print(resp.output_text)

In this example, the MCP tool connects to the deepwiki MCP server, which hosts various documentation. This allows your application to answer detailed questions based on any documentation available within the wiki.

Given MCP’s growing adoption, this integration is a game-changer; now developers can easily plug into remote MCP servers without needing to handle the MCP SDK or complex integrations themselves.

Additional Tools and What’s Next

OpenAI is expected to continue adding more tools to the Response API over time. So far, they have introduced the Computer Use tool, which builds on their Computer-Using Agent (CUA) model, as well as the code interpreter tool. These additions will make the Response API even more powerful and agentic.

Response API Streaming With Semantic Events

The Chat Completions API’s streaming feature delivered raw text chunks as they were generated. In contrast, the Response API introduces semantic streaming, where each streamed result is a structured event providing rich information beyond just text.

For example, the API emits semantic events like:

• ResponseCreatedEvent — signals when a streaming response starts

• ResponseCompletedEvent — signals when the stream finishes

• ResponseOutputTextDelta — delivers partial text updates as they arrive

Beyond text, the Response API also streams events related to tool usage, such as ResponseFileSearchCallSearching and ResponseFileSearchCallCompleted, enabling developers to track tool interactions in real time.

This event-driven approach makes streaming more informative and easier to integrate into complex, agentic workflows.

Leave a comment

Is This Vendor Lock-In?

The Response API brings powerful new capabilities, but it also raises an important question: Is this a form of vendor lock-in?

Many of the features introduced, like built-in tools, native state management, and semantic streaming, are tightly integrated into the OpenAI platform. These enhancements don't carry over to other model providers. So once you start building with the Response API, switching to a different provider may require significant rework, reducing the incentive to move away from OpenAI.

That said, OpenAI isn’t forcing developers to use these features. They’re optional enhancements designed to improve developer experience and agent capabilities. You can still use the Response API in a minimal, model-only fashion, just like Chat Completions, or continue using the Chat Completions API itself.

This also opens up new possibilities: as the Response API becomes more widely adopted, we may see community-built wrappers that abstract these features and make them usable across different model backends.

Subscribe now

Conclusion

The Response API brings a clear evolution in how AI agents are built. With native support for tools, built-in state management, and a more event-driven interface, it simplifies many of the complexities developers previously had to handle manually. If you're building an agentic application and you're comfortable working within the OpenAI ecosystem, the Response API is the clear choice. It provides more out of the box, with less boilerplate.

However, if your project needs to remain model-agnostic or you’re aiming for maximum portability, the Chat Completions API might still be the better fit. Thanks to its simplicity and wide adoption, it remains a solid choice. The Response API’s backward compatibility also means you can still use it in a familiar, lightweight way without adopting the full toolset.

Ultimately, the Response API represents OpenAI’s vision for the future of AI development: more structured, more capable, and more agent-driven. Whether that future aligns with your next build depends on your goals. The good news is you’ve got options, and both APIs are here to stay.

“What’s your check-in time?”, “Do you have free parking?” or “Can I cancel my reservation?”

… alongside routine booking and cancellation requests.

With AI, there’s a better way. A conversational hotel support agent can provide instant answers, handle reservations, and even manage booking changes without ever involving a human agent.

And with the OpenAI Agent SDK, building such a system is not only possible but also remarkably straightforward.

In this article, we’ll walk through how to build a fully functional hotel reservation support agent using the Agent SDK.

You’ll see how to design multi-agent workflows where responsibilities are cleanly divided: triaging user intent, answering FAQs, and processing reservations or cancellations.

We’ll show how to connect your agents to real tools like room managers and FAQ providers, and how to maintain conversation state using context.

By the end, you’ll understand:

1. How to define and orchestrate AI agents using the OpenAI Agent SDK

2. How to create and register tools that interact with external systems

3. How to use context to carry user-specific information across turns

Whether you're exploring AI for customer service or want a hands-on project with the Agent SDK, this guide will show you how to build something useful, practical, and production-ready.

Understanding Multi-Agent Systems

A Multi-Agent System (MAS) is a framework where multiple autonomous agents interact within an environment to achieve individual or collective goals. Each agent operates independently, perceiving its surroundings and making decisions based on its objectives and knowledge. These agents can collaborate, coordinate, or even compete, depending on the system's design and desired outcomes. Such systems are particularly effective in handling complex, distributed tasks that are beyond the capabilities of a single agent.

In the context of hotel reservations, a MAS can be structured with specialized agents handling distinct responsibilities:

• Primary Agent: Acts as the initial point of contact, interpreting user intents and directing queries to the appropriate specialized agent.

• Reservation Agent: Manages room bookings, cancellations, and modifications, interfacing with the hotel's reservation system.

• FAQ Agent: Provides information on hotel amenities, policies, and other frequently asked questions.

This division of labor ensures that each agent focuses on its area of expertise, leading to more efficient and accurate responses.

To build these multi-agent systems, developers can use powerful toolkits like OpenAI Agent SDK, Google ADK, and CrewAI. These platforms simplify the process of creating, managing, and coordinating AI agents that work together seamlessly in real-world applications. In this tutorial, we’ll be using the OpenAI Agent SDK to demonstrate how to build a hotel reservation support agent from the ground up.

Setting the Stage

Before we start building intelligent agents, let’s set up the foundation of our system. In this section, we’ll walk through how to install the necessary tools, configure your environment, and create the external system that our agents will interact with.

Installing the OpenAI Agent SDK

You can install the SDK via pip:

pip install openai-agents

Next, you'll need an OpenAI API key to run the agent. Store it as an environment variable.

export OPENAI_API_KEY="your-api-key-here"

Building the RoomManager

Before diving into agents and SDK features, let’s first build the external system that agents will interact with: a simple in-memory room reservation manager.

This RoomManager class will simulate a hotel’s inventory of rooms, tracking availability, handling reservations, and releasing bookings. Our agents will call methods from this class via tools in later sections.

Create a new file named room_manager.py, and add the following code:

from typing import Optional

class RoomManager:
    def __init__(self):
        self.rooms = {}
        for i in range(1, 21):
            room_type = "standard" if i <= 15 else "deluxe"
            self.rooms[f"room_{i}"] = {"type": room_type, "available": True}

    def find_available_room(self, room_type: str) -> Optional[str]:
        for room_id, info in self.rooms.items():
            if info["type"] == room_type and info["available"]:
                return room_id
        return None

    def reserve_room(self, room_id: str) -> bool:
        if room_id in self.rooms and self.rooms[room_id]["available"]:
            self.rooms[room_id]["available"] = False
            return True
        return False

    def release_room(self, room_id: str) -> bool:
        if room_id in self.rooms and not self.rooms[room_id]["available"]:
            self.rooms[room_id]["available"] = True
            return True
        return False


room_manager = RoomManager()

This class keeps things simple and self-contained. It supports:

• Finding the next available room of a given type

• Reserving a room (marking it as unavailable)

• Releasing a reserved room (making it available again)

Now that our system is ready to simulate a hotel backend, we can begin integrating it with tools and agents using the Agent SDK.

Understanding Context Management in the Agent SDK

When working with large language models (LLMs), context usually means the conversation history or the text the model uses to generate its next response.

In the OpenAI Agent SDK, context means something different. It is a structured Python object that holds important information about the ongoing conversation or session. This context is passed to the agent runner, which shares it among agents and tools as needed.

How Context Differs in Agent SDK

• LLM context: The text-based conversation history that helps the model understand the current interaction.

• Agent SDK context: A stateful Python object that stores key information (like a user’s booking details) and is used by the agent runner to keep track of what has happened so far.

This makes it easy for agents to remember important details across multiple steps. For example, if a user reserves a room, the room ID can be saved in the context. Later, if the user wants to cancel, the cancellation agent can access that same room ID from the context without asking the user again.

Defining Our Context Object

For our hotel reservation system, the context will store the user’s reserved room ID. Here's how you can define it in a file named context.py:

# context.py

from dataclasses import dataclass
from typing import Optional

@dataclass
class UserBooking:
    room_id: Optional[str] = None

This UserBooking object is passed to the agent runner and shared with agents and tools during the conversation. It helps keep the interaction smooth and natural by remembering the user’s booking information across multiple turns.

Defining the Tools

In this section, we define the tools our agents will use. These tools utilize the RoomManager we defined earlier and act as a bridge between the agents and the RoomManager class. They also interact with the context manager to maintain the current user’s room_id throughout the conversation.

First, here are the necessary imports you’ll need at the top of the file:

from agents import function_tool, RunContextWrapper
from context import UserBooking
from room_manager import room_manager

Reserving a Room

The reserve_room_by_type function allows the agent to reserve an available room of the requested type (either "standard" or "deluxe"). It uses the RoomManager to find a suitable available room. If a room is found and successfully reserved, it updates the user’s context with the reserved room’s ID. This helps keep track of the user’s current booking throughout the session.

@function_tool
async def reserve_room_by_type(wrapper: RunContextWrapper[UserBooking], room_type: str) -> str:
    "Reserve an available hotel room of a specified type (standard or deluxe)."

    room_id = room_manager.find_available_room(room_type)
    if not room_id:
        return f"No available {room_type} rooms."

    if room_manager.reserve_room(room_id):
        wrapper.context.room_id = room_id
        return f"Room {room_id} has been reserved for you."
    else:
        return f"Failed to reserve room {room_id}."

Freeing a Reserved Room

The free_reserved_room function releases a room that the user has previously reserved. It checks the user’s context for a currently reserved room ID. If one exists, it uses the RoomManager to release the room and clears the room ID from the context. This keeps the user session consistent and ensures that room availability is updated properly.

@function_tool
async def free_reserved_room(wrapper: RunContextWrapper[UserBooking]) -> str:
    "Releases the currently reserved room for the user."

    room_id = wrapper.context.room_id
    if not room_id:
        return "You don't have any active room reservation."

    if room_manager.release_room(room_id):
        wrapper.context.room_id = None
        return f"Room {room_id} has been released and is now available."
    else:
        return f"Failed to release room {room_id}."

Providing FAQ Answers

The get_faqs function returns a static string containing common hotel FAQs, such as room types, prices, check-in/out times, and amenities. Unlike the other tools, this one does not use the RoomManager or context because it simply serves as an informational resource for the FAQ agent.

@function_tool
async def get_faqs() -> str:
    "Returns answers to common hotel reservation FAQs."

    faqs = """
    1. What types of rooms are available?
    - Standard: Cozy and affordable
    - Deluxe: Spacious with additional amenities

    2. Room prices:
    - Standard: $100/night
    - Deluxe: $180/night

    3. Check-in/out:
    - Check-in: 2:00 PM
    - Check-out: 11:00 AM

    4. Cancellation policy:
    - Free up to 24 hours before check-in

    5. Amenities:
    - Free breakfast and parking
    """
    return faqs

Together, these tools connect the AI agents to the hotel’s backend room management system and maintain the user’s booking context during conversations. This separation of concerns keeps the agents focused on dialog and decision-making while delegating business logic to dedicated components.

Place all this code, including the imports and the three functions, into a file named tools.py in your project directory.

Defining the Agents

In this section, we define the core AI agents that will handle different parts of the hotel reservation system. Each agent has a clear role and set of instructions. They use the tools we defined earlier and can hand off conversations between each other to create a seamless multi-agent workflow.

Start with the necessary imports:

from agents import Agent
from agents.extensions.handoff_prompt import RECOMMENDED_PROMPT_PREFIX
from context import UserBooking
from tools import reserve_room_by_type, free_reserved_room, get_faqs

FAQ Agent

The faq_agent specializes in answering common questions about the hotel, such as room types, pricing, check-in/out policies, and amenities.

• It relies entirely on the get_faqs tool to provide answers.

• It is instructed not to use any knowledge outside of this tool.

• If it encounters a question it can’t answer or one outside the FAQs, it transfers control back to the primary agent.

faq_agent = Agent[UserBooking](
    name="FAQ Agent",
    handoff_description="Agent for answering frequently asked questions about the hotel",
    instructions="""
        You are an FAQ agent for a hotel reservation system. Your job is to answer questions about room types, pricing, check-in times, and other policies.

        # Routine
        1. Use the `get_faqs` tool to answer any customer question.
        2. Do not use your own knowledge — rely solely on the tool output.
        3. If you cannot answer the question, or it falls outside FAQs, transfer back to the primary agent.""",
    tools=[get_faqs],
)

Reservation Agent

The reservation_agent manages room bookings and cancellations.

• It guides the user through either reserving a new room or cancelling an existing booking.

• For new reservations, it:

  • Asks for the preferred room type,

  • Uses the reserve_room_by_type tool to find and book a room,

  • Saves the reserved room ID in the user context,

  • Confirms the reservation.

• For cancellations, it:

  • Checks if the user has a current reservation,

  • Uses the free_reserved_room tool to release the room,

  • Confirms cancellation and clears the booking from the context.

  • Notifies the user if no active reservation exists.

• If the user request does not fit these intents, it hands off to the primary agent.

• It can hand off conversations to the FAQ agent if needed.

reservation_agent = Agent[UserBooking](
    name="Reservation Agent",
    handoff_description="Agent for handling all hotel reservations and cancellations",
    instructions="""You are a hotel reservation agent. Your job is to help users reserve rooms and cancel existing bookings.
        # Routine
        1. Determine if the user wants to make a new reservation or cancel an existing one.
        2. If making a reservation:
            a. Ask for the preferred room type (standard or deluxe).
            b. Use the `reserve_room_by_type` tool to find and reserve an available room.
            c. Save the booked room ID to the user's context.
            d. Confirm the reservation.
        3. If cancelling:
            a. Check the user's context for an existing room ID.
            b. If found, use the `free_reserved_room` tool to cancel the booking.
            c. Confirm the cancellation and clear the context.
            d. If no room is reserved, inform the user.

        If the request does not match these intents, transfer back to the primary agent.""",
    tools=[reserve_room_by_type, free_reserved_room],
    handoffs=[faq_agent],
)

Primary Agent

The primary_agent is the triage or "front desk" agent.

• It interprets user intent and routes the conversation to the appropriate specialist agent:

  • To the Reservation Agent for booking or cancellation requests,

  • To the FAQ Agent for questions about policies and amenities.

• If the intent is unclear or changes, the primary agent takes control or redirects accordingly.

• The RECOMMENDED_PROMPT_PREFIX adds system-level context explaining the multi-agent system and how handoffs work, so the agent knows to switch seamlessly between specialists without exposing this to the user.

primary_agent = Agent[UserBooking](
    name="Primary Agent",
    instructions=(
        f"{RECOMMENDED_PROMPT_PREFIX} "
        "You are the primary triage agent in a hotel reservation system. You understand user intent and hand off to the appropriate specialist."
        "1. If the user wants to reserve or cancel a room, hand off to the Reservation Agent."
        "2. If they ask about hotel amenities, pricing, or policies, hand off to the FAQ Agent."
        "3. If unsure or the request changes, take control or redirect accordingly."
    ),
    handoffs=[reservation_agent, faq_agent],
)

Circular Handoffs

To keep the conversation flowing naturally, both the reservation and FAQ agents also have the primary agent in their handoffs list. This allows them to hand back control if needed.

reservation_agent.handoffs.append(primary_agent)
faq_agent.handoffs.append(primary_agent)

This multi-agent setup creates a natural flow where the primary agent acts as the central coordinator, passing requests to specialist agents that use tailored instructions and tools. This design keeps responsibilities clear and allows each agent to focus on its expertise while providing a smooth user experience.

Place all this code, including imports and agent definitions, in a file named hotel_agents.py in your project directory.

Share

Putting It All Together and Running the Agents

Now that we have defined our context, tools, and agents, it’s time to bring everything together and run our multi-agent system in action.

The core of the system is the Runner class from the OpenAI Agent SDK, which manages agent execution, context passing, and conversation flow, including seamless handoffs between agents.

Here is a simple example of how you can set up an interactive loop that:

• Takes user input from the console,

• Passes it to the current agent (starting with the primary agent),

• Handles multi-turn conversations,

• Prints out agent responses and any tool calls or handoffs.

import asyncio
from agents import Runner, MessageOutputItem, ItemHelpers, HandoffOutputItem, ToolCallItem, ToolCallOutputItem
from context import UserBooking
from hotel_agents import primary_agent

async def main():
    # Initialize user context that keeps track of user state like current room reservation
    context = UserBooking()
    
    # Start with the primary agent, which triages and routes user requests
    current_agent = primary_agent
    
    # Conversation history input items
    input_items = []

    while True:
        # Get user input from the console
        user_input = input("Enter your message: ")
        if user_input.lower() == "exit":
            print("Exiting the hotel reservation assistant.")
            break

        # Append user message to the conversation history
        input_items.append({"content": user_input, "role": "user"})

        # Run the agents system with the current agent, conversation input, and user context
        result = await Runner.run(
            starting_agent=current_agent,
            input=input_items,
            context=context,
        )
        
        # Process and display outputs from the agent runner
        for new_item in result.new_items:
            agent_name = new_item.agent.name
            
            if isinstance(new_item, MessageOutputItem):
                # Agent response message
                print(f"{agent_name}: {ItemHelpers.text_message_output(new_item)}")
            
            elif isinstance(new_item, HandoffOutputItem):
                # Inform user about agent handoff
                print(f"Handed off from {new_item.source_agent.name} to {new_item.target_agent.name}")
                # Update current agent to the new agent after handoff
                current_agent = new_item.target_agent
            
            elif isinstance(new_item, ToolCallItem):
                # Indicate when the agent is calling a tool
                print(f"{agent_name}: Calling a tool...")
            
            elif isinstance(new_item, ToolCallOutputItem):
                # Output from a tool call
                print(f"{agent_name}: Tool call output: {new_item.output}")
            
            else:
                print(f"{agent_name}: Skipping item: {new_item.__class__.__name__}")

if __name__ == "__main__":
    asyncio.run(main())

Here’s a breakdown of the code above:

• UserBooking Context: This keeps track of the user's current reservation state, like the room_id they booked.

• Primary Agent: Acts as the triage agent directing queries to FAQ or reservation specialists.

• Input Items: All user messages are accumulated to provide full conversation context for the agents.

• Runner.run: This asynchronously processes user input with the current agent and returns messages, handoffs, and tool calls.

• Output Handling: The loop prints agent messages, notifies about handoffs, and displays tool call outputs, maintaining a smooth dialogue flow.

• Agent Switching: When a handoff happens, the current_agent updates to the new agent to continue the conversation.

Leave a comment

At this point, our main program is complete and saved as main.py. The full project structure looks like this:

hotel-reservation-agent/
├── main.py
├── context.py
├── tools.py
├── room_manager.py
└── hotel_agents.py

To start the hotel reservation assistant, run the following command:

python main.py

You can find the complete code available here.

Conclusion

In this article, we built a multi-agent hotel reservation system using the OpenAI Agent SDK. We covered how to structure AI agents with clear roles, manage user context, and connect tools to interact with an external room manager.

By orchestrating seamless handoffs between specialized agents, we created a natural, conversational experience for hotel booking and FAQs.

Looking ahead, this system can be enhanced by integrating a real database instead of an in-memory store, adding user authentication, and expanding agents to handle more complex workflows like payment processing or personalized recommendations. The modular design makes it easy to extend and adapt for real-world deployment.

Subscribe now