Matt LewandowskiDid you know that you can get a well structured JSON Object {} back from OpenAI's API, without a super hacky prompt? Well you can, and it's super easy!
I recently built a feature on my website, that will generate a Sprint Retrospective template using OpenAI's completions API and I want to show you how I did it.
Getting started
What do I mean by a well-structured JSON Object? I mean the response from the API will be a serialized object, every time. I don't have to worry about the completion of adding in random information, that will make my JSON.parse() fail. This is the type definition I am expecting back from the API.
type RetroTheme = {
name: string;
description: string;
columns: { name: string; color: string; description: string }[];
};
This is both the type that my frontend is expecting and the type that the OpenAI API is returning. You can achieve this with "Tools", which used to be called "Functions". You can learn more about these tools in the official documentation here.
Although, I found the documentation of most other examples online somewhat difficult to understand. The examples they use are too simple and are not built for just ensuring we are returned structured data, so I will walk you through my understanding as well as how I have done it.
Setup
whether you are using a library, or calling the library directly, most of this information is the same. There might be small details that are different, but the concepts are the same.
For my setup, I am using the node version of the openai sdk.
This means my setup looks like this:
import OpenAI from "openai";
/** This is a server-side client only. The key is not available to the client. */
export const openai = new OpenAI({
apiKey: process.env.OPEN_AI_KEY,
});
This will initialize a client with your api-key, so that you can start calling the API. Before we call the API, we need to create a tool. Tools in OpenAI have many uses, but our use case is strictly ensuring the API returns us structured data, the way we define it.
The Tool
First, let's define a "Tool". We will use this later when we want to actually get data from the API.
The tool I am creating is to ensure the data returned fits into this object:
type RetroTheme = {
name: string;
description: string;
columns: { name: string; color: string; description: string }[];
};
This is the tool (smaller version below it):
const retroFunction: OpenAI.Chat.ChatCompletionCreateParams.Function = {
name: "CreateRetrospectiveTheme",
description:
"Create a creative agile retrospective theme to be used in a virtual meeting.",
parameters: {
type: "object",
properties: {
name: {
type: "string",
description:
"The name of the retrospective theme. Should be related to the theme. Should start with an emoji. For example: 💀 Pirate Adventure",
},
description: {
type: "string",
description:
"A creative description and opener for the retrospective template. For example: Ahoy, mateys! Embark on a thrilling pirate adventure as we navigate the high seas of our project. Hoist the sails, consult the treasure map, and share your tales of plunder and peril. Together, we'll chart a course for smoother sailing and greater riches ahead!",
},
columns: {
type: "array",
description:
"The columns of the retrospective board. Each one should be creative and related to the theme, while also being clear and actionable.",
items: {
type: "object",
properties: {
name: {
type: "string",
description:
"Name of the column. Should start with an emoji. Should be creative and related to the entire theme. For example: 💰 Treasures",
},
color: {
type: "string",
description: "Color of the column. Should fit the name.",
enum: [
"#ee2c1d",
"#E91E63",
"#9C27B0",
"#5c46e8",
"#203de5",
"#1b6fb2",
"#03A9F4",
"#009688",
"#009306",
"#66942a",
"#CDDC39",
"#FFEB3B",
"#FFC107",
"#FF9800",
"#FF5722",
"#000000",
"#9E9E9E",
"#795548",
],
},
description: {
type: "string",
description:
"A short, one sentence, description of the column, so it's easy to understand what to put into it. For example: Valuable lessons and successes",
},
},
},
},
},
required: [
"name",
"description",
"columns",
"columns.name",
"columns.color",
"columns.description",
],
},
};
Does it look pretty complicated? Let's remove some of the descriptions so it is easier to read!
const retroFunction: OpenAI.Chat.ChatCompletionCreateParams.Function = {
name: "CreateRetrospectiveTheme",
parameters: {
type: "object",
properties: {
name: {
type: "string",
},
description: {
type: "string",
},
columns: {
type: "array",
items: {
type: "object",
properties: {
name: {
type: "string",
},
color: {
type: "string",
enum: ["#ee2c1d", "#E91E63", "#9C27B0"],
},
description: {
type: "string",
},
},
},
},
},
required: [
"name",
"description",
"columns",
"columns.name",
"columns.color",
"columns.description",
],
},
};
So here we are defining our function. A function has a name and parameters. Our parameter is an object, and we define what that object needs to look like. We also add required fields, so the AI knows that these must be included.
Since our AI is going to be filling in these fields, it's important to have descriptions for each field. This will tell the AI exactly what to generate for each property of the object. If you take a look at the long example, you can see I am being pretty descriptive.
This of course adds more tokens to the request, which is worth keeping in mind, depending on your scale.
Using the Tool
so now that we have defined our schema, we need to provide this tool to the completion api:
tools: [
{
function: retroFunction,
type: "function",
},
],
Since we want this tool to be automatically called on the request, we need to also add it to our tool_choice:
tool_choice: {
type: "function",
function: {
name: "CreateRetrospectiveTheme",
},
},
Looks good, now let's add in to our messages. We have a system message that tells the AI what it is. This helps get better responses. Then we have our user's request, where we provide the details of what we want from our tool. The messages look like this:
messages: [
{
role: "system",
content:
"You are an agile retrospective AI. You are tasked with creating creative agile retrospective themes to be used in a virtual meeting. You will be given a query, and you will respond with a creative agile retrospective theme that revolves around the query.",
},
{
role: "user",
content: `The theme is: ${query}. There should be between 4 and 6 columns. Generate the response in ${languageMap[languageCode]} (${languageCode}).`,
},
],
putting it all together looks like this
export const getAIRetroTheme = async (
query: string,
languageCode: string,
): Promise<RetroTheme> => {
const resp = await openai.chat.completions.create({
model: "gpt-3.5-turbo-0125",
tools: [
{
function: retroFunction,
type: "function",
},
],
tool_choice: {
type: "function",
function: {
name: "CreateRetrospectiveTheme",
},
},
messages: [
{
role: "system",
content:
"You are an agile retrospective AI. You are tasked with creating creative agile retrospective themes to be used in a virtual meeting. You will be given a query, and you will respond with a creative agile retrospective theme that revolves around the query.",
},
{
role: "user",
content: `The theme is: ${query}. There should be between 4 and 6 columns. Generate the response in ${languageMap[languageCode]} (${languageCode}).`,
},
],
});
return JSON.parse(resp.choices[0].message.tool_calls[0].function.arguments);
};
To better understand our JSON.parse() above, let's take a look at the response we get from the API.
{
"id": "chatcmpl-0",
"object": "chat.completion",
"created": 1714973965,
"model": "gpt-3.5-turbo-0125",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_qabAvyFyD5A9K9Vff2cq95Xo",
"type": "function",
"function": {
"name": "CreateRetrospectiveTheme",
"arguments": "{\"name\":\"Dogs on Holiday\",\"description\":\"Reflecting on our sprint through the lens of dogs enjoying a well-deserved holiday\",\"columns\":[{\"name\":\"Pawsitive Aspects\",\"color\":\"#ee2c1d\",\"description\":\"Highlight what went well during the sprint\"},{\"name\":\"Collarborative Moments\",\"color\":\"#E91E63\",\"description\":\"Recognize teamwork and collaboration within the team\"},{\"name\":\"Barking Mad Ideas\",\"color\":\"#9C27B0\",\"description\":\"Share innovative and creative suggestions for improvement\"},{\"name\":\"Puppy Challenges\",\"color\":\"#ee2c1d\",\"description\":\"Identify obstacles and difficulties encountered during the sprint\"}]}"
}
}
]
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 167,
"completion_tokens": 137,
"total_tokens": 304
},
}
The reason we are getting the data from arguments is because of the way the tools work. OpenAI is actually putting together the arguments needed to call our tool. When we define what is required to call the tool, We are defining the arguments that it will generate. Since it has successfully called our Tool, we know we can access the arguments it generated to do so.
That was a lot of code blocks in a row, so take a break and check out this retrospective template I generated 😈💨
Using gpt-3.5-turbo takes about 5 seconds per generation, which is super fast!
Validate
**Edit: this is a new section
It's important to note that this is not always 100% accurate. AI will hallucinate and even this structured method cannot prevent that. There are however steps that we can take to reduce the risk.
Validate your response:
This is super important and I missed it at first. You cannot trust the response every time, though you can most of the time. I use zod to validate the response from OpenAI. This will throw an error if the response is not what I am expecting.
const retroTheme = z.object({
name: z.string(),
description: z.string(),
columns: z
.array(
z.object({
name: z.string(),
color: z.enum(enumColors),
description: z.string(),
}),
)
.min(3),
});
export type RetroTheme = z.infer<typeof retroTheme>;
const response = JSON.parse(
resp.choices[0]?.message.tool_calls?.[0]?.function.arguments ?? "{}",
);
return retroTheme.parse(response);
Add a retry function
Since you are validating you response, you should retry if it's not what you want. This will retry 3 times before finally returning the error to the user. The odds of this are unlikely, but still possible.
const retry = async (fn: () => Promise<RetroTheme>, retries = 3) => {
try {
return await fn();
} catch (e) {
if (retries <= 0) {
throw e;
}
return await retry(fn, retries - 1);
}
};
Conclusion
By leveraging OpenAI's tools and a well-defined JSON schema, we can ensure that the API returns structured data exactly as we expect it. This approach eliminates the need for hacky prompts and provides a reliable way to generate creative and engaging retrospective themes.
The ability to generate structured data opens up a world of possibilities for integrating AI-generated content into our applications. Whether you're building a retrospective template generator or any other feature that requires structured output, this technique can save you time and effort while delivering high-quality results.
If you are looking to implement something like this and have any questions feel free to reach out!
If you want to try out my retrospective template generator, you can do so Here!
