What we learned from building a ChatGPT plugin prototype

OpenAI recently released ChatGPT Plugins, which allows developers to integrate 3rd-party APIs into the ChatGPT product experience. At Supaglue, we’re building a developer platform for user-facing SaaS integrations, and we were curious how Plugins could work with our APIs.

Last week, we got developer access to the feature and spent a few hours building a Supaglue plugin prototype. In this blogpost, we’ll go over what we did, what we learned, and some of the challenges we ran into along the way.

Making OpenAI aware of Supaglue’s APIs

To build a ChatGPT Plugin, you need to do two things:

1. Make your service discoverable

2. Make your API discoverable

To learn about new services, ChatGPT Plugins use Well-known URIs (IETF RFC-8615). By hosting a {your_domain}/.well-known/ai-plugin.json manifest file, it lets you publish metadata about your service to their servers. The manifest file includes some key information about your service:

• A description of what your API does so that ChatGPT knows when to use the API
• The location of your OpenAPI bundle describing the available APIs, once ChatGPT has decided to use your API
• The authentication mechanism through which ChatGPT can securely communicate with your API

To learn about your API, ChatGPT Plugins needs a specification for your service’s APIs, and they use the OpenAPI standard for this.

Fortunately for us, Supaglue’s core product is an API and we had adopted the OpenAPI standard already. But despite having done most of the heavy lifting, we needed to clear several hurdles before we got the prototype to work for our use case.

Authentication & request limitations

Supaglue works with your customer’s contact (and other PII), so we needed to provide a secure way for users to interact with our Supaglue Plugin. ChatGPT Plugins support a few authentication strategies:

• No auth
• Service level
• User level
• OAuth

Unfortunately, these didn't meet our needs. For context, Supaglue is a multi-tenant API product where each of our customers can also serve multiple customers through our platform. When developers make calls to our APIs, they provide two pieces of information to identify who the call is intended for: 1) a customer ID and 2) a customer’s customer ID. Today we request this information in the form of a custom HTTP header or as query parameters in the URL.

While ChatGPT Plugin authentication strategies support multi-tenant authentication, they don’t directly support our “multi-multi-tenant” use case. In addition, Plugins doesn’t allow us to attach custom headers to API requests to identify our customer’s customers.

For the purposes of our prototype, we simply hardcoded an example customer’s-customer’s credentials into our server to work around these limitations.

Resolving API discovery

We ran into another Supaglue-specific obstacle in the API discovery step that was related to our uniform API signatures across multiple SaaS categories. For example, our CRM and Sales Engagement APIs for getting a list of contacts look like the following:

• GET /crm/v1/contacts
• GET /engagement/v1/contacts

Today, we create separate OpenAPI bundles for each SaaS category because it's easier to manage from a tooling standpoint. Yet ChatGPT Plugins expect only one OpenAPI bundle, so we had to merge our CRM and Sales Engagement OpenAPI bundles into one. Merging introduced some ambiguity into our OpenAPI specification: the two endpoints above both originally had the OpenAPI attribute "operationId" set as "getContacts" and "description" definition as "Get a list of contacts". Running some ChatGPT prompts to use these APIs resulted in poor results.

To resolve this issue, we disambiguated the operationId, summary, and description attributes by making them more precise, e.g. getCRMContacts and getEngagementContacts instead of just getContacts.

API response semantics

After we got ChatGPT to talk to the right APIs, we had to do some prompt engineering to get ChatGPT to interpret the structure and semantics of Supaglue’s responses correctly. For example, our initial prompt of “I’d like to get a list of crm contacts and show me their emails only” returned a list of hallucinated emails.

The problem was that Supaglue returns email as an array email_addresses, which ChatGPT didn’t know how to interpret.

We tried a few more prompts that led to similarly nonsensical answers. The prompt below, “I’d like to get a list of crm contacts who live outside of the united states” returned generic placeholder names from different countries, but none of this data was in the API response from Supaglue.

To resolve the issue, we went back to our OpenAPI spec and added English descriptions to each of our response’s fields:

type: array
items:
  type: object
  description: a list of emails associatd with this crm contact
  properties:
    email_address:
      type: string
      example: hello@supaglue.com
    email_address_type:
      type: string
      enum:
        - primary
        - work
  required:
    - email_address
    - email_address_type
example:
  - email_address: hello@supaglue.com
    email_address_type: work

Running the same prompts above now returned correct results:

Constructing a workflow using natural language

After we got authentication and read APIs working, we wanted to apply our plugin to automate a common sales prospecting workflow: fetching leads from a CRM, applying some processing, and adding their emails to an outbound sequence.

For our prototype, we built on the prompt above and added two additional steps to construct the following workflow:

1. Read from Supaglue’s CRM API to fetch 3 contacts
2. Remove fake emails
3. Add the post-processed contacts into an Outreach sequence

We approached this incrementally using trial-and-error and pretty quickly got ChatGPT to execute the workflow successfully and return correct results.

To test the limits of the system, we added a fourth step to the workflow: "show me all data on 3 recent crm contacts on new lines (scramble their last name, entire email, and phone), remove fake contacts, and add them as prospects into outreach”.

The goal of the “scramble” step was to anonymize PII for ChatGPT’s output. While we were able to scramble the emails, we weren't able to get ChatGPT to anonymize PII only for its own output but not for writing to Outreach. We tried a few variations including being more explicit, “… (scramble their last name, entire email, and phone for ChatGPT output), but ultimately didn't get the model to behave as intended:

Learnings and next steps

ChatGPT Plugins has the potential to unlock significant efficiency and automation in many sales use cases and workflows. However, they also introduce several new considerations for developers:

1. Metadata – in a pre-LLM software development world, engineers often ignore or underinvest in metadata for code (e.g. labels and descriptions) since code, not metadata, was what machines executed. In a post-LLM world where metadata is used for machine execution, it’s critical to provide high-quality metadata.
2. B2B platform features - today, ChatGPT plugins is designed to power consumer experiences. But it doesn't support custom headers and "multi-multi-tenancy" requirements that B2B API products like Supaglue has, which prevented us publishing our prototype. Fortunately, LangChain and other tools in the rapidly growing AI/LLM ecosystem do support some of these features, and we'll be exploring them next.

Overall, we were impressed by ChatGPT plugins in our prototyping journey and learned a lot along the way. We also know we’re just scratching the surface. If you’re building or thinking about building a ChatGPT plugin, we’d love to chat.

‍