Using the AI Chatbot API

Sometimes you need to apply customisations beyond what you can do with custom CSS. For these cases, you can use the raw API yourself, and create your own completely custom frontend. This can be useful for creating an Angular or React component, or for cases where you have to create your own custom JavaScript. This article explores the API in details, allowing you to roll your own JavaScript or client side code consuming the API.

We will be using JavaScript here, but assuming you can find a SignalR client-side library for your platform of choice, you’ll typically just have to exchange the client-side SignalR code to accommodate for your specific client. Microsoft provides client-side SignalR libraries for iOS, Android, JavaScript, and probably all other popular platforms out there.

API Endpoint Method

The chatbot is primarily using one HTTP method. This is a POST method, and its fully qualified URL depends upon the name of your cloudlet - But its relative URL is as follows; /magic/system/openai/chat.

This HTTP POST endpoint can be invoked either by sending your content as JSON, or alternatively pass in your content as multipart/form-data. You can choose what content type to send, but basically the difference is that if you send your content as multipart/form-data, then you can upload files, such as images, and have the AI analyse these as you see fit. The endpoint accepts the following arguments;

  • prompt - This is the prompt supplied by the user. Mandatory.
  • type - This is the machine learning type you wish to use. Mandatory.
  • references - Whether the endpoint should return URL references to RAG training snippets being used to answer the question. Optional, defaults to false.
  • chat - Whether the endpoint should invoke the LLM. If false, allows for returning only references. Optional, defaults to true.
  • recaptcha_response - If your machine learning type is configured to use Google reCAPTCHA, then this is the reCAPTCHA token. If not, it’s expected to be a Magic PoW token. Notice, we advice all our clients to use Magic PoW CAPTCHA since it’s much smaller in size, and also “good enough” for most use cases. Read more about Magic’s PoW CAPTCHA here. Mandatory argument.
  • user_id - User ID. This can be a randomly created ID of some sort, for instance stored in the browser’s localStorage, but to allow for seeing all prompts generated by individual users, we recommend you generate a user ID that can somehow be reused, while also being unique, such as for instance username, etc. Mandatory argument.
  • session - ID uniquely identifying the user’s session. Similar to user ID, but must be unique for each session.
  • stream - If true, the AI chatbot will stream its response back to the client using SignalR instead of returning the content as the invocation’s response. Optional, defaults to false.
  • data - Additional context, allowing the client to supply additional context to the backend. Optional.
  • referrer - HTTP referrer field, allowing you to understand from the server side what URL the request was originating from. Optional argument. The cloudlet will try to determine the HTTP Referrer header’s value, and use this by default, unless you provide an explicit override using this argument.
  • file - Only relevant if you invoke the endpoint with multipart/form-data content, at which point this is expected to contain a file reference to for instance an image or something you want the LLM to analyse. Optional argument.

Below is a typical example payload in JSON.

{
  "prompt": "How's the weathed in Larnaca?",
  "type": "weather_information",
  "references": false,
  "chat": true,
  "recaptcha_response": "MAGIC_CAPTCHA_TOKEN_OR_RECAPTCHA_TOKEN",
  "user_id": "UNIQUE_USER_ID_OR_USERNAME",
  "session": "UNIQUE_SESSION_ID",
  "stream": false,
  "referrer": "https://foo.somewhere.com"
}

The above invocation will not stream the response, but rather return the answer immediately as the response of your HTTP invocation, due to its stream argument being false. Typically you would want to rather use streaming by setting this to true.

Streaming data

If you set the above stream argument to true, the above endpoint will not wait the LLM’s response, but rather stream the response one token at the time to a SignalR channel with its name being the same as your session value, and the endpoit will return immediately long before the response is ready. This implies you first have to connect to the cloudlet using SignalR from your client, for then to listen to SignalR events using your unique session ID.

This argument should as a general rule of thumb almost always be set to true, to allow for streaming the response. The reasons for this is that the CDN we’re using will shut down the HTTP connection if the cloudlet takes more than 60 seconds to answer. And with multiple questions, maybe analysing files too, and also possibly multiple function invocations within the cloudlet, the LLM might easily end up in a situation where it requires more than 60 seconds to provide an answer.

This implies the steps for initiating a chat session with your cloudlet becomes as follows;

  1. Connect to cloudlet using SignalR
  2. Subscribe to SignalR messages having the same name as your session value, which must be a unique value
  3. Transmit your chat message with the same session you used to connect to SignalR
  4. Handle messages from within your SignalR on callback

Below is some example JavaScript code to understand how to deal with individual messages.

this.socket = new signalR.HubConnectionBuilder()
.withAutomaticReconnect()
.withUrl(this.ainiro_settings.url + '/sockets', {
  skipNegotiation: true,
  transport: signalR.HttpTransportType.WebSockets,
}).build();

// Subscribing to messages.
this.socket.on(this.session, (args) => {

  // Creating object from JSON.
  const obj = JSON.parse(args);

  // Checking if we're finished.
  if (obj.finished === true) {

    // The cloudlet is done responding to the prompt
    
  } else if (obj.error === true) {

    // The LLM is finished, but with errors

  } else if (obj.function_waiting) {

    // Waiting for function invocation.

  } else if (obj.function_result) {

    // Function invocation succeeded.

  } else if (obj.function_error) {

    // Function invocation failed.

  }

  // Verifying we've got some sort of message given.
  if (obj.message) {

    // Appending message to temporary response.
    this.response += obj.message;

    // Use the above `this.response` somehow, typically involving transforming it from Markdown to HTML, etc ...
  }
});

// Starting socket connection.
this.socket.start();

The above assumes you’ve got a this.socket property on your object, being the actual sockect connection where the cloudlet should transmit streamed messages. In addition it assumes you’ve got a this.response property, intended to contain the complete response from the LLM. Notice, only the delta parts are transmitted as obj.message, and not the entire message - So it’s up to your client-side code to keep track of messages previously sent - Which is why the above this.response += obj.message works.

Streaming is the most difficult parts to get correct when implementing your own client-side code, but it’s crucial to get these parts right, and it’s also highly rewarding to see the response from the LLM being streamed back to the client, one token at the time.

Also notice the function related parts of the above JavaScript callback. These are invoked when the cloudlet is executing an “AI function”, and you will get information about the function invocation in this callback, allowing you to display visual clues to the caller that the LLM is waiting for a function invocation to finish.

It is also important that you clean up your SignalR object by making sure that the socket connection is closed in a clean manner once you’re done with your chat session, and you obviously need a client-side SignalR library to connect to the cloudlet. The latter should be easily accomplished regardless of what client you’re building, since Microsoft have created client-side SignalR libraries for all popular platforms.

Below is a list of all possible callbacks you can expect, with a simple explanation explaining its purpose.

  • finished - SignalR message transmitted by the backend when the request has finished
  • error - SignalR message transmitted if the request triggered an error of some sort
  • function_waiting - SignalR message transmitted when the cloudlet is waiting for an AI function invocation to finish
  • function_result - SignalR message transmitted when an AI function is finished
  • function_error - SignalR message transmitted when an AI function invocation resulted in an error of some sort

If you handle the above types on your object callback, you should be set for dealing with most scenarios. Notice, if you need further help, AINIRO provides support and custom software development services for our platform - At which point you can contact us below.