Using AI providers

Golem comes with an extensive set of libraries for connecting to various AI and AI related third party providers. These libraries are available as regular Rust crates that can be added as dependencies to Rust-based Golem agents. They provide abstractions for various areas, and pluggable implementations for them. This way the actual third party provider can be switched any time without modifying the Golem agent’s code.

The Rust crates described below are Rust-only. TypeScript, Effect TypeScript, Scala, MoonBit, and Go agents can call provider HTTP APIs directly. A third-party client library must work with the guest runtime’s supported HTTP transport; libraries that require native sockets, JVM networking, or native Node.js extensions are not interchangeable with these APIs.

Call a provider directly

These examples send the same non-streaming OpenAI Chat Completions request. Pass apiKey from your agent’s secret configuration, not from a hard-coded string or a public method parameter. Use a model available to your provider account; the examples use gpt-4o-mini. They return the provider’s complete JSON response rather than assuming a particular choice or tool-call shape. Call the helper from an agent method. See HTTP requests for transport details.

TypeScript


async function askProvider(apiKey: string, prompt: string): Promise<unknown> {
  const response = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: prompt }],
    }),
  })
  const body = await response.text()
  if (!response.ok) throw new Error(`Provider returned HTTP ${response.status}`)
  return JSON.parse(body)
}

Effect TypeScript


import { Effect } from "effect"
import {
  FetchHttpClient,
  HttpClient,
  HttpClientRequest,
  HttpClientResponse,
} from "effect/http"
 
function askProvider(apiKey: string, prompt: string) {
  return HttpClientRequest.post("https://api.openai.com/v1/chat/completions").pipe(
    HttpClientRequest.setHeader("Authorization", `Bearer ${apiKey}`),
    HttpClientRequest.bodyJson({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: prompt }],
    }),
    Effect.flatMap(HttpClient.execute),
    Effect.flatMap(HttpClientResponse.filterStatusOk),
    Effect.flatMap((response) => response.json),
    Effect.provide(FetchHttpClient.layer),
  )
}

Return this Effect from a handler, or use yield* askProvider(apiKey, prompt) in Effect.gen. Effect’s HttpClient handles the request and response; FetchHttpClient.layer uses the supported global fetch transport, without a Node.js HTTP client. Non-2xx statuses and JSON decoding failures remain in the Effect error channel.

Scala


import scala.concurrent.Future
import scala.scalajs.concurrent.JSExecutionContext.Implicits.queue
import scala.scalajs.js
import scala.scalajs.js.Thenable.Implicits._
 
def askProvider(apiKey: String, prompt: String): Future[js.Dynamic] = {
  val payload = js.Dynamic.literal(
    model = "gpt-4o-mini",
    messages = js.Array(js.Dynamic.literal(role = "user", content = prompt))
  )
  for {
    response <- js.Dynamic.global.fetch(
      "https://api.openai.com/v1/chat/completions",
      js.Dynamic.literal(
        method = "POST",
        headers = js.Dynamic.literal(
          "Authorization" -> s"Bearer $apiKey",
          "Content-Type" -> "application/json"
        ),
        body = js.JSON.stringify(payload)
      )
    ).asInstanceOf[js.Promise[js.Dynamic]].toFuture
    body <- response.text().asInstanceOf[js.Promise[String]].toFuture
  } yield {
    if (!response.ok.asInstanceOf[Boolean])
      throw new RuntimeException(s"Provider returned HTTP ${response.status}")
    js.JSON.parse(body)
  }
}

This is Scala.js interop with the guest runtime’s global fetch, not a JVM HTTP client.

MoonBit

Import golemcloud/golem_sdk/http as @http, moonbitlang/core/encoding/utf8 as @utf8, moonbitlang/core/buffer as @buffer, and moonbitlang/core/json as @json in the component’s moon.pkg.


async fn ask_provider(api_key : String, prompt : String) -> Json {
  // Encode the provider request as JSON.
  let payload : Json = {
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": prompt }],
  }
  let bytes = @utf8.encode(payload.stringify())
 
  // Add authentication and identify the request body as JSON.
  let headers = @http.Fields::fields()
  guard headers.append(
    "authorization",
    @utf8.encode("Bearer \{api_key}").to_fixedarray(),
  ) is Ok(_) else {
    abort("Invalid authorization header")
  }
  guard headers.append(
    "content-type",
    @utf8.encode("application/json").to_fixedarray(),
  ) is Ok(_) else {
    abort("Invalid content-type header")
  }
 
  // Stream the encoded body and configure the HTTPS request.
  let body = @http.Stream::produce(async fn(sink) {
    guard sink.write_all_bytes(bytes[:]) else {
      abort("Upload closed")
    }
    sink.close()
  })
  let (request, sent) = @http.Request::new(
    headers,
    Some(body),
    @http.Future::ready(Ok(None)),
    None,
  )
  guard request.set_method(Post) is Ok(_) else {
    abort("Invalid method")
  }
  guard request.set_scheme(Some(Https)) is Ok(_) else {
    abort("Invalid scheme")
  }
  guard request.set_authority(Some("api.openai.com")) is Ok(_) else {
    abort("Invalid authority")
  }
  guard request.set_path_with_query(Some("/v1/chat/completions")) is Ok(_) else {
    abort("Invalid path")
  }
 
  // Send the request and check that its body was transmitted successfully.
  let response = match @http.send(request) {
    Ok(response) => response
    Err(_) => abort("Provider transport failed")
  }
  guard sent.get() is Ok(_) else {
    abort("Request transmission failed")
  }
 
  // Read the full response and check completion before decoding it.
  let status = response.get_status_code()
  let (incoming, completed) = @http.Response::consume_body(
    response,
    @http.Future::ready(Ok(())),
  )
  defer incoming.drop()
  let buffer = @buffer.new()
  while incoming.read(4096) is Some(chunk) {
    for byte in chunk {
      buffer.write_byte(byte)
    }
  }
  match completed.get() {
    Ok(Some(trailers)) => trailers.drop()
    Ok(None) => ()
    Err(_) => abort("Provider response body failed")
  }
 
  // Reject unsuccessful statuses, then return the provider's JSON response.
  guard status >= 200 && status < 300 else {
    abort("Provider returned HTTP \{status}")
  }
  @json.parse(@utf8.decode(buffer.to_bytes())) catch {
    _ => abort("Invalid provider JSON")
  }
}

This uses the SDK’s Preview 3 WASI HTTP wrapper, including request transmission and response completion checks. The compact helper aborts on errors; use an application error type in production.

Go


package provider
 
import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
 
    _ "github.com/golemcloud/golem/sdks/go/golem"
)
 
func AskProvider(apiKey, prompt string) (json.RawMessage, error) {
    payload, err := json.Marshal(map[string]any{
        "model": "gpt-4o-mini",
        "messages": []map[string]string{{"role": "user", "content": prompt}},
    })
    if err != nil { return nil, err }
    request, err := http.NewRequest(http.MethodPost,
        "https://api.openai.com/v1/chat/completions", bytes.NewReader(payload))
    if err != nil { return nil, err }
    request.Header.Set("Authorization", "Bearer "+apiKey)
    request.Header.Set("Content-Type", "application/json")
    response, err := http.DefaultClient.Do(request)
    if err != nil { return nil, fmt.Errorf("provider transport failed") }
    defer response.Body.Close()
    if response.StatusCode < 200 || response.StatusCode >= 300 {
        return nil, fmt.Errorf("provider returned HTTP %d", response.StatusCode)
    }
    var result json.RawMessage
    if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
        return nil, fmt.Errorf("invalid provider JSON")
    }
    return result, nil
}

The Golem SDK installs the WASI HTTP transport for net/http. Do not replace it with a transport that dials native sockets.

Do not log the authorization header, prompts, or provider response bodies. Set input and response size limits in production. HTTP status failures and invalid JSON need application error handling. Do not automatically retry a failed generation: a lost response does not prove the provider did not process or charge for the request.

Rust provider libraries

The Rust libraries support these areas:

Large Language Models (LLM)
Graph Databases
Search Engines
Web Search Engines
Speech to Text
Video Generation

To use one of these libraries in a Rust agent, add the corresponding core crate and the desired provider crate(s) as dependencies. The golem.yaml files coming from the agent templates include commented-out entries for each possible choice of implementations, so just uncomment the desired ones and provide any necessary configuration (usually in the form of environment variables).

The rest of the page describes these Rust libraries, their implementations, and their core traits (without a full documentation of every type involved).

LLMs

The list of supported LLM providers is the following:

The core trait for LLMs (from the golem-llm crate):


pub trait Guest {
    type ChatStream: GuestChatStream;
 
    /// Make a single call to the LLM. To continue the conversation:
    /// - append tool responses and new messages to the events and use send again
    /// - or use the chat-session wrapper, which helps in maintaining the chat events
    fn send(events: Vec<Event>, config: Config) -> Result<Response, Error>;
 
    /// Makes a single call to the LLM and gets back a streaming API
    /// to receive the response in chunks.
    fn stream(events: Vec<Event>, config: Config) -> ChatStream;
}
 
pub trait GuestChatStream {
    fn poll_next(&self) -> Option<Vec<Result<StreamEvent, Error>>>;
    fn get_next(&self) -> Vec<Result<StreamEvent, Error>>;
}

Graph Databases

The list of supported graph databases is the following:

The core trait for graph databases (from the golem-graph crate):


/// Connection
pub fn connect(config: ConnectionConfig) -> Result<Graph, GraphError>;
 
/// The Transaction resource exposes all CRUD and traversal operations
impl Transaction {
    // === QUERY OPERATIONS ===
    pub fn execute_query(&self, options: &ExecuteQueryOptions) -> Result<QueryExecutionResult, GraphError>;
 
    // === TRAVERSAL OPERATIONS ===
    pub fn find_shortest_path(&self, options: &FindShortestPathOptions) -> Result<Option<Path>, GraphError>;
    pub fn find_all_paths(&self, options: &FindAllPathsOptions) -> Result<Vec<Path>, GraphError>;
    pub fn get_neighborhood(&self, options: &GetNeighborhoodOptions) -> Result<Subgraph, GraphError>;
    pub fn path_exists(&self, options: &PathExistsOptions) -> Result<bool, GraphError>;
    pub fn get_vertices_at_distance(&self, options: &GetVerticesAtDistanceOptions) -> Result<Vec<Vertex>, GraphError>;
    pub fn get_adjacent_vertices(&self, options: &GetAdjacentVerticesOptions) -> Result<Vec<Vertex>, GraphError>;
    pub fn get_connected_edges(&self, options: &GetConnectedEdgesOptions) -> Result<Vec<Edge>, GraphError>;
 
    // === VERTEX OPERATIONS ===
    pub fn create_vertex(&self, options: &CreateVertexOptions) -> Result<Vertex, GraphError>;
    pub fn create_vertices(&self, vertices: &[CreateVertexOptions]) -> Result<Vec<Vertex>, GraphError>;
    pub fn get_vertex(&self, id: &ElementId) -> Result<Option<Vertex>, GraphError>;
    pub fn update_vertex(&self, options: &UpdateVertexOptions) -> Result<Vertex, GraphError>;
    pub fn delete_vertex(&self, id: &ElementId, delete_edges: bool) -> Result<(), GraphError>;
    pub fn find_vertices(&self, options: &FindVerticesOptions) -> Result<Vec<Vertex>, GraphError>;
 
    // === EDGE OPERATIONS ===
    pub fn create_edge(&self, options: &CreateEdgeOptions) -> Result<Edge, GraphError>;
    pub fn create_edges(&self, edges: &[CreateEdgeOptions]) -> Result<Vec<Edge>, GraphError>;
    pub fn get_edge(&self, id: &ElementId) -> Result<Option<Edge>, GraphError>;
    pub fn update_edge(&self, options: &UpdateEdgeOptions) -> Result<Edge, GraphError>;
    pub fn delete_edge(&self, id: &ElementId) -> Result<(), GraphError>;
    pub fn find_edges(&self, options: &FindEdgesOptions) -> Result<Vec<Edge>, GraphError>;
 
    // === TRANSACTION CONTROL ===
    pub fn commit(&self) -> Result<(), GraphError>;
    pub fn rollback(&self) -> Result<(), GraphError>;
    pub fn is_active(&self) -> bool;
}

Search Engines

The list of supported search engines is the following:

The core trait for search engines (from the golem-search crate):


pub trait Guest {
    type SearchStream: GuestSearchStream;
 
    // Index lifecycle
    fn create_index(options: CreateIndexOptions) -> Result<(), SearchError>;
    fn delete_index(name: IndexName) -> Result<(), SearchError>;
    fn list_indexes() -> Result<Vec<IndexName>, SearchError>;
 
    // Document operations
    fn upsert(index: IndexName, doc: Doc) -> Result<(), SearchError>;
    fn upsert_many(index: IndexName, docs: Vec<Doc>) -> Result<(), SearchError>;
    fn delete(index: IndexName, id: DocumentId) -> Result<(), SearchError>;
    fn delete_many(index: IndexName, ids: Vec<DocumentId>) -> Result<(), SearchError>;
    fn get(index: IndexName, id: DocumentId) -> Result<Option<Doc>, SearchError>;
 
    // Query
    fn search(index: IndexName, query: SearchQuery) -> Result<SearchResults, SearchError>;
    fn stream_search(index: IndexName, query: SearchQuery) -> Result<SearchStream, SearchError>;
 
    // Schema inspection
    fn get_schema(index: IndexName) -> Result<Schema, SearchError>;
    fn update_schema(index: IndexName, schema: Schema) -> Result<(), SearchError>;
}
 
pub trait GuestSearchStream {
    fn get_next(&self) -> Option<Vec<SearchHit>>;
    fn blocking_get_next(&self) -> Vec<SearchHit>;
}

Web Search Engines

The list of supported web search engines is the following:

The core trait for web search engines (from the golem-web-search crate):


pub trait GuestWebSearch {
    type SearchSession: GuestSearchSession;
 
    /// Start a search session, returning a search context
    fn start_search(params: SearchParams) -> Result<SearchSession, SearchError>;
 
    /// One-shot search that returns results immediately (limited result count)
    fn search_once(params: SearchParams)
        -> Result<(Vec<SearchResult>, Option<SearchMetadata>), SearchError>;
}
 
pub trait GuestSearchSession {
    /// Get the next page of results
    fn next_page(&self) -> Result<Vec<SearchResult>, SearchError>;
 
    /// Retrieve session metadata (after any query)
    fn get_metadata(&self) -> Option<SearchMetadata>;
}

Speech to Text

The list of supported speech to text providers is the following:

The core trait for speech to text providers (from the golem-stt crate):


pub trait SttTranscriptionGuest {
    fn transcribe(req: TranscriptionRequest) -> Result<TranscriptionResult, SttError>;
 
    fn transcribe_many(
        requests: Vec<TranscriptionRequest>,
    ) -> Result<MultiTranscriptionResult, SttError>;
}

Video Generation

The list of supported video generation providers is the following:

The core traits for video generation providers (from the golem-video crate):


pub trait VideoGenerationGuest {
    fn generate(input: MediaInput, config: GenerationConfig) -> Result<String, VideoError>;
    fn poll(job_id: String) -> Result<VideoResult, VideoError>;
    fn cancel(job_id: String) -> Result<String, VideoError>;
}
 
pub trait LipSyncGuest {
    fn generate_lip_sync(video: LipSyncVideo, audio: AudioSource) -> Result<String, VideoError>;
    fn list_voices(language: Option<String>) -> Result<Vec<VoiceInfo>, VideoError>;
}
 
pub trait AdvancedGuest {
    fn extend_video(options: ExtendVideoOptions) -> Result<String, VideoError>;
    fn upscale_video(input: BaseVideo) -> Result<String, VideoError>;
    fn generate_video_effects(options: GenerateVideoEffectsOptions) -> Result<String, VideoError>;
    fn multi_image_generation(options: MultImageGenerationOptions) -> Result<String, VideoError>;
}