LLM (Ollama)

Introduction

The LLM (Ollama) input integrates Composer with a self-hosted Ollama server, letting a large language model generate text on demand for on-screen graphics — subtitles, lower-thirds, tickers, captions, commentary. Drive it from the property panel or from ScriptEngine; read responses as a property or via a JavaScript callback.

Quick start

1. Install Ollama and pull a chat/instruct model — see Installing Ollama and pulling a model below.
2. In Composer, add the LLM (Ollama) input from the AI category.
3. Set Server URL to the address of your Ollama server (default: http://localhost:11434).
4. Click Connect. The Available Models dropdown is populated from the server.
5. Select a model from Available Models, type a prompt in User Prompt, and click Send Prompt.
6. The response appears in Last Response and is delivered to the Callback Function if one is defined.

Installing Ollama and pulling a model

Ollama is a free, open-source runtime for large language models. It runs on Windows and Linux, uses the GPU automatically when one is available, and exposes a small HTTP API on localhost:11434. Install it from ollama.com/download — for OS-specific install steps, configuration, networking, GPU pinning, and Modelfile authoring, follow the upstream docs at docs.ollama.com.

The LLM Input requires a chat or instruct model — most Ollama models qualify. Avoid pure embedding models (typically with embed in the name); they produce numeric vectors, not text. Pull the model from a terminal:

ollama pull <name>:<tag>                 # from ollama.com/search
ollama pull hf.co/<user>/<repo>:<tag>    # from huggingface.co/models (GGUF builds)

API key authentication

Composer can still send a Bearer token when the endpoint requires one. Set the COMPOSER_OLLAMA_APIKEY environment variable on the Composer machine and the LLM Input attaches it to every request as Authorization: Bearer <value>. Three situations where this matters:

• Ollama Cloud — https://ollama.com requires a key. Use the Ollama Cloud key here.
• A self-hosted Ollama behind a reverse proxy that's been configured to validate a Bearer token before forwarding to Ollama on localhost. The key value is whatever the proxy expects.
• Any Ollama-API-compatible service that does Bearer authentication.

Leave the variable unset when connecting to a plain local ollama serve — no key is needed and none would be checked.

setx COMPOSER_OLLAMA_APIKEY "your-secret-key"

export COMPOSER_OLLAMA_APIKEY=your-secret-key

Environment=COMPOSER_OLLAMA_APIKEY=your-secret-key

Why self-hosted AI

All inference runs on hardware you control. Practical advantages over hosted cloud services:

• Privacy — prompts and responses never leave the local network.
• No per-token cost — inference runs on existing hardware.
• Low latency — no round-trip to a cloud provider.
• Offline operation — no internet dependency once the model is pulled.
• Resource isolation — run Ollama on a dedicated AI box to keep CPU/GPU cycles available for Composer's video pipeline.
• Version control — pin a model version per project; updates are explicit, never silent.

On-premise AI presenter pipeline

Combined with the Text-to-Speech input, the LLM Input completes a fully on-premise AI presenter pipeline:

prompt → text response → synthesised speech → on-screen graphics

A short System Prompt (e.g. "You are an energetic stadium announcer.") keeps voice and tone consistent across a show.

Common use cases

• AI presenter and voice-over — chain the response into Text-to-Speech for an on-screen narrator, analyst, or commentator.
• Creative on-screen copy — walk-on intros, goal celebrations, sponsor reads, segment hooks, stadium-style PA announcements.
• Live graphics copy — lower-thirds, tickers, callouts, news crawls, score-bug text — driven from the property panel or from scripts triggered by game state, scene transitions, or Animator events.
• Live text augmentation — translate, summarise, or rephrase incoming wire copy, social-media posts, or viewer questions before they hit the screen.
• Pre-show prep — generate teasers, lower-third copy, and segment lead-ins for the operator to schedule into the rundown.

Things to keep in mind

LLMs are powerful but probabilistic — plan around these realities when deploying them in broadcast:

• Always editorially review. Treat generated text as copy that needs an operator's eye before it goes on air, not as verified fact.
• Hallucinations happen. Models can produce confidently-wrong names, numbers, dates, and quotations. Verify against the source for anything fact-sensitive.
• Identical prompts can produce different outputs. That's by design — sampling adds variety. Use the System Prompt to constrain tone and length, and the Seed setting to lock outputs reproducibly during testing.

LLM (Ollama) - Settings

Connection

Connection — where the Ollama server lives and how to authenticate against it.

Model

Model — pick which installed Ollama model handles the conversation.

Response Tuning

Response Tuning — per-request sampling knobs. Auto-populated from the selected model's Modelfile (where it overrides) or Ollama's hardcoded floor (where it does not). Edit any value to override for the next prompt.

Prompt

Prompt — the system prompt and the user prompt that shape each request.

Response

Response — what came back from the model on the latest exchange.

Chat History

Chat History — save and load chat conversations to disk so they survive between sessions.

Status

Status — what the input is doing right now and how full the chat context is.

Inherits from: AbstractInput, AbstractAudioProcessing, AbstractAudioMetering.

See also: LLM (Ollama) in Script Engine Objects.

Shared input properties

Every input — regardless of source type — exposes the following property groups. They are surfaced in the property panel only when Show advanced options is enabled on the input.

Icon

• Icon text — short text shown on the input's icon in the Inputs list. Useful as a quick visual label (channel number, mic name, camera position) to tell otherwise-similar inputs apart at a glance. Empty by default; has no effect on rendering or routing.

Audio mixer

• Hide in audio mixer — when on, hides the input from the audio mixer view without disabling its audio. Useful for de-cluttering the mixer while keeping the audio routed (e.g. fixed background music, ambient beds, pre-aligned playout). [default=false]

Render Options

• Invisible (Do not render in scene) — when on, the input is skipped during rendering and produces no picture on any layer or scene. Audio routing is unaffected. Toggle from a script for cued-in / cued-out behaviour during a show. [default=false]
• Do not render input — disables the input's internal render entirely (no decode or capture work is done). Stronger than Invisible: that one renders but doesn't display; this one stops the input from doing any work at all. Useful for reducing CPU / network load on heavy sources (e.g. high-bitrate RTMP / SRT streams, large media files) when the input is temporarily not needed. Audio meters are cleared while disabled. [default=false]
• Do not render input controller — chooses what drives the Do not render input flag. Let Composer decide (the default) hands control to the project-level Render Tuning optimiser, which automatically pauses inputs that aren't used by any active scene. Manual Configuration ignores Render Tuning and lets the Do not render input toggle control the flag directly — use this to keep a network source warm even when it's currently off-air, or to take a heavy input down by hand regardless of scene activity. [default=Let Composer decide]

Optional TAGS

• TAGS — one or more free-form tag words used to classify this input (typically space- or comma-separated). Picked up by Composer's Smart Search to filter or find inputs by category — e.g. camera, music, interview, sponsor. Has no effect on rendering.

Audio configuration and processing options

For inputs capable of processing audio, additional audio configuration and processing options are available through the audio mixer and the Channel Strip Inspector.

• Audio mixer — monitor levels, adjust gain and pan, mute / solo inputs, and configure auxiliary sends to Audio Channel Strip submix buses, all from a centralised mixer-style interface.
• Channel Strip Inspector — advanced per-strip audio processing for the selected input:
  • Input trim, stereo remapping, and audio delay
  • Channel mapping (8-channel mode unlocks the full MAPPING tab)
  • Gate
  • Low-cut filter
  • Equaliser (5-band parametric)
  • Compressor
  • Sidechain ducking (a second compressor whose gain reduction is driven by another input's level — e.g. dipping music under a voice-over)
  • Limiter

For the full audio signal flow, see Audio processing workflow.

Connecting to Ollama on a different machine

1. On the Ollama machine, set the OLLAMA_HOST environment variable to 0.0.0.0:11434 so the server binds to all interfaces (by default it listens only on localhost).
2. Restart Ollama.
3. In Composer, set Server URL to http://<ollama-machine-ip>:11434.
4. Ensure port 11434 is open in any firewall between the two machines.

Securing Ollama with a reverse proxy

The standard pattern for adding TLS, authentication, or both to a self-hosted Ollama is to put a reverse proxy in front of it. Ollama keeps listening on localhost:11434; clients talk only to the proxy:

  Client  ──https──▶  Reverse Proxy  ──http──▶  Ollama
 (Composer)        (nginx / Caddy)         (localhost:11434)

The proxy handles two jobs Ollama can't do itself: TLS termination, and Bearer-token validation. Configure either, both, or neither depending on what the deployment needs.

nginx — TLS only

server {
    listen 443 ssl;
    server_name ollama.example.com;

    ssl_certificate     /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;

    location / {
        proxy_pass http://localhost:11434;
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

proxy_buffering off keeps streamed responses flowing smoothly; proxy_read_timeout 300s prevents long generations from timing out on large models.

nginx — TLS plus Bearer-token authentication

Reject any request that doesn't carry the expected Bearer token before forwarding to Ollama:

server {
    listen 443 ssl;
    server_name ollama.example.com;

    ssl_certificate     /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;

    location / {
        if ($http_authorization != "Bearer your-secret-key") { return 401; }

        proxy_pass http://localhost:11434;
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

On the Composer machine, set COMPOSER_OLLAMA_APIKEY=your-secret-key so the LLM Input sends the matching Authorization: Bearer your-secret-key header. See API key authentication above for per-OS env-var setup.

Caddy — TLS only

A minimal Caddy configuration auto-provisions a Let's Encrypt certificate on first request:

ollama.example.com {
    reverse_proxy localhost:11434
}

Caddy — TLS plus Bearer-token authentication

ollama.example.com {
    @authorized header Authorization "Bearer your-secret-key"
    handle @authorized { reverse_proxy localhost:11434 }
    respond 401
}

In Composer, set Server URL to https://ollama.example.com with either proxy. The TLS handshake is handled transparently by the underlying HTTP client.

Things to keep in mind

Large language models vary considerably from one to the next, and the same model can produce different answers to the same prompt across runs.

Model size and hardware requirements

Model size is measured in parameters, abbreviated B for billion. Size correlates with the model's reasoning capacity and the breadth of its general knowledge, but also with VRAM requirements and per-token latency.

For most live broadcast use-cases, a Small / Mid (4B–9B) model strikes the best balance of latency, output quality, and VRAM headroom alongside Composer's video pipeline.

Quantization

Most models are published in multiple quantizations (Q4, Q5, Q8, FP16, …). Lower numbers mean smaller files and lower VRAM usage at a small cost in output quality. Ollama defaults to a sensible quantization per model — typically Q4_K_M, which balances size and quality well. The default is appropriate for nearly all workflows; switch only with a specific reason.

System prompts

A precise System Prompt ("You are a sports commentator. Reply in one sentence, present tense, no emojis.") substantially improves consistency of tone, length, and format. Adherence varies by model: mid-size and larger models tend to follow system prompts reliably, while tiny models often revert to their default style. Validate the system prompt against representative user prompts before going live.

Stateless inference and chat history

Ollama is stateless: the server retains no information about previous prompts or replies — each request arrives at the model without context.

Conversational behaviour is reconstructed entirely on the Composer side. The LLM Input maintains the chat history locally and resends the full conversation with every request: system prompt, every prior user message, every prior assistant reply, followed by the new user message. The model reads the complete transcript fresh, generates the next response, and the cycle repeats.

Because Ollama is stateless, the conversation only exists for as long as it fits inside a single request to the model. Long histories cost more tokens, take longer to process, and increase the likelihood of the model latching onto irrelevant earlier content — for independent prompts, prefer New Chat between requests.

Language coverage

Most models perform best in English. Multilingual capability varies considerably — some models support only a handful of languages well, while others cover dozens. For non-English use, evaluate several candidate models with realistic prompts before committing.

Prompt specificity

Output quality is shaped directly by prompt quality. Vague prompts produce vague answers. Specify format, length, tone, and constraints explicitly. Compare:

• "Write something about football." — unpredictable.
• "Write one sentence of energetic match commentary for a goal scored by Player X in the 87th minute." — consistent, usable.

Saving and loading chats

Turn Enable chat history on and every exchange is saved automatically to <Documents>/Composer/LLM Chats/. Files are created and rewritten for you — there is no Save button.

• To resume a previous chat, pick it from Available chats. The full session is restored, and the next prompt continues the conversation.
• To start fresh, click New Chat (or pick the empty top entry in Available chats). Your model and System Prompt are preserved; only the conversation resets.
• To browse, rename, or delete saved chats, click Open Chats Folder.

If a saved chat references a model that's no longer installed on the server, the load is refused with a warning above the dropdown — ollama pull the missing model (or pick a different chat) to recover.

Token usage and context

LLMs have a fixed context window — the maximum number of tokens that can be considered in a single request. A token is roughly ¾ of an English word (1000 tokens ≈ 750 words). The chat history is sent in full on every request and grows with each exchange.

Typical context-window sizes:

• 4K–8K tokens — older or compact models; fills quickly in extended conversations.
• 32K–128K tokens — most modern mid-size models.
• 1M+ tokens — a small number of cutting-edge models.

How to read the Status panel:

• Tokens Used is the size of the most recent request — that is, the size of the chat history — not the length of the reply.
• Context Window is the model's maximum.
• When Tokens Used approaches Context Window, start a New Chat to avoid imminent truncation.

When the history exceeds the context window, Ollama silently drops the oldest user and assistant messages before the request reaches the model; the System Prompt is preserved. The Truncation Count field increments on each occurrence — once it has, earlier turns are gone for good.

Overriding the context window

Each Ollama model ships with a Modelfile-defined default context window — often 2048 or 4096 tokens — which is frequently smaller than what the underlying model can actually attend over (a model trained for 32K tokens may still default to 4096 in its Modelfile). The Context size dropdown in the Model section lets the project override this per-request without touching the Ollama CLI or building a derived model:

• Model default (default) — Composer omits the override; Ollama uses the Modelfile value.
• 2048 … 131072 — Composer attaches options.num_ctx = <selected> to every chat request. Ollama allocates a KV cache of that size and reloads the model on the first request after the value changes.

Trade-offs: larger contexts allow longer chats and more in-context material before truncation, but VRAM usage scales linearly with num_ctx. Setting a value larger than the model's compiled context length is allowed but does not improve quality — the model can still only meaningfully attend over its trained window. The selected value is locked once a chat is active; a change takes effect on the next prompt sent after a New Chat.

Response Tuning

Six properties under Response Tuning shape how the model picks each next word. Five are visible by default; Seed is hidden behind Composer's Show Advanced Options toggle since it's only useful for development and testing.

For most projects, the defaults are correct. When you select a model, the LLM Input reads the model's Modelfile and pre-fills the six fields with the model author's recommended values (falling back to Ollama's defaults for anything the Modelfile doesn't specify). Adjust them only when you have a specific reason — most changes make output worse.

The six options are per-request — they are not locked while a chat is active and may be changed mid-conversation. Switching models replaces five of them (Temperature, Top P, Top K, Seed, Max Output Tokens) with the new model's defaults; Stop Sequences are preserved across model changes since they're your custom stops, not the model's.

When to adjust

Stop sequences

The Stop Sequences field is for your own stop strings only — for example User: to prevent the model from roleplaying both sides of a conversation, or a marker like --- to halt at a specific boundary. Add one per line. The model halts the moment it produces any of them.

Each model also has its own internal "I'm done speaking" signals — special tokens that look something like <|eot_id|>, <end_of_turn>, or <|im_end|>, depending on how the model was trained. The LLM Input applies these automatically in the background — you don't see them, you don't need to add them, and you can't accidentally remove them.

When to lock the Seed

For typical broadcast use, leave Seed at -1 — variety is what you want when copy is going on air. Lock it to a specific number only during development and testing, when you need reproducible output: comparing prompts, debugging odd responses, A/B testing other knobs, or recording a demo. Switch back to -1 once the prompt is finalised and going on air.

The seed value is a label, not a magnitude. Seed = 42 isn't "more random" than Seed = 1,000,000 — they just produce different deterministic outputs. Any positive integer works equally well; the only practical concern is writing down the value so you can reproduce the same output later.

How Top P and Top K actually work

Top P is NOT a fraction of words — it's a probability threshold.

How it works:

1. The model assigns a probability to every word in its vocabulary (~50,000 words). Most are tiny; a handful carry most of the probability. They sum to 1.
2. Sort the probabilities highest-first.
3. Walk down the list, adding each probability to a running total.
4. Stop the moment the running total reaches or exceeds the Top P value.
5. The words gathered so far = the eligible pool.

Top P doesn't pick the final word — it just gathers the most-likely words until the threshold is hit. The actual picking happens later, after Temperature scaling and the Seed-driven random draw.

Example: after "The capital of Sweden is", the model assigns Stockholm probability 0.85, Göteborg 0.08, and tiny values to everything else. With Top P = 0.3, the running total hits 0.85 in one step — already past 0.3 — so the pool is just {Stockholm}. Counterintuitively, lower Top P often means fewer words, not more. That's why useful values cluster in 0.9–1.0 — below that, the pool collapses to one or two words and the model has almost no choice in what to say next.

Top K is the same idea with a different stopping rule: "stop when I've gathered K words." Always exactly K candidates, regardless of how confident the model is. Top P adapts to the probability shape; Top K doesn't. Top K = 0 disables the filter entirely — the pool is then determined by Top P alone.

When both are set, they compose: Top K gathers first (the top K words), then Top P gathers further within that K-word subset until its threshold is hit. The smaller of the two pools wins.

Pipeline order

The six options have three distinct roles when the model picks each word:

• Top P / Top K → gather the candidate pool.
• Temperature → reshape the weighting within that pool (more peaked = predictable; flatter = creative).
• Seed → pick a word from the pool, weighted by the reshaped odds.

After each new word is added to the response, two halt conditions are checked:

• Stop Sequences — if the response so far ends with any of your stop strings, halt immediately.
• Max Output Tokens — if the response has reached the token limit, halt immediately.

Practical consequence: Temperature = 0.0 collapses the pool to a single word — making Top P, Top K, and Seed irrelevant. That's why Temperature 0 always gives the same answer.

Thinking / reasoning models

Reasoning-capable models — those that advertise a thinking capability — perform an internal reasoning phase before producing the final answer.

• Enable thinking: Yes — the reasoning phase streams separately. The Status field shows Thinking during this phase, then Receiving when the final answer begins. Reasoning tokens are not included in Last Response.
• Enable thinking: No — the reasoning phase is skipped where the model permits. Faster responses, lower quality on tasks that benefit from reasoning.
• On models without a thinking capability, the setting is ignored.

ScriptEngine examples

These examples assume the standard Composer script boilerplate at the top of the file:

var VindralEngine = importNamespace('VindralEngine');
var Project = VindralEngine.Project.RunningInstance;

See Script Engine — Overview for the full host API.

Feed responses to a Text input

When a response is received, the input invokes the JavaScript function named in Callback Function with a single argument — a JSON string with prompt, response, model, and messageCount fields:

function onLlmResponse(payload) {
    var data = JSON.parse(payload);
    Project.SetObjectProperty("Subtitle", "Text", data.response);
}

Set Callback Function to onLlmResponse. "Subtitle" is the name of a Text input that displays the reply on screen.

Drive the input from a script

The Send Prompt and New Chat commands are scriptable. Set the User Prompt and execute the command:

function askModel(question) {
    var llm = Project.GetInputByName("LLM (Ollama)");
    llm.UserPrompt = question;
    llm.SendPromptCommand.Execute();
}

Load a specific saved chat

var llm = Project.GetInputByName("LLM (Ollama)");
llm.AvailableChats.SelectedValue = "MyProject_20260423_142000_Greetings.jsonl";

The filename must match the entry under <Documents>/Composer/LLM Chats/ exactly (no directory path). Missing files or missing models surface a warning via the property-panel note rather than throwing an exception.

Troubleshooting

Technical notes

• Responses stream token-by-token from the server. The Last Response property is updated once at the end of the stream with the full text.
• Changing the selected model mid-session resets the chat history because different models tokenize and interpret conversations differently. To prevent accidental loss, the model dropdown is locked while a chat is active — start a New Chat to unlock it.
• The System Prompt is locked for the same consistency reason: changing it mid-conversation would let earlier turns and later turns operate under different instructions, producing inconsistent behaviour. New Chat preserves the existing System Prompt text and re-enables editing.
• The Context size dropdown is locked while a chat is active for two reasons: lowering it mid-chat would force Ollama to truncate older history to fit the smaller cache (silent context loss), and raising it would force Ollama to unload and reload the model with a larger KV cache (slow first prompt, possible CPU offload, possible VRAM exhaustion). New Chat preserves the selection and re-enables editing.
• Saved chat files use a compact JSONL format under <Documents>/Composer/LLM Chats/ and are rewritten automatically when older exchanges drop out of the model's context window — file size stays bounded by the context window, no matter how long the session runs. A .jsonl.lock sidecar prevents two Composer instances from writing to the same chat at once and is removed automatically when the chat is closed.
• Callback Function runs synchronously when the response arrives — route the text to the appropriate operator and return; defer any heavy work.
• The component produces no video or audio frames — it is a data-only input that exposes text via ScriptEngine and the Last Response property.

Shared-weight custom models

When you build a custom model with a Modelfile that starts with FROM some-base-model — e.g. a custom assistant with a baked-in system prompt — Ollama does not duplicate the underlying weight blob on disk or in VRAM. Both tags point at the same bytes. The first one you chat with triggers the load; a subsequent chat against the other tag is served from the same VRAM instantly.

The side effect: Ollama's /api/ps endpoint lists the blob only under the name that triggered the load. If you created my-assistant from a base model, sent one prompt to the base, and then switched to my-assistant, /api/ps keeps reporting the base name as the running model — even though your next prompt is being answered by my-assistant.

The LLM Input handles this transparently: when the exact tag-name match fails but exactly one model is listed as running, the panel shows that model's VRAM / Processor / Context stats. Those numbers correctly describe what your chat is actually running on. The only visible cue that this happened is a Debug-level log line.

Related components

• Text to Speech — chain the LLM response into Text to Speech for a fully on-premise AI presenter pipeline: prompt → text → synthesised speech → on-screen graphics.
• Speech To Text — the inverse direction: capture spoken audio, transcribe to text, and feed the transcripts back into the LLM Input as prompts. Useful for live Q&A bots, automated commentary based on audio cues, and voice-driven graphics workflows.