EN ▾

unfilteredaiapi.comhttps://api.unfilteredaiapi.com/v1

Unfiltered AI

Raw, uncensored LLM API for developers.

Unfiltered AIGet API key

Conversation memory in a 100k window: budgeting, trimming and summaries

A 100,000-token window sounds enormous until a long roleplay or story fills it. This article treats context as a ledger: you set aside room for the reply, protect a small set of pinned facts, fold older turns into a rolling summary and trim the rest. Every step comes with short Python you can adapt.

Conversation memory in a 100k window: budgeting, trimming and summaries

Key points

  • Dedicated inference providers like Unfiltered AI offer consistent uncensored behavior on a single model, avoiding the randomness of aggregator routing.
  • Aggregators like OpenRouter provide access to multiple models but may vary in how strictly they apply content filters depending on the specific model version.
  • Unfiltered AI uses an open-weight model on dedicated hardware with a 100k context window, priced at $0.15/$0.50 per million tokens.
  • For strict NSFW or roleplay consistency, a dedicated uncensored LLM API eliminates the need for complex prompt engineering to bypass generic filters.

Think of the window as a ledger

The model has no memory between calls. What feels like remembering is you resending text, and every resent token is a line in a ledger with a hard ceiling: 100,000 tokens, shared by the prompt and the reply. Spend more on the prompt and you leave less room for the answer; ask for too much of both and the request is rejected with a 400.

A workable ledger looks like this. Reserve 1,500 tokens for the reply, because that is what you will pass as max_tokens. Reserve another 3,000 as margin, since any local token count is an estimate. That leaves 59,500 tokens of budget for the prompt. Inside that budget, four kinds of content compete: the system prompt, pinned facts, a rolling summary, and the verbatim recent turns. The rest of this article is about deciding who wins.

The priority order is simple to state. System and pinned content are never cut. The summary is replaced rather than cut. Recent turns are cut oldest-first. Anything dropped from the recent turns is not lost, because it gets folded into the summary first.

Counting without a tokenizer

Use the returned usage counts to check your estimates. A characters-per-token shortcut can vary with language, punctuation and code. Leave room for the output and reduce input when estimates approach the combined context limit.

That is why the margin exists, and why it pays to calibrate. After each call, compare the characters you sent with usage.prompt_tokens and nudge your ratio toward the observed value. A small smoothing factor keeps one odd message from throwing it off.

class Calibrator:
    """Learns the real chars-per-token ratio from the usage block."""
    def __init__(self):
        self.ratio = 4.0

    def update(self, sent_messages, prompt_tokens):
        chars = sum(len(m["content"]) for m in sent_messages)
        if prompt_tokens > 0 and chars > 2000:
            observed = chars / prompt_tokens
            self.ratio = 0.8 * self.ratio + 0.2 * observed   # smooth it

    def est(self, text):
        return int(len(text) / self.ratio) + 4

If you swap the calibrator's est into the trimmer, the budget tightens or loosens by itself as the conversation's real character emerges. A story heavy in dialogue and short words tends to cost more tokens per character than expository prose, so the observed ratio is worth watching.

Trimming: newest first, never mid-pair

Walk recent history backwards and add only messages that fit the budget. If even the newest exchange is too large, truncate or reject it deliberately. Keep required context and allowed roles in a valid sequence; selecting recent turns does not guarantee coherent answers.

# memory.py
WINDOW = 64_000      # prompt + completion share this
MAX_OUT = 1_500      # what we ask for in max_tokens
MARGIN = 3_000       # slack for estimation error
BUDGET = WINDOW - MAX_OUT - MARGIN   # 59,500 tokens for the prompt

def est_tokens(text: str) -> int:
    # Rough heuristic: about 4 characters per token for English prose.
    return len(text) // 4 + 4

def build_messages(system, pinned, summary, turns):
    """Return (messages_to_send, dropped_turns). turns is oldest-first."""
    head = [{"role": "system", "content": system}]
    if pinned:
        head.append({"role": "system", "content": "Pinned facts (always true):\n- " + "\n- ".join(pinned)})
    if summary:
        head.append({"role": "system", "content": "Story so far: " + summary})

    used = sum(est_tokens(m["content"]) for m in head)
    kept = []
    for m in reversed(turns):                  # walk from the newest turn backwards
        cost = est_tokens(m["content"])
        if used + cost > BUDGET:
            break
        kept.append(m)
        used += cost
    kept.reverse()
    while kept and kept[0]["role"] != "user":  # never start on an orphaned assistant turn
        kept.pop(0)
    dropped = turns[: len(turns) - len(kept)]
    return head + kept, dropped

The function returns two things: the messages to send and the turns that fell out. Returning the dropped turns, instead of discarding them, is the design decision that makes summaries possible. Nothing is deleted silently.

Pinning facts that must survive

A summary is lossy by nature. Character facts that must never drift, such as a name, a secret, a promise or a rule of the world, should not depend on a summariser choosing to keep them. Store them as a short list outside the history and inject them as a dedicated system message on every call.

  • Pin identity: names, relationships, species, age (adults only), occupation.
  • Pin constraints: things the narrator must not do or reveal yet.
  • Pin state that changes rarely: location, season, who holds what.
  • Do not pin atmosphere. Mood belongs in the summary, where it can evolve.

Keep the pinned block under roughly 400 tokens. If it grows past that, it starts to behave like a second system prompt and the model begins to treat every pinned line as equally urgent. Review the list occasionally and retire facts that stopped being true.

Rolling summaries

When turns fall out of the window, ask the model to fold them into an existing summary. The prompt matters more than the code: tell it what to keep (who is present, what changed, promises, unresolved threads, current place) and what to drop (filler, repeated description), and cap the length in words. A cap keeps the summary from slowly swelling into the very problem it solves.

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.unfilteredaiapi.com/v1", api_key=os.environ["API_KEY"])

SUMMARY_PROMPT = (
    "Compress the material below into a running summary of at most 250 words. "
    "Keep: who is present, what changed, promises and secrets, unresolved threads, "
    "the current location and mood. Drop: filler and repeated description. "
    "Write plain past-tense prose, no bullet points."
)

def fold_into_summary(old_summary, dropped_turns):
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in dropped_turns)
    user = f"Existing summary:\n{old_summary or '(none)'}\n\nNew material:\n{transcript}"
    r = client.chat.completions.create(
        model="uncensored",
        messages=[{"role": "system", "content": SUMMARY_PROMPT}, {"role": "user", "content": user}],
        temperature=0.2,
        max_tokens=450,
    )
    return r.choices[0].message.content.strip()

Use a low temperature. A summary should be faithful, not inventive. A cheap safeguard is to read the summary yourself every twenty or so folds during development; errors compound because each new summary is built on the previous one.

Putting the pieces together

The class below connects trimming, pinning and summarising in one loop. Each call builds the prompt, checks whether anything was dropped, folds it into the summary if so, and rebuilds. The rule is intentionally lazy: it only summarises when the window forces it, so short conversations never pay for a summary call.

from memory import build_messages, fold_into_summary, client, MAX_OUT

class Conversation:
    def __init__(self, system, pinned):
        self.system, self.pinned = system, pinned
        self.summary, self.turns = "", []

    def say(self, text):
        self.turns.append({"role": "user", "content": text})
        messages, dropped = build_messages(self.system, self.pinned, self.summary, self.turns)
        if dropped:                                   # something fell out of the window
            self.summary = fold_into_summary(self.summary, dropped)
            self.turns = self.turns[len(dropped):]
            messages, _ = build_messages(self.system, self.pinned, self.summary, self.turns)
        r = client.chat.completions.create(
            model="uncensored", messages=messages, max_tokens=MAX_OUT, temperature=0.8
        )
        answer = r.choices[0].message.content
        self.turns.append({"role": "assistant", "content": answer})
        return answer

convo = Conversation(
    system="You narrate a slow-burn adult mystery set in a mountain hotel.",
    pinned=["Ines Duarte is the night manager and is lying about the ledger.",
            "It is the third night of the blizzard; the road is closed.",
            "The narrator never reveals the culprit before the player accuses someone."],
)
print(convo.say("I ask Ines why the guest book has a page missing."))

This pairs naturally with the streaming relay in the FastAPI backend article, where the session store would hold summary, pinned and turns instead of one flat list. For structured NPC state on the same pattern, see the game dialogue guide.

The cost of remembering

Long memory is not free, because the whole prompt is billed again on every call. Assume two setups. In the first, the history is allowed to fill 50,000 prompt tokens and each reply is 500 tokens. Input costs 50,000 x $0.15 / 1,000,000 = $0.0075, output costs 500 x $0.50 / 1,000,000 = $0.00025, so each turn costs about $0.00775. In the second, you keep 8,000 tokens of recent turns plus a 600-token summary and 400 pinned tokens, about 9,000 in total. Input is $0.00225, output $0.00025, so each turn is near $0.0028.

Over 1,000 turns that is roughly $13 versus $2.75 under these assumptions. The summary approach also costs an occasional extra call of perhaps 3,000 input and 450 output tokens, about $0.0012, which is negligible beside the savings. Rates are on the pricing page.

The practical advice: do not fill the window just because it is there. Use the large window as insurance for the moments a scene truly needs long verbatim context, and let summaries carry the rest.

Where memory schemes quietly fail

Three failure patterns show up again and again. The first is summary drift: a detail is paraphrased, the paraphrase is summarised again, and after a dozen folds the original fact is unrecognisable. Pinning is the cure for anything you cannot afford to lose, and a hard word cap prevents the summary from absorbing everything.

The second is the cliff. With a pure trim and no summary, the oldest turn simply vanishes and the character suddenly forgets something it knew a moment ago. Users notice. Running the summariser one step before the window forces it, for example when the prompt passes about 80 percent of the budget, smooths the transition.

The third is role confusion after trimming, when history begins mid-exchange. The leading-assistant check in the trimmer exists for this. Add a test that builds a long fake conversation, trims it, and asserts the first kept message is from the user and that the estimated total stays under budget. It takes five minutes and prevents a class of bugs that are hard to spot in manual play.

Questions and answers

What happens if prompt plus max_tokens exceeds 100,000?

The request is not rejected: max_tokens is trimmed to the room left in the window, and the reply stops at the end of the window. Only a prompt longer than 100,000 tokens returns a 400 bad request.

Can I count tokens exactly before a request?

Not exactly in advance. Estimate locally, keep a safety margin, and calibrate against the usage.prompt_tokens value returned with each response.

How often should I summarise?

Only when turns fall out of the window. That keeps short conversations free of extra calls.

Are pinned facts and summaries billed?

Yes. Everything you send as a prompt token is billed at $0.15 per million input tokens on each call, so keep both compact.

Your key is one form away

Create an account, copy the key, change the base URL. That is the whole setup.

Get API keyRead the docs