EN ▾
Get API key

Best Speculative Decoding for Uncensored LLM Use

Speculative decoding is a server-side speed technique. This article explains how it works, which variants fit an uncensored model, and what you can do from the client side of an API.

Updated

A small fast element drafting ahead while a larger block verifies each step, illustrating speculative decoding

Cheap AI APIs

Key points

  1. Speculative decoding drafts several tokens cheaply, then checks them in one pass of the main model.
  2. With correct sampling, the output distribution matches ordinary decoding; only the speed changes.
  3. On a hosted API the setup is the provider's choice, not a request parameter.
  4. This site does not publish speed figures; measure on your own prompts with the script below.

Short answer

If you are asking about the best speculative decoding for uncensored llm deployments, the answer depends on who holds the server. When you run the model yourself, a draft model or a head trained for that exact model usually wins; when you call a hosted API, you cannot choose at all, because speculative decoding is a property of the inference server and not a field in the request. No parameter of POST /v1/chat/completions switches it on.

This API does not state which decoding method sits behind it, and this article will not guess. What follows is the general technique, how to pick among methods if you self-host, and what you can measure and tune from the outside.

How speculative decoding works

A language model normally writes one token per forward pass, and each pass has to read the model weights from memory. That makes generation limited by memory bandwidth rather than arithmetic. Speculative decoding exploits the idle arithmetic.

  1. A cheap draft mechanism proposes the next few tokens, say four.
  2. The full model scores all four positions in a single forward pass.
  3. It accepts the longest prefix that matches what it would have produced, and corrects the first mismatch.
  4. Generation continues from there.

With the published rejection-sampling scheme, the accepted tokens follow the same distribution as plain sampling from the main model. So quality does not change by design. You get several tokens for the price of about one pass when the draft guesses well, and roughly one token per pass when it does not. The speed-up is the average number of accepted tokens, which is called the acceptance rate in the literature.

Methods and when each fits

MethodHow it draftsGood fit
Small draft modelA smaller model from the same family proposes tokensGeneral chat, if a close small sibling exists
Prompt lookup / n-gramCopies spans that already appear in the promptEditing, summarising, code rewriting, retrieval answers
Extra decoding heads (Medusa-style)Added heads predict several future tokensSelf-hosting with training budget
Feature-level drafting (EAGLE-style)A light module predicts from the model's own hidden statesSelf-hosting where a trained module exists

Popular inference engines, including vLLM and llama.cpp, support speculative decoding in some form, though options and maturity differ by version, so check the documentation of the release you run.

No method is best in the abstract. Prompt lookup costs nothing to set up and shines when the output repeats the input. A draft model helps most when it agrees with the target on style. Trained heads give steady gains on the model they were trained for.

Why acceptance rate decides the speed-up

A simple model shows the effect. If the drafter proposes k tokens and each one is accepted with probability p, independently, the main model produces on average (1 - p^(k+1)) / (1 - p) tokens per pass. This ignores the time spent drafting, so real gains are smaller. With k = 4:

Acceptance pTokens per passReading
0.83.36Draft agrees well; large gain
0.51.94Modest gain, easily eaten by draft cost
0.31.43Little gain; may be slower overall

The numbers explain why matching the drafter to the target matters more than picking a famous method. A mediocre drafter at p = 0.3 barely beats plain decoding, while a well-matched one at p = 0.8 produces more than three tokens per pass.

What changes for an uncensored model

The method itself is indifferent to content policy. The practical catch is draft agreement. Many small models are released with refusal behaviour baked in. An uncensored target model writes things such a draft would never propose, or would hedge. When the draft keeps predicting a disclaimer and the target keeps writing the scene, the acceptance rate falls, and the extra drafting work can cost more time than it saves.

That gives a simple rule for self-hosters. Use a draft that was tuned in the same direction as the target, or a method that needs no separate model: prompt lookup, or heads and modules trained on the target itself. Verify with a trial run on your own prompts, because creative writing at high temperature is the hardest case for any drafter. Randomness lowers agreement, and fiction tends to run hot.

For the API on this site, the model is one open-weight model run by an independent provider, tuned to answer without content refusals for lawful adult use. Sexual content involving minors is not allowed under any setting.

What you can control as an API user

You cannot toggle the drafter, but several request settings affect how fast a reply feels.

  • Stream the response. With stream: true the first words arrive while the rest is still being generated. Perceived speed improves regardless of server tricks. Remember a stream holds its concurrency slot until it ends; a full key allows 8 at once.
  • Limit output. max_tokens is optional, but a cap bounds the worst-case wait and cost.
  • Lower temperature. If your server does use speculation, low-temperature output is easier to predict, so it tends to go faster. Treat that as something to test, not a promise.
  • Keep prompts short. Less input to process before the first token appears.
  • Parallelise within limits. Up to 8 concurrent requests and 300 per minute on a full key improve throughput of a batch, even if one reply is not faster.

Speed does not change what you pay. Billing is $0.15 per 1M input tokens and $0.50 per 1M output tokens either way, and failed requests are not charged. See the pricing page for the details.

Measure it yourself

Do not trust anyone's speed number, including a provider's, until you reproduce it with your own prompts. This script times a non-streamed request and divides completion tokens by wall time. It runs the same prompt at two temperatures with a fixed seed.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.cheapaiapis.com/v1", api_key="YOUR_KEY")

def run(prompt, temperature):
    t0 = time.time()
    r = client.chat.completions.create(
        model="uncensored",
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
        max_tokens=400,
        seed=7,
    )
    dt = time.time() - t0
    n = r.usage.completion_tokens
    return n, dt, n / dt

for temp in (0.0, 0.8):
    n, dt, tps = run("Write a short scene set in a lighthouse.", temp)
    print(f"temp={temp} tokens={n} seconds={dt:.1f} tokens/s={tps:.1f}")

Run it several times at different hours and take the median. The figure includes network time and queueing, so it is an end-to-end number, which is what your users experience. If you want time to first token, switch to stream=True and record when the first chunk arrives. The quickstart shows the streaming call.

You can use the trial credit for this. One click on the API key page gives $0.50, valid for 7 days, no card. The script above uses at most 400 output tokens per run, so ten runs of 400 output and 20 input tokens cost 10 x ($0.000003 + $0.0002) = $0.00203, a fifth of a cent.

If you plan to self-host

Start without speculation and record a baseline. Then add prompt lookup if your workload copies text from the prompt. Next try a draft model that shares the target's tokenizer and tuning direction. Measure tokens per second and the share of drafted tokens accepted, at your real temperature. Stop when the gain stops being worth the extra memory, since a draft model also needs GPU space.

If self-hosting is more work than you want, a hosted endpoint with per-token billing removes the question. The article on cheap API myths and facts covers what to check before trusting any low price.

Questions and answers

Read the docs
Does speculative decoding change the quality of answers?

Not when implemented with the standard acceptance rule, because accepted tokens follow the main model's distribution. It changes speed only.

Can I enable speculative decoding with a request parameter?

No. It is configured on the inference server. Nothing in the chat completions request turns it on for this API.

Is it faster at temperature 0?

Often, since predictions are easier to match, but it varies by method and prompt. Time your own workload.

Which method is best for roleplay and fiction?

Creative text at high temperature is hard to draft. Prompt lookup helps little there, so a draft or heads trained on the same model is the better bet if you self-host.

Your key is one form away

Create an account, copy the key, change the base URL. That is the whole setup.

Get API key

https://api.cheapaiapis.com/v1