Skip to content

LiquidAI LFM3 text generation with the Python SDK

LiquidAI LFM3 text generation with the Python SDK.

Run with python examples/python/lfm3.py. The architecture is detected automatically; Q4K quantization reduces weight memory.

"""
LiquidAI LFM3 text generation with the Python SDK.
Run with `python examples/python/lfm3.py`.
The architecture is detected automatically; Q4K quantization reduces weight memory.
"""
from mistralrs import ChatCompletionRequest, Runner, Which
runner = Runner(
which=Which.Plain(model_id="LiquidAI/LFM3-1.7B"),
in_situ_quant="q4k",
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[
{
"role": "user",
"content": "Explain what graphene is in two short paragraphs.",
}
],
max_tokens=256,
)
)
print(res.choices[0].message.content)
print(res.usage)

Source: examples/python/lfm3.py