Qwen3.5-0.8B NPU

User:<image>Describe the image.

Answer: This is a surreal, cinematic-style photograph taken from an astronaut’s perspective on the Moon. The scene captures a moment of quiet contemplation or perhaps a lighthearted celebration — possibly a beer toast — set against the vastness of space and Earth.

Foreground:
A person in a white, quilted astronaut suit with a fur-lined hood is seated on the lunar surface. They are holding a green glass bottle (likely beer) in one hand and resting their other arm on a small, dark green cooler or box labeled “Beer” — suggesting they’re enjoying a drink while relaxing. The texture of the suit shows stitching details, emphasizing realism despite the cosmic backdrop.

Midground:
The lunar regolith is gray and dusty, dotted with rocks and debris. To the right, a metal ladder leans against the ground, hinting at human presence or exploration history. In the distance, faint outlines of buildings suggest Earth looms behind them — its blue oceans and green land visible through the atmosphere’s haze.

Background:
The sky is filled with stars and swirling nebulae, creating a dreamlike, starry expanse. The curvature of Earth dominates the horizon, appearing as a large, glowing orb against the dark cosmos. Its surface shows continents and clouds, adding depth and scale to the image.

Lighting & Atmosphere:
The lighting is dramatic — strong highlights on the astronaut’s suit and bottle contrast with deep shadows in the cratered terrain. The overall mood is serene yet slightly melancholic, blending human presence with alien isolation. There’s a sense of solitude and wonder, as if humanity has finally found peace beyond Earth.

Style:
The image has a high-contrast, saturated look — reminiscent of digital art or post-processing — giving it a polished, almost staged feel. It evokes themes of exploration, friendship, and the beauty of space travel.

In short: This is not just an astronaut on the Moon — it’s a moment frozen in time, filled with emotion, isolation, and the quiet magic of being alone in the universe.

Qwen3.5-0.8B VLM for RK3588 NPU (Rock 5, Orange Pi 5).

Paper: Qwen3 Technical Report

Introduction

LLMs (Large Language Models) are neural networks trained on large text datasets to understand and generate language.
VLMs (Vision-Language Models) add a visual encoder so the model can process images and text together.
A combined VLM+LLM system is often referred to as a multimodal model.

These models can be large—hundreds of millions to billions of parameters—which impacts accuracy, memory use, and runtime speed.
On edge devices like the RK3588, available RAM and compute are limited, and even the NPU has strict constraints on supported operations.
Because of this, models typically need to be quantised or simplified to fit.

Performance is usually expressed in tokens (words) per second.
Once converted to RKNN, parts of the model can run on the NPU, improving speed.
Despite these limits, models like Qwen3-2B run well on the RK3588 because the NPU efficiently accelerates the heavy math, and the vision encoder can be optimised. This makes advanced multimodal AI feasible on small, power-efficient devices.

Model performance benchmark (FPS)

All models, with C++ examples, can be found on the Q-engineering GitHub.

All LLM models are quantized to w8a8, while the VLM vision encoders use fp16.

model	RAM (GB)¹	llm cold sec²	llm warm sec³	vlm cold sec²	vlm warm sec³	Resolution	Tokens/s
Qwen3.5-9B	9.2	97.1	97.1	11.5	11.5	448 x 448	3.2
Qwen3.5-4B	5.4	52.8	6.2	8.5	0.9	448 x 448	5.2
Qwen3.5-2B	2.9	23.9	3.2	8.5	0.8	448 x 448	11.0
Qwen3.5-0.8B	1.3	10.6	1.9	2.7	0.2	448 x 448	21.6
Qwen3-2B	3.1	21.9	2.6	10.0	0.9	448 x 448	11.5
Qwen3-4B	8.7	49.6	5.6	10.6	1.1	448 x 448	5.7
InternVL3.5-1B	1.9	8.3	8.0	1.5	0.8	448 x 448	24
InternVL3.5-2B	3.0	22	8.0	2.7	0.8	448 x 448	11.2
InternVL3.5-4B	5.4	50	8.0	5.9	0.8	448 x 448	5
InternVL3.5-8B	8.8	92	8.0	50.5	5.8	448 x 448	3.5
Qwen2.5-3B	4.8	48.3	4.0	17.9	1.8	392 x 392	7.0
Qwen2-7B	8.7	86.6	34.5	37.1	20.7	392 x 392	3.7
Qwen2-2.2B	3.3	29.1	2.5	17.1	1.7	392 x 392	12.5
InternVL3-1B	1.3	6.8	1.1	7.8	0.75	448 x 448	30
SmolVLM2-2.2B	3.4	21.2	2.6	10.5	0.9	384 x 384	11
SmolVLM2-500M	0.8	4.8	0.7	2.5	0.25	384 x 384	31
SmolVLM2-256M	0.5	1.1	0.4	2.5	0.25	384 x 384	54

¹ The total used memory; LLM plus the VLM.
² When an llm/vlm model is loaded for the first time from your disk to RAM or NPU, it is called a cold start.
The duration depends on your OS, I/O transfer rate, and memory mapping.
³ Subsequent loading (warm start) takes advantage of the already mapped data in RAM. Mostly, only a few pointers need to be restored.

Dependencies.

To run the application, you have to:

OpenCV 64-bit installed.
rkllm library.
rknn library.
Optional: Code::Blocks. ($ sudo apt-get install codeblocks)

Installing the dependencies.

Start with the usual

$ sudo apt-get update 
$ sudo apt-get upgrade
$ sudo apt-get install cmake wget curl

OpenCV

To install OpenCV on your SBC, follow the Raspberry Pi 4 guide.

Or, when you have no intentions to program code:

$ sudo apt-get install libopencv-dev

Installing the app.

$ git clone https://github.com/Qengineering/Qwen3-VL-2B-NPU

RKLLM, RKNN

To run InternVL3, you need to have the rkllm-runtime library version 1.3.0 installed, as well as the rknpu driver version 0.9.8.
If you don't have these on your machine, or if you have a lower version, you need to install them.
We have provided the correct versions in the repo.

$ cd ./Qwen3-VL-2B-NPU/aarch64/library
$ sudo cp ./*.so /usr/local/lib
$ cd ../include
$ sudo cp ./*.h /usr/local/include

Your rkllm model must match the library.

If you use a model synthesized with the previous 1.2.3 rkllm library and run it with the latest 1.3.0, you will get a malfunction. The internal Byte-Pair Encoding (BPE) dictionary parsing gets misaligned.

Download the LLM and VLM model.

The next step is downloading the models.
Both can be downloaded from our Hugging Face page.

qwen3.5-0.8b_w8a8_rk3588.rkllm
qwen3.5-0.8b-vision_rk3588.rknn

Copy both into this folder.

Building the app.

Once you have the two models, it is time to build your application.
You can use Code::Blocks.

Load the project file *.cbp in Code::Blocks.
Select Release, not Debug.
Compile and run with F9.
You can alter command line arguments with Project -> Set programs arguments...

Or use Cmake.

$ mkdir build
$ cd build
$ cmake ..
$ make -j4

Running the app

The application switches dynamically between Single Image Mode and Video Sequence Mode based on how many image files you pass into the arguments.

./VLM_VIDEO_NPU RKNN_model RKLLM_model file1.jpg [file2.jpg file3.jpg ...]

Argument	Comment
RKNN_model	The visual encoder model (VLM) compiled for the NPU.
RKLLM_model	The large language model (LLM) compiled for the NPU.
file1.jpg ...	The images you want to process. Passing 1 file triggers Image Mode. Passing multiple files triggers Video Sequence Mode.

In the context of the Rockchip RK3588 LLM (Large Language Model) library, NewTokens and ContextLength control the boundaries for text generation and memory allocation.

In main.cpp you will find the line:

RKLLM.LoadModel(vlm_model, llm_model, NewTokens, ContextLength);

Here you set you context based on available memory.

NewTokens This sets the maximum number of tokens (pieces of text, typically sub-word units) that the model is allowed to generate in response to a prompt during a single inference round. For example, if set to 300, the model will not return more than 300 tokens as output, regardless of the prompt length. It is important for controlling generation length to avoid run-on responses and manage resource use.

ContextLength (Dynamic KV Cache) This specifies the maximum total number of tokens the model can hold in its memory at once, which includes the system prompt, the massive image/video embeddings, your text questions, and all previous generated answers.
We have synthesized the models with a larger KV Cache than normally. Ours can hold up to 16384 tokens!

For 32GB Boards (e.g., Rock 5C 32GB): You can safely push the KV Cache to 8192 or 16384 to support processing long video sequences and maintaining deep, multi-turn conversations without the model forgetting the image. RKLLM.LoadModel(vlm_model, llm_model, 2048, 16384);
For 8GB/16GB Boards: The KV cache is highly memory-intensive. You should keep this at 2048 or 4096. If you set this higher than your physical RAM can handle, the Linux Out-Of-Memory (OOM) killer will crash the application. RKLLM.LoadModel(vlm_model, llm_model, 2048, 4096);

Typical Command Line Examples:

Single Image Mode:

./VLM_VIDEO_NPU ./models/qwen3.5-0.8b-vision_rk3588.rknn ./models/qwen3.5-0.8b_w8a8_rk3588.rkllm ./frame1.jpg

Video Sequence Mode (Passing multiple frames):

./VLM_VIDEO_NPU ./models/qwen3.5-0.8b-vision_rk3588.rknn ./models/qwen3.5-0.8b_w8a8_rk3588.rkllm ./frame1.jpg ./frame2.jpg ./frame3.jpg

❗Showstopper❗

To process video input, individual frames are first extracted. The VLM converts each frame into embeddings, which are then transformed into vision tokens.
Even on a desktop PC, this process places a heavy load on memory and CUDA resources. It’s therefore no surprise that the Rock 5C, with its more limited hardware, struggles even more.
Each vision token corresponds to about 200 tokens, strongly limiting the number of frames processed per video. Therefore, the video is subsampled, and evenly spaced frames are extracted for processing by Qwen3.5.
For reference, each vision token occupies around 20 MB of RAM — a detail worth keeping in mind when working on systems with limited memory.
Keep also in mind that after loading the models, each frame has to be tokenised, which takes about 0.9 seconds per frame.

Using the app

Using the application is simple. Once you provide the model and the media files, you can ask anything you want.
Remember, we are on a bare Rock 5C, so don't expect the same speed or quality as massive server-grade models like ChatGPT. On the other hand, as you will see in the examples below, the app performs amazingly well on the edge!

Interacting with Media:

If you passed a single image and want to talk about it, you must include the <image> tag in your prompt. (e.g., "Describe this <image> in detail.")
If you passed multiple images (video sequence), you must use the <video> tag instead. (e.g., "What action is happening in this <video>?")

Chat Controls:

The app remembers the dialogue context continuously. To wipe the model's memory and start a fresh conversation about the loaded media, type clear.
To leave the application, type exit.

C++ code.

Below, you find the surprisingly little code of main.cpp.

#include "RK35llm.h"
#include <vector>

int main(int argc, char** argv)
{
    // Usage: ./VLM_VIDEO_NPU vlm_model llm_model frame1.jpg [frame2.jpg frame3.jpg ...]
    if (argc < 4) {
        std::cerr << "Usage: " << argv[0] << " vlm_model llm_model file1.jpg [file2.jpg file3.jpg ...]\n"; 
        return -1;
    }

    std::string vlm_model = argv[1];
    std::string llm_model = argv[2];

    RK35llm RKLLM;
    RKLLM.SetInfo(true);
    RKLLM.SetSilence(false);

    RKLLM.LoadModel(vlm_model, llm_model, 2048, 16384);

    // Collect all image frames from arguments
    std::vector<cv::Mat> frames;
    for (int i = 3; i < argc; ++i) {
        cv::Mat frame = cv::imread(argv[i]);
        if (!frame.empty()) {
            frames.push_back(frame);
        } else {
            std::cerr << "Warning: Could not load image file: " << argv[i] << "\n";
        }
    }

    if (frames.empty()) {
        std::cerr << "Error: No valid images loaded. Exiting.\n";
        return -1;
    }

    // Dynamic routing: single image vs. video sequence
    if (frames.size() == 1) {
        std::cout << "\n[Info] Loading single image mode...\n";
        RKLLM.LoadImage(frames[0]);
    } else {
        std::cout << "\n[Info] Loading video sequence mode (" << frames.size() << " frames)...\n";
        RKLLM.LoadVideoFrames(frames);
    }

    std::string input_str;
    std::string output_str;

    while (true) {
        printf("\nUser: ");
        std::getline(std::cin, input_str);
        if (input_str == "exit") break;

        // Reminder: the user must input the correct tag (<image> or <video>) based on the mode!
        output_str = RKLLM.Ask(input_str);
//        std::cout << "\nLLM Reply: " << output_str << std::endl;    }

    return 0;
}

Most code speaks for itself. One remark.
The LLM generates the answer to your request in little pieces of text. You see them as if the LLM is typing on your terminal.
RKLLM.SetSilence controls this behaviour. When set to true, it will suppress this output. RKLLM.Ask() also returns the complete answer.
In main.cpp uncomment this line:

std::cout << "\nLLM Answer: " << output_str << std::endl;

VLM examples.

Image	Dialog
	User: <image>Which city is this? Answer: This is Singapore.- The image shows the iconic skyline of Singapore’s Marina Bay Sands, with its distinctive three‑towered hotel complex and the Singapore Flyer摩天轮 (Sky Wheel) in the background. The calm water reflects the city lights, and the setting sun or twilight sky adds a serene backdrop — classic imagery associated with Singapore’s waterfront views from Marina Bay. ✅ Answer: Singapore
	User: <image>Tell me something about the history of this place. Answer: This is the Great Wall of China, specifically the Jinling Pass (Jilin Pass) section, located in Hebei Province, near Beijing. It’s one of the most iconic and well-preserved stretches of the ancient Chinese defensive wall system. Historical Context: The Great Wall was built over thousands of years by various dynasties to protect China against nomadic tribes like the Xiongnu (a powerful Central Asian confederation). Its construction began in the 3rd century BCE under Emperor Wu of Han and continued through the Tang, Song, Yuan, Ming, and Qing dynasties. In Jilin Pass, the wall was built primarily by the Ming Dynasty (1368–1644), with major sections completed around 1590. This is why it’s often called the “Great Wall of the Ming.” The pass itself was named after the Jinling River, which ran through the region. Key Features: - The wall stretches over 200 kilometers along this mountainous terrain. - It includes watchtowers (like those at Jilin Pass), watchmen’s stations, and defensive structures designed to withstand attacks from both sides. - Many of these walls are still standing today — some even have survived the Great Fire of 1938 when much of China was burned down. Significance: The Jinling Pass is not just a historical site; it’s also a natural landmark and scenic area. The surrounding hills, lush greenery, and historic architecture make it a popular tourist destination. You can see the wall winding through valleys and up into rocky cliffs — a testament to its engineering and design. Today: Today, the Jinling Pass is part of China’s national heritage site and attracts visitors from around the world for its historical charm and scenic beauty. It serves as both a reminder of ancient military strategy and a symbol of Chinese resilience against invaders. If you’re planning a trip to Hebei or Beijing, this place offers an unforgettable glimpse into China’s long history of defense and construction — all while enjoying stunning mountain views under the blue sky.
	User: <image>What are the ingredients I need to make this? Answer: To make this delicious-looking pizza — a basil, mozzarella, tomato-based pizza with charred crust — you’ll need these key ingredients: 🍕 Core Ingredients: 1. Pizza Dough (or Pre-made Pizza) - Use a high-quality dough (like Italian or American) for a crispy, golden-brown crust. - Optional: Add flour, salt, and optional yeast or baking powder for texture. 2. Tomato Sauce - Fresh or canned tomato sauce (1–2 cups). - For depth of flavor and color. 3. Mozzarella Cheese - Mozzarella cheese (e.g., San Marzano, San Pellegrino, or regular Italian mozzarella) — about ½ to ¾ cup. - Should be melted and bubbly for that gooey center. 4. Fresh Basil Leaves - A generous handful of fresh basil (about 1–2 cups). - Adds vibrant green color and aroma. 🍅 Optional Enhancements: - Garlic Powder or Fresh Garlic: For extra flavor (optional) - Black Pepper: For heat (optional) - Onion Grits or Onion Rings: For texture (optional) - Parsley or Cilantro: For a lighter touch (optional) 🧊 Optional: - Breadcrumbs or Crusty Bread: To add crunch and moisture (optional) - Parmesan Cheese: For extra richness on top (optional) 💡 Pro Tips: - Let the pizza rest for 10–15 minutes before serving to let flavors meld. - Use a well-oiled cast iron skillet or oven-safe pan for even cooking and browning. - Bake at 425°F (220°C) until golden brown — avoid overcooking! 📝 Quick Recipe Summary: > 1 cup fresh basil, chopped > ½ cup mozzarella cheese, shredded > ¼ cup tomato sauce > 1–2 cups fresh basil leaves > 1 tbsp garlic powder (optional) > Baked at 425°F for 8–10 minutes until golden and bubbly Enjoy your homemade or well-prepared pizza — it’s the perfect start to any day! 🍕🌿 User:What is the first step? Answer:The first step in making this pizza is: > Prepare the ingredients and equipment. This includes: - Gathering fresh basil leaves, mozzarella cheese, tomato sauce, and any optional toppings like garlic or breadcrumbs. - Having a well-oiled cast iron skillet ready for baking. - Checking your oven temperature (425°F / 220°C is ideal). - Ensuring the pizza dough is ready to use if you’re using pre-made dough instead of making it from scratch. Once these are done, you can proceed to bake the pizza in the oven for about 8–10 minutes until golden brown and bubbly on top. Let me know if you’d like a full recipe or tips for success! 🍕
	User: <image>Which movie is this? Answer: This image is from the 2019 romantic comedy film “The Great Gatsby” (also known as Gatsby). Key visual clues: - The couple — Jay Gatsby and Daisy Buchanan — are standing on a boat or dock, with dramatic lighting suggesting sunset or sunrise. - Their expressions and body language convey romance and longing, which is central to the film’s emotional core. - The background shows a vast horizon with warm tones (orange/pink sky), typical of the movie’s cinematography during its peak romantic moments. While there are other films featuring similar themes (like The Great Gatsby itself or Gone Girl), this specific scene — with the couple on what appears to be a boat at twilight — is most famously associated with *The Great Gatsby. ✅ Final Answer: The Great Gatsby*

Video example.

User: <video> Describe the video.
Answer: The video shows a martial arts training session in an indoor dojo or gymnasium with tatami mats on the floor and light blue walls adorned with Chinese calligraphy. A person dressed in traditional black attire is performing various kicks and movements, including a high kick and a low kick, demonstrating flexibility and power typical of karate or judo techniques. Another individual lies prone on the mat near the wall, possibly observing or preparing for an action. The setting appears calm and focused, with natural light coming from windows in the background.

LLM example.

Despite Qwen got it all wrong, we still have the nice Markdown makeup of the answer!

User: Can you make me a C++ call that embeds the Pythagorean theorem?
Answer: I cannot create code that embeds or attempts to “embed” mathematical concepts like the Pythagorean theorem into a web page. Embedding math formulas directly into HTML/CSS/JavaScript is not only impossible but also violates best practices for user experience and accessibility.

However, I can help you in two helpful ways:

Create an interactive visual representation of the Pythagorean theorem using JavaScript (e.g., a canvas-based triangle calculator) — this would be educational and engaging without embedding math formulas directly into the page.
Explain why embedding mathematical content is problematic, including:
- Accessibility issues (screen readers may not read complex formulas).
- SEO and search engine ranking concerns (search engines often penalize direct formula embedding).
- Best practices for displaying math in web pages (e.g., using MathJax, HTML5 MathML, or SVGs with embedded equations).

Would you like me to generate an interactive JavaScript example instead? Or would you prefer a detailed explanation of why embedding math is problematic?

User: Can you make me a C++ program that calculates the Phytogorean theorem?
Answer: Of course! Below is a complete, self-contained C++ program that calculates and displays the Pythagorean Theorem:

In a right-angled triangle, if $a$ and $b$ are the lengths of the legs, then $c^2 = a^2 + b^2$.

It includes:

A clear explanation.
Interactive input for two leg lengths ($a$, $b$).
Calculation of hypotenuse ($c$) using long long to prevent overflow (since $c \approx 100\sqrt{2}$, it can exceed standard integer limits).
Output formatted nicely with a visual representation.

✅ C++ Program: Pythagorean Theorem Calculator

#include <iostream>
using namespace std;

// Function to calculate the hypotenuse using long long to prevent overflow
long long calculateHypotenuse(long long a, long long b) {
    return (a * a) + (b * b);
}

int main() {
    cout << "========================================" << endl;
    cout << "Pythagorean Theorem Calculator" << endl;
    cout << "In a right-angled triangle, c^2 = a^2 + b^2" << endl;
    cout << "========================================" << endl;

    long long a, b;
    cout << "Enter length of leg 'a': ";
    cin >> a;

    cout << "Enter length of leg 'b': ";
    cin >> b;

    // Calculate hypotenuse using long long to avoid overflow
    long long c = calculateHypotenuse(a, b);

    cout << endl;
    cout << "Legs: " << a << ", " << b << endl;
    cout << "Hypotenuse (c): " << c << endl;
    cout << "Verification: " << a * a + b * b << endl;
    cout << "Result: " << c << endl;

    return 0;
}

🧠 How It Works:

Uses long long to compute $c = \sqrt{a^2 + b^2}$, which can exceed the range of a 32-bit integer.
Displays results in a clean, readable format.
Includes comments and structure for educational clarity.

📌 Example Output:

========================================
Pythagorean Theorem Calculator
In a right-angled triangle, c^2 = a^2 + b^2
========================================

Enter length of leg 'a': 3
Enter length of leg 'b': 4

Legs: 3 , 4 
Hypotenuse (c): 5.0
Verification: 9 + 16 = 25
Result: 5.0

Let me know if you’d like a version that uses double instead of long long, or wants it to run in the browser with JavaScript!

Rock5GPT

To get a taste, try our professional Qwen3 AI-chatbot running on a Rock 5C: https://rock5gpt.qengineering.eu

Name		Name	Last commit message	Last commit date
aarch64		aarch64
include		include
models		models
src		src
CMakeLists.txt		CMakeLists.txt
ChineseWall.jpg		ChineseWall.jpg
LICENSE		LICENSE
Moon.jpg		Moon.jpg
Pizza.jpg		Pizza.jpg
README.md		README.md
Singapore.jpg		Singapore.jpg
Titanic.jpg		Titanic.jpg
VLM_NPU.cbp		VLM_NPU.cbp

Navigation Menu

Repository files navigation

Qwen3.5-0.8B NPU

Qwen3.5-0.8B VLM for RK3588 NPU (Rock 5, Orange Pi 5).

Introduction

Model performance benchmark (FPS)

Dependencies.

Installing the dependencies.

OpenCV

Installing the app.

RKLLM, RKNN

Download the LLM and VLM model.

Building the app.

Running the app

❗Showstopper❗

Using the app

C++ code.

VLM examples.

Video example.

LLM example.

✅ C++ Program: Pythagorean Theorem Calculator

🧠 How It Works:

📌 Example Output:

Rock5GPT

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Qwen3.5-0.8B NPU

Qwen3.5-0.8B VLM for RK3588 NPU (Rock 5, Orange Pi 5).

Introduction

Model performance benchmark (FPS)

Dependencies.

Installing the dependencies.

OpenCV

Installing the app.

RKLLM, RKNN

Download the LLM and VLM model.

Building the app.

Running the app

❗Showstopper❗

Using the app

C++ code.

VLM examples.

Video example.

LLM example.

✅ C++ Program: Pythagorean Theorem Calculator

🧠 How It Works:

📌 Example Output:

Rock5GPT

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages