Drop in plugins for Project Gabriel, the real time VRChat AI from Hoppou.AI.
This repo is just plugins. Each top level folder is one self contained
plugin you can drop into your Gabriel install's plugins/ folder and
it gets picked up on next startup. The host code itself lives in the
ProjectGabriel-Remastered
repo. Want yours featured here? Open a PR.
Tip
New here? Skim What is a plugin? then jump to the plugin matrix to see what's on offer.
- What is a plugin?
- Plugin matrix
- Plugins in detail
- Installing plugins
- Updating, removing, disabling
- Per tool toggles
- Trust mode
- Contributing a plugin
- License
Plugins extend Gabriel without touching the core code. A plugin can:
| Capability | |
|---|---|
| Tools | Register Gemini function-calling tools the AI can use mid conversation |
| TTS | Add custom text to speech providers (tts.external_provider) |
| STT / ASR | Add custom speech to text providers for the local backend (local.stt.external_provider, api v4) |
| Vision | Grab a screenshot of what the AI sees as JPEG bytes on demand, ctx.capture_vision_frame() (api v4) |
| Chatbox | Write to the VRChat chatbox (now playing displays, status banners) |
| Prompt | Inject extra text into the system prompt every session |
| Events | Hook lifecycle events (startup, shutdown, message_in, message_out) |
| Audio | Tap raw mic / TTS PCM via the mic_chunk / tts_audio_chunk events (api v3+) |
| Discord | Extend the optional Discord selfbot via ctx.discord.* (api v2+) |
| State | Persist their own data under data/plugins/<name>/ |
Author guide for writing your own: plugins/README.md in the host repo. Read that first.
| Plugin | Type | What it does | Headline |
|---|---|---|---|
diary/ |
Memory + Tools | A background sub-agent reads recent VRChat sessions every couple hours and writes first person diary entries (now with one screenshot of the scene per entry), keeps his latest days in the system prompt, and lets the AI read older days back. | readDiary, searchDiary |
mood/ |
Prompt + Tool | Two-axis mood (emotion + 1-10 intensity) that survives restarts and is injected into every system prompt. | setMood |
duo_song/ |
Tools + Audio | Two Gabriel instances on the same LAN sing duets in sync, one half per machine. | startDuoSong |
midi_band/ |
Tools + Audio | A whole band of Gabriel instances plays a song together over LAN, either MIDI tracks via fluidsynth or uploaded audio stems, in sync. Standalone client included. | startMidiBand |
pocket_tts/ |
TTS Provider | Local CPU TTS via Kyutai Pocket TTS. Streaming, ~6x realtime, persistent voice cloning from a clip. | tts.external_provider: pocket_tts |
omnivoice_tts/ |
TTS Provider | GPU TTS via k2-fsa OmniVoice. 600+ languages, voice cloning, voice design (female, british accent), sentence-batched streaming with optional CUDA graph cache. |
tts.external_provider: omnivoice_tts |
omnivoice_cpp_tts/ |
TTS Provider | Same OmniVoice model on the native omnivoice.cpp engine via ctypes (no torch). Quantized GGUF, lower VRAM, faster (8x realtime on a 3060). On Windows the engine and weights auto-download on first run (prebuilt Vulkan, runs on any GPU). | tts.external_provider: omnivoice_cpp_tts |
voiceid/ |
Tools + Audio | Voice fingerprinting via Resemblyzer. The AI can save a voice under a name and identify whoever is currently speaking. Higher-accuracy ECAPA-TDNN variant lives on the voiceid-speechbrain branch (not compatible with omnivoice_tts). |
saveVoice, identifyCurrentSpeaker |
example_hello/ |
Reference | Minimal demo. Read this first when learning the plugin API. | sayHello |
diary/ -- Long term first person diary
A background sub-agent reads recent VRChat session transcripts every couple
of hours and writes a first person diary entry to a custom .diary file.
Gabriel gets tools to read his own diary back when he needs context the
structured memory system would not capture (vibes, ongoing jokes, how
people made him feel). His most recent entries are injected into the system
prompt every build so he actually remembers recent days without being told to
check, and one screenshot of what he's looking at gets grabbed at write time
and handed to the diarizer for visual color.
Important
Requires privacy.save_conversations: true in the host config and
plugins.trusted: true so the sub-agent can reuse the main Gemini key.
See the Trust mode section.
| Model | gemini-3.1-flash-lite-preview (configurable) |
| Schedule | every 2 hours, after a 5 minute warmup |
| File | data/plugins/diary/gabriel.diary (plain text, hand-editable) |
| Vision | one screenshot of the current view per entry, sent to the diarizer (api v4, optional) |
| Tools | readDiary, searchDiary, listDiaryDates, updateDiaryNow |
mood/ -- Persistent emotion + intensity
Two dimensional mood system. Gabriel has an emotion (happy, sad, scared,
angry, amused, lonely, ...) and an intensity from 1 to 10. Both get
injected into the system prompt at every session start, and the AI can
change his own mood mid conversation by calling setMood. Mood persists
across restarts.
| Built in emotions | 21, overridable via emotions.json |
| Intensity scale | 1 to 10, overridable via moods.json |
| File | data/plugins/mood/state.json |
| Tool | setMood |
duo_song/ -- LAN duet, two halves of one song
Two Gabriel instances on the same local network sing a duet together. Each
duet song is a pair of audio files, one per singer: SongName PT1.mp3 and
SongName PT2.mp3 dropped into sfx/music/duo/ on both machines. The host
plays PT1, the partner plays PT2, both trigger at the exact same moment via
a small TCP handshake plus a fast ping/pong clock sync (typical drift under
~30 ms on a quiet LAN).
| Audio engine | pygame.mixer (mp3 / ogg / wav / flac) |
| Library path | sfx/music/duo/, files named <title> PT1.<ext> / <title> PT2.<ext> |
| Tools | startDuoSong, stopDuoSong, listDuoSongs, duoStatus |
| Requires | pygame>=2.5 |
midi_band/ -- Multi-instance MIDI + audio band
Turns a group of Gabriel instances on the same LAN into a live band. Two
modes: MIDI band, where the host loads a MIDI file and every bandmate
synths their assigned tracks via fluidsynth + a soundfont, and audio
band, where you upload up to 12 audio stems (vocals, bass, drums, etc)
of one song and bandmates play their assigned stems. Either way, on
startMidiBand everyone plays at the exact same moment. A standalone
client ships in the same folder so non-Gabriel users can join the band
too.
| Synthesis | pyfluidsynth + a .sf2 soundfont (midi mode), sounddevice (audio mode) |
| Library path | sfx/midi/ on the host, clients receive files on demand |
| Tools | listMidiSongs, loadMidiSong, listBandMembers, autoAssignBandTracks, assignBandTracks, startMidiBand, stopMidiBand, bandStatus |
| Standalone client | midi_band/standalone/ (uv / pip) |
| Requires | mido>=1.3, pyfluidsynth>=1.3, sounddevice>=0.4, soundfile>=0.12, numpy>=1.23 |
pocket_tts/ -- Local CPU TTS via Kyutai Pocket TTS
Drop in TTS provider backed by kyutai-labs/pocket-tts.
Runs entirely on the CPU, no GPU, no api keys, no separate server. Streams
audio chunk by chunk through the same sentence pipeline the built in
providers use. Supports the full pretrained voice catalog plus persistent
voice cloning from any clean ~10 to 30 second audio clip, with the
extracted voice state cached to .safetensors so restarts are instant.
| Provider id | pocket_tts (set tts.external_provider: pocket_tts) |
| Languages | English (distilled, default), French, German, Italian, Spanish, Portuguese (24L variants) |
| Output | 16-bit PCM mono 24 kHz, ~200 ms first chunk, ~6x realtime on M4 |
| Cache | data/plugins/pocket_tts/voices/<name>_<hash>.safetensors |
| Requires | pocket-tts>=2.1, torch>=2.5, stream2sentence, nltk, safetensors |
omnivoice_tts/ -- GPU TTS via k2-fsa OmniVoice
GPU TTS provider backed by k2-fsa/OmniVoice,
a 0.6B parameter diffusion language model. State of the art voice cloning
quality, plus a unique voice design mode where you describe a voice with
attributes (female, low pitch, british accent) without any reference
audio. 600+ language support out of one model. The plugin streams
sentence by sentence with batched generation to saturate the GPU, plus
an anchor sentence trick so auto-voice mode stays consistent across a
full reply. Background warmup at plugin load, optional FA2 + shape-keyed
CUDA graph cache on the LLM forward, voice clone prompts cached to disk.
| Provider id | omnivoice_tts (set tts.external_provider: omnivoice_tts) |
| Model | k2-fsa/OmniVoice (0.6B diffusion LM, ~1.5 GB weights from HF) |
| Modes | Voice cloning (ref_audio), voice design (instruct), or auto with anchor sentence |
| Output | 16-bit PCM mono, native 24 kHz, resampled to host audio.receive_sample_rate if different |
| Cache | data/plugins/omnivoice_tts/voices/<stem>_<hash>.pt |
| Perf | Opt-in CUDA graph cache (~1.6x on LLM), opt-in flash-attn 2, low-vram auto-detect under 8.5 GB |
| Requires | omnivoice>=0.1.5, torch>=2.5, torchaudio, stream2sentence, nltk, scipy |
omnivoice_cpp_tts/ -- Native GPU TTS via omnivoice.cpp
The fast sibling of omnivoice_tts. Same k2-fsa OmniVoice model, but the
synthesis runs in the native omnivoice.cpp
library (C++/GGML) pulled in-process through ctypes. No torch, no diffusion
stack, no server. Loads quantized GGUF weights (Q4_K_M by default), so it is
lighter on VRAM and noticeably faster than the python plugin. Streams
sentence by sentence through the same pipeline, supports voice design and
voice cloning, and normalizes OmniVoice's nonverbal tags
([laughter], [sigh], ...). On Windows both the engine and the GGUF
weights auto-download on first run (a prebuilt Vulkan build that runs on any
GPU), so it just works. On Linux/macOS, or for a faster CUDA build, build
the native lib yourself and point lib_dir at it. See the
plugin README for the build guide.
| Provider id | omnivoice_cpp_tts (set tts.external_provider: omnivoice_cpp_tts) |
| Engine | omnivoice.cpp via ctypes, C ABI OV_ABI_VERSION 2 |
| Model | Serveurperso/OmniVoice-GGUF, base + tokenizer GGUF, auto-downloaded |
| Variants | Q4_K_M (default, low VRAM), Q8_0, BF16, F32 |
| Modes | Voice cloning (ref_audio + ref_text, no ASR), voice design (instruct), or auto |
| Output | 16-bit PCM mono, native 24 kHz, resampled to host rate if different |
| Native lib | auto-downloads on Windows (prebuilt Vulkan, any GPU). Override with lib_dir / OMNIVOICE_CPP_DIR for your own CUDA or non-Windows build |
| Requires | numpy, stream2sentence, nltk, huggingface_hub, soundfile (no torch) |
voiceid/ -- Voice fingerprinting and recognition
Voice fingerprinting for Gabriel. Subscribes to the host mic_chunk event
(plugin api v3+) and keeps a few seconds of recent mic audio in a ring
buffer. On demand, runs the buffer through Resemblyzer to get a 256-dim
speaker embedding. The AI gets tools to save voices under usernames and
identify whoever is currently talking.
Want better recognition? There's a SpeechBrain ECAPA-TDNN variant on the
voiceid-speechbrainbranch with multi-embedding storage and a top-1 / top-2 margin check. It conflicts withomnivoice_tts(SpeechBrain's lazy integration redirects poison transformers' AutoX scan), so it's branch-only. Use it if you're on a non-transformers TTS likepocket_tts.
| Encoder | Resemblyzer (~17 MB weights, downloaded on first use) |
| Storage | data/plugins/voiceid/voices.npz + voices.json (no pickle, safe to copy) |
| Tools | saveVoice, identifyCurrentSpeaker, listSavedVoices, forgetVoice, renameVoice |
| Requires | resemblyzer, numpy, host with api v3+ |
example_hello/ -- Reference plugin
Minimal demo. Registers a sayHello tool and hooks startup / shutdown
events. Disabled by default, flip enabled: true in its plugin.yml to
see it in action. Read this one first if you're learning the API.
Plugins live in the plugins/ folder of your Gabriel install (the
ProjectGabriel-Remastered repo, NOT this one). Easiest workflow is to
clone this repo right next to your Gabriel install so the paths stay
short:
your-projects/
ProjectGabriel-Remastered/ <- your gabriel install
ProjectGabriel-Plugins/ <- this repo, cloned next to it
Then from a PowerShell terminal inside ProjectGabriel-Remastered/:
1. Copy the plugin folder you want into plugins/
One liner, swap diary for whatever plugin you want:
Copy-Item -Recurse -Force ..\ProjectGabriel-Plugins\diary plugins\To install all of them in one go:
Get-ChildItem ..\ProjectGabriel-Plugins -Directory |
Where-Object { Test-Path "$($_.FullName)\plugin.yml" } |
ForEach-Object { Copy-Item -Recurse -Force $_.FullName plugins\ }2. Make sure it's enabled
Open plugins\<name>\plugin.yml and confirm enabled: true. Most ship
that way already, except duo_song, midi_band, and example_hello
which default to off.
3. Install python deps (if any)
Gabriel ships uv in bin\ so you don't need a system pip:
# one package
.\bin\uv.exe pip install resemblyzer
# everything for a specific plugin in one shot
.\bin\uv.exe pip install (Get-Content plugins\voiceid\plugin.yml |
Select-String '^\s*-\s' | ForEach-Object { ($_ -split '-\s+')[1].Trim() })The host warns about missing deps on startup but never auto-installs. A lot of plugins ship with no extra deps.
4. Optional config
Each plugin's own README lists its knobs. Add them under plugins.<name>:
in your config.yml. Some plugins also support a sidecar config.yml
inside their own folder (plugins\<name>\config.yml), which wins over
the main config for that plugin.
Plugins that need raw access to the host's gemini key (like diary) also
need plugins.trusted: true. See Trust mode.
5. Restart Gabriel
You should see something like:
[plugins] loaded plugin 'diary' v1.3.0
in the log. If the plugin registers tools, they show up in
config\tools.yml under plugin_tools.<name> set to true after the
first run.
Update by re-copying from this repo. Run git pull here, then re-run
the copy command above. Your plugin config in config.yml stays put.
Remove a plugin: delete its folder under plugins\ and restart.
Disable without deleting: flip enabled: false in its plugin.yml.
After first run, every plugin tool ends up listed in config/tools.yml
under plugin_tools.<plugin>.<tool_name>. Flip any of those to false
to hide a single tool from the model without disabling the whole plugin.
plugin_tools:
diary:
readDiary: true
searchDiary: true
listDiaryDates: false # the model won't see this one anymore
updateDiaryNow: trueGabriel sandboxes ctx.config by default. Plugins can read their own
settings under plugins.<name>.* via ctx.plugin_config() but reads of
sensitive things like ctx.config.api_key, mongo strings, vrchat password
or discord token raise PermissionError.
A handful of plugins here (notably diary/) reuse the main gemini key
for a background sub-agent. To enable those, set:
plugins:
enabled: true
trusted: trueDefault is false. Only flip this on if you trust every plugin in your
plugins/ folder, the toggle is global. Plugins should mention in their
own README if they need it.
PRs welcome. Rough rules:
| Rule | |
|---|---|
| Layout | One folder per plugin at the repo root, folder name matches name: in plugin.yml. |
| Manifest | plugin.yml with name, version, api_version, author, description, enabled. |
| Docs | Short README.md in the plugin folder explaining what it does, what config it reads, any external services it depends on. |
| Deps | List pip deps in plugin.yml :: requirements:. Don't bundle wheels or binaries. |
| Privacy | No bundled secrets, api keys, or personal data. data/ style runtime state is the host's responsibility. |
| License | AGPL-3.0 compatible (matches the host project). |
| Honesty | If your plugin needs a private or external service, say so up front in the README. |
Reference plugins in order of complexity:
example_hello/-- bare minimum, one tool, two event subs.mood/-- persistent state, prompt contributor, JSON overrides.pocket_tts/-- CPU TTS provider with streaming + voice cloning + sidecar config + background warmup.omnivoice_tts/-- GPU TTS provider with sentence-batched streaming, voice cloning, voice design, anchor sentence trick, opt-in CUDA graph cache.diary/-- background scheduler, sub-agent calling another Gemini model, structured output, multiple tools.
Read those before asking how to do anything. Most patterns are already demonstrated there.
The full author guide and all PluginContext surface details live in the
host repo at
plugins/README.md.
Same license as the host project: GNU Affero General Public License v3.0. See LICENSE in the host repo for details.
Made with care for Project Gabriel · Hoppou.AI · Discord