AudioHax is a proof-of-concept project that converts images into music by scanning and mapping pixel data to musical parameters, applying music theory logic, and outputting MIDI to a software synthesizer in real time.
- Image-to-MIDI mapping using OpenCV.
- Multi-instrument concurrent playback.
- Configurable SoundFonts with FluidSynth.
- Basic music theory–driven chord mapping.
- Rust (Cargo) – Install Rust
- OpenCV
- Windows: Use prebuilt binaries from OpenCV Releases
- macOS: Install via Homebrew (
brew install opencv) - Linux: Install via package manager (e.g.,
sudo apt install libopencv-dev)
- loopMIDI (Windows only) – Download loopMIDI
- Virtual MIDI Driver
- macOS: Built-in IAC Driver
- Linux: ALSA MIDI (
sudo modprobe snd_virmidi)
- FluidSynth – FluidSynth Downloads or
brew install fluidsynth/sudo apt install fluidsynth - A General MIDI–compatible
.sf2SoundFont file
Example: GeneralUser GS
AudioHax/ │ Cargo.toml │ .gitignore ├───assets │ └───images | └───example.jpg ├───src │ └───main.rs │ └───image_analysis.rs │ └───image_source.rs │ └───chord_engine.rs │ └───mapping_loader.rs │ └───midi_output.rs
-
Install Rust & Cargo:
rustup-init.exe -
Install OpenCV:
Extract to C:\opencv
Set environment variables:
setx OPENCV_DIR "C:\opencv\build" setx PATH "$($Env:PATH);C:\opencv\build\x64\vc15\bin" -
Install loopMIDI and create a port named AudioHaxOut.
-
Install FluidSynth (ensure fluidsynth.exe is in PATH).
-
Install Rust:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-
Install OpenCV:
brew install opencv
-
Enable IAC Driver:
Open Audio MIDI Setup → Window > Show MIDI Studio → Double-click IAC Driver → Enable device.
-
Install FluidSynth:
brew install fluidsynth
-
Install Rust:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-
Install OpenCV:
sudo apt install libopencv-dev
-
Enable Virtual MIDI:
sudo modprobe snd_virmidi
-
Install FluidSynth:
sudo apt install fluidsynth
-
Install libclang:
sudo apt install llvm-dev libclang-dev clang
-
Install ALSA:
sudo apt install libasound2-dev
cargo clean cargo update cargo build --release
Set "New port-name:" as "AudioHaxOut"
Click + to add port
Leave running
Replace PATH_TO_SF2 with your .sf2 file:
Windows:
fluidsynth -a dsound -p AudioHaxOut -m winmidi "PATH_TO_SF2"
macOS:
fluidsynth -a coreaudio -o midi.driver=coremidi -o midi.coremidi.id="AudioHaxOut" "PATH_TO_SF2"
Linux:
fluidsynth -a alsa -o midi.driver=alsa_seq -o midi.alsa_seq.device=AudioHaxOut "PATH_TO_SF2"
cargo run --release -- play
-
Image Loading – The image is read from assets/images/example.jpg.
-
Analysis – Pixel data is scanned and mapped to note/chord information.
-
Chord Engine – Music theory logic decides what notes/chords to play.
-
MIDI Output – Notes are sent to the virtual MIDI port.
-
Audio Rendering – FluidSynth plays the notes using your SoundFont.
┌───────────────────────┐
│ Input Image (.jpg) │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ image_source.rs │
│ - Loads image │
│ - Prepares scan │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ image_analysis.rs │
│ - Scans regions │
│ - Maps pixels → notes │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ chord_engine.rs │
│ - Music theory logic │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ midi_output.rs │
│ - Sends to MIDI port │
└───────────┬───────────┘
│
▼
loopMIDI / IAC / ALSA Virtual MIDI │ ▼ FluidSynth → Audio Output