This is my keyboard, Keychron V1:
It’s a pretty fire keyboard for the price, and I have to say that it did transition me into a mechanical keyboard fan and now I definitely can tell how bad my previous keyboards were.
It also features bisexual RGB lighting under the keys, of which there are 81 in a 75% layout.
They are all addressable RGB LEDs, which means that they are individually controllable.
/keyboard-display/bilighting.jpeg){kind=link}
Wheeee… Included effects…
But this is no fun. Couple of static effects? Surely there is a better* usage for 81!!!! pixels. So let’s try using it as a display.
My keyboard choice wasn’t random
Honestly I would have continuing using my previous keyboard for a hundred more years, if the keyboard that fell in my hands wasn’t the worst I’ve ever used. The problem is not even how it feels (although it def wasn’t good), it’s insane levels of ghosting.
Imagine trying to type grub, but instead you get grugrubru. Even the cheapest bulk-buy office-grade keyboards don’t do that (in my experience). But, that was the case for mine, and grugrubru happened to a bunch of other words too. So, I had to pick a keyboard to eventually buy. (in the end it was gifted to me, tyvm to that person, but the choice of which one to get was still made by me)
One of the criteria I had while picking a keyboard was QMK support. Why? Well, QMK is an “open-source firmware for microcontrollers that control computer keyboards.” This means that I can easily flash anything onto it, from simply switching layouts to turning it into a joystick (racing games on a rotary knob anyone?) and I love a piece of hardware that I can modify the hell out of. And yeah, at one point I did get an idea to make a “display” out of it.
If you are wondering what was the other criteria: mechanical, tactile switches, addressable RGB lighting (hehe) and 75% layout.
Useless repurpose (or not?)
Now for the reason you are reading this: displaying stuff.
I want to make an actual screen, so we will at least need to send some data to control what’s shown on the keyboard (and maybe also save the typing capabilities of it.)
However, QMK has no out of box way to control individual LEDs through USB, although there is an old issue about implementing an API for QMK.
But, there is a way to communicate between PC and keyboard - RawHID. In essence, this is just a little way to send and receive 32 byte packages through HID.
32 bytes is a hard limitation for one packet, but no one stops you from implementing a protocol on top of it to support packets spanning multiple reports (word’s on the street this is how Level 4 protocols ended up being created). We don’t actually need to do that though, since setting one LED to an RGB color should only take 4 bytes (1 for LED index, 3 for color). In fact, that means we can put 8 LED changes into one report.
Actually, due to how I made my little protocol, one additional byte was spent on a packet ID. That’s fixable though, either by packing first LED ID with the packet ID (e.g. making the first bit an indicator for the LED packet and using the other 7 to control up to 128 LEDs) or settings LEDs sequentially, which would mean that each color needs only 3 bytes (upping the LED count we can change at one up to 10.)
RawHID transmission is kinda slow, so optimizing a protocol like that in this case could be beneficial, but here I haven’t done that. Just use this info in case you need to make a keyboard into a display efficiently (and please tell me if you have an actual use case for all this)
So it’s time to modify some code! QMK is thankfully pretty easy to work with - this page gives everything you need to do some QMKing and describes the process better than I do, but in short:
- Install the QMK CLI - I used uv tool install qmk even though I’m on Arch Linux1, because I wanted to see what uv is all about
- Keychron V1’s keyboard files are upstream, so one quick qmk setup is enough to get a working environment.
- cd ~/qmk_firmware (assuming default location)
QMK is split into configurations for keyboard models and keymaps, and really we need to only modify two files - rules.mk (the actual configuration) and keymap.c that defines the keymap (who would’ve thought) and some keyboard-specific code. They are located in keyboards/(maker)/(model)/(variant) (maker might be omitted), and in the subfolder keymaps you can guess what is stored.
My keyboard is a keychron/v1/ansi_encoder (encoder refers to the rotary knob), and the default (and only) keymap is called default, so let’s just try compiling and flashing stock firmware:
~/qmk_firmware ❯ qmk compile -kb keychron/v1/ansi_encoder -km default # ... Copying keychron_v1_ansi_encoder_default.bin to qmk_firmware folder [OK] ~/qmk_firmware ❯ qmk flash -kb keychron/v1/ansi_encoder -km default Flashing for bootloader: stm32-dfu Bootloader not found. Make sure the board is in bootloader mode. See https://docs.qmk.fm/#/newbs_flashing
Oh yeah, we need to connect the keeb in DFU mode - quick look at the manual reveals that you need to hold a button under the space bar while connecting to PC.
Oh boy, my keyboard is not clean…
Nov 08 14:11:52 stopper kernel: usb 1-10: new full-speed USB device number 10 using xhci_hcd Nov 08 14:11:52 stopper kernel: usb 1-10: device descriptor read/64, error -71 Nov 08 14:11:52 stopper kernel: usb 1-10: device descriptor read/64, error -71 Nov 08 14:11:52 stopper kernel: usb 1-10: new full-speed USB device number 11 using xhci_hcd Nov 08 14:11:53 stopper kernel: usb 1-10: device descriptor read/64, error -71 Nov 08 14:11:53 stopper kernel: usb 1-10: device descriptor read/64, error -71 Nov 08 14:11:53 stopper kernel: usb usb1-port10: attempt power cycle Nov 08 14:11:53 stopper kernel: usb 1-10: new full-speed USB device number 12 using xhci_hcd Nov 08 14:11:53 stopper kernel: usb 1-10: Device not responding to setup address.
IT SHOULD HAVE BEEN EASY but USB strikes once again.
…is what I wanted to say at that moment, but whoops, apparently I’ve just been holding the reset button for too long. Sorry USB.
... Erase done. Download done. File downloaded successfully Submitting leave request... Transitioning to dfuMANIFEST state ~/qmk_firmware ❯
Yay, now I can’t type normally because no VIA anymore 😭😭😭. Should’ve expected that…2
Whatever, at least we confirmed that we can build and flash firmware. Changing existing keymaps is not recommended - QMK docs say to instead make a personal one, so I’ll just follow along:
# Setting defaults for qmk compile and other CLI commands ~/qmk_firmware ❯ qmk config user.keyboard=keychron/v1/ansi_encoder user.keyboard: None -> keychron/v1/ansi_encoder Ψ Wrote configuration to /home/stopper/.config/qmk/qmk.ini ~/qmk_firmware ❯ qmk config user.keymap=stopperw user.keymap: None -> stopperw Ψ Wrote configuration to /home/stopper/.config/qmk/qmk.ini # Cloning the default keymap ~/qmk_firmware ❯ qmk new-keymap Ψ Generating a new keymap Ψ Created a new keymap called stopperw in: /home/stopper/qmk_firmware/keyboards/keychron/v1/ansi_encoder/keymaps/stopperw. Ψ Compile a firmware with your new keymap by typing: qmk compile -kb keychron/v1/ansi_encoder -km stopperw. # Checking it out ~/qmk_firmware ❯ cd ~/qmk_firmware/keyboards/keychron/v1/ansi_encoder/keymaps/stopperw # I will shorten the path a little bit for readability ~/qmk_firmware/.../stopperw ❯ eza -la --git .rw-r--r-- 5.4k stopper 8 Nov 13:25 -N keymap.c .rw-r--r-- 25 stopper 8 Nov 13:25 -N rules.mk
Heeey, here are those files I mentioned earlier! Let’s see…
~/qmk_firmware/.../stopperw ❯ cat rules.mk ENCODER_MAP_ENABLE = yes
Hmmm, pretty empty. But that doesn’t matter.
rules.mk files are recursively applied on top of one another. For this keyboard, there is also a file in keyboards/keychron/v1/ansi_encoder/rules.mk, but we don’t need to touch it.
To enable Raw HID, all that’s need to be done is to add RAW_ENABLE = yes to this file. Note that everything we do from now on isn’t compatible with VIA or VIAL. Quick check…
# No need for -kb and -km anymore. ~/qmk_firmware/.../stopperw ❯ qmk compile # ... Compiling: quantum/raw_hid.c [OK] # ... Copying keychron_v1_ansi_encoder_stopperw.bin to qmk_firmware folder [OK]
Apparently RTFM pays off sometimes.
As for RawHID’s usage, docs are again pretty descriptive about what’s need to be done, that being adding a raw_hid_receive function to our keymap - they even provide a simple code example:
// keymap.c /* ... */ void raw_hid_receive(uint8_t *data, uint8_t length) { uint8_t response[length]; memset(response, 0, length); response[0] = 'B'; if (data[0] == 'A') { raw_hid_send(response, length); } }
~/qmk_firmware ❯ qmk compile && qmk flash # ... ./.../stopperw/keymap.c:82:9: error: implicit declaration of function 'raw_hid_send'; did you mean 'host_raw_hid_send'? [-Wimplicit-function-declaration] 82 | raw_hid_send(response, length);
No one mentioned you need to #include "raw_hid.h" though… Anyways, after fixing that, it compiles and… well it seems that we need some client software now. Docs helpfully provide a Python example, that I will throw away because I like Rust more.
Quick lib.rs search showed me the hidapi crate, it even uses the same underlying library as the Python example, and it seems really easy to use!
// main.rs use hidapi::HidApi; fn main() { let api = HidApi::new().unwrap(); // Print out information about all connected devices for device in api.device_list() { println!("{:#?}", device); } }
# Called the project `bell7`. No idea why. ~/bell7 ❯ lsusb # ... Bus 003 Device 012: ID 3434:0311 Keychron Keychron V1 # ... ~/bell7 ❯ cargo run # ... HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } HidDeviceInfo { vendor_id: 13364, # 0x3434 product_id: 785, # 0x0311 } # ...
Hmm, I hoped for a little bit more information. And also, what’s the deal with these 7 identical device infos for my keyboard? It’s the correct VID/PID for my keyboard though, so let’s just hardcode them and see what can be done.
Quick fact about USB: USB-IF is responsible for giving out Vendor IDs for every manufacturer of USB capable devices and registering Product IDs for them, and really you are not supposed to sell USB-having stuff without one (and getting one is hella expensive).
But, all that is really optional for personal stuff. Also, under some conditions, you can use pid.codes to use their VID without paying for your own, or VID 0xF055 for FOSS stuff (unofficial though, don’t recommend selling with this one)
Keychron did get VID 0x3434 for themselves (you can lookup {V,P}IDs in Linux’s USB ID repository)
let (vid, pid) = (0x3434, 0x0311); let device = api.open(vid, pid).unwrap(); println!("{:?}", device.get_product_string()); println!("{:?}", device.get_manufacturer_string()); println!("{:?}", device.get_serial_number_string());
~/bell7 ❯ cargo run
Running `target/debug/bell7`
thread 'main' (62889) panicked at src/main.rs:7:37:
called `Result::unwrap()` on an `Err` value: HidApiError { message: "Failed to open a device with path '/dev/hidraw6': Permission denied" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Can’t say I’m surprised, had to deal with that when VIA was giving the most unhelpful error messages
(saw a real “task failed successfully” there - “HID device connected” with an X icon lol) when connecting my keyboard.
I can just sudo chmod 777 /dev/hidraw* again, but today I want to do it properly, with a udev rule.
# /etc/udev/rules.d/51-hidraw.rules KERNEL=="hidraw*", SUBSYSTEM=="hidraw", MODE="0660", GROUP="uucp"
This rule gives 660 permissions for HID device raw access for everyone in group uucp (its the group for RS-232 serial ports so I thought that it’ll kinda fit in).
~/bell7 ❯ cargo run Running `target/debug/bell7` Ok(Some("Keychron V1")) Ok(Some("Keychron")) Ok(Some("3E0******"))
Now that it’s confirmed to work, we can talk try talking through HID.
// this is the only report size RawHID supports const REPORT_SIZE: usize = 32; fn main() { let api = HidApi::new().unwrap(); let (vid, pid) = (0x3434, 0x0311); let device = api.open(vid, pid).unwrap(); let mut send_buf = [0u8; REPORT_SIZE]; send_buf[0] = b'A'; let len = device.write(&send_buf).unwrap(); println!("Wrote {:?} bytes", len); let mut recv_buf = [0u8; REPORT_SIZE]; let len = device.read(&mut recv_buf).unwrap(); println!("Response: {:?} ({:?} bytes)", recv_buf, len); }
~/bell7 ❯ cargo run Running `target/debug/bell7` Wrote 32 bytes # ... considerable delay ... Response: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] (8 bytes)
That’s… not what I expected.
No, no, no, I’ve totally RTFMed…
// notice the added +1 let mut send_buf = [0u8; REPORT_SIZE + 1]; // according to hidapi's docs, first byte is the HID Repord ID; // it's left as 0x00 as QMK RawHID doesn't use report IDs send_buf[1] = b'A'; let len = device.write(&send_buf).unwrap(); println!("Wrote {:?} bytes", len); let mut recv_buf = [0u8; REPORT_SIZE]; let len = device.read(&mut recv_buf).unwrap(); println!("Response: {:?} ({:?} bytes)", recv_buf, len);
~/bell7 ❯ cargo run Running `target/debug/bell7` Wrote 33 bytes # ... considerable delay ... Response: [8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] (8 bytes)
Well now I did everything correctly! And I’m still not getting a 'B'! Can’t blame me!
But I wonder if it has something to do with there being 7 of my keyboard… Does HidDeviceInfo contain something useful?
for device_info in api.device_list() { println!( "{:04x}:{:04x} - {:?}", device_info.vendor_id(), device_info.product_id(), device_info.product_string() ); println!( "Usage page: {:04x} | Usage: {:02x}", device_info.usage_page(), device_info.usage() ); }
~/bell7 ❯ cargo run Running `target/debug/bell7` # ... 3434:0311 - Some("Keychron V1") Usage page: 0001 | Usage: 06 3434:0311 - Some("Keychron V1") Usage page: ff60 | Usage: 61 3434:0311 - Some("Keychron V1") Usage page: 0001 | Usage: 02 3434:0311 - Some("Keychron V1") Usage page: 0001 | Usage: 01 3434:0311 - Some("Keychron V1") Usage page: 0001 | Usage: 80 3434:0311 - Some("Keychron V1") Usage page: 000c | Usage: 01 3434:0311 - Some("Keychron V1") Usage page: 0001 | Usage: 06
Aha, it does! These are actually not just duplicates, but different HID interfaces with their own usages… that are not printed when using the Debug implementation of this struct. As stated in Raw HID docs, the usage values we are looking for are 0xff60 0x61, and there is indeed one matching device. If only there was an easy way to filter devices…
let device = api .device_list() .filter(|x| x.usage_page() == 0xff60 && x.usage() == 0x61) .next() .expect("No matching device found.") .open_device(&api) .unwrap();
And this is why I love Rust.
~/bell7 ❯ cargo run Running `target/debug/bell7` Wrote 33 bytes Response: [66, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] (32 bytes)
66 is indeed decimal for ASCII 'B', aka the response we expected. Finally, it’s time to do something useful**.
Something useful
As a good keyboard firmware, QMK comes with 4 different lighting modules: Backlight, LED Matrix, RGB Lighting, RGB Matrix. There is so many, because some are for addressable LEDs, some are for full-board RGB only, and well you get the point. Which one is used can be determined by looking at keyboard’s code (or searching through it, that’s probably way easier):
# rg is ripgrep, regex search through file contents basically ~/qmk_firmware/keyboards/keychron/v1 ❯ rg rgb info.json 10: "rgb_matrix": { v1.c 41: switch (rgb_matrix_get_flags()) { 43: rgb_matrix_set_flags(LED_FLAG_NONE); 44: rgb_matrix_set_color_all(0, 0, 0); 47: rgb_matrix_set_flags(LED_FLAG_ALL); 51: if (!rgb_matrix_is_enabled()) { 52: rgb_matrix_set_flags(LED_FLAG_ALL); 53: rgb_matrix_enable(); 61:bool rgb_matrix_indicators_advanced_kb(uint8_t led_min, uint8_t led_max) { 62: if (!rgb_matrix_indicators_advanced_user(led_min, led_max)) { return false; } 68: if (!rgb_matrix_get_flags()) { ansi_encoder/keyboard.json 25: "rgb_matrix": true ...
Personally, I have a nagging suspicion that it uses RGB Matrix.
That comes with an API and a defined
led_color_t struct
that provides a key to LED index matrix by physical position - but for now, just as a proof of concept,
I’ll stick to just setting the LED by indexes. (they did not, in fact, use LED indexes just “for now”)
Let’s just set the first LED to pure red when the dreaded A arrives and see what happens:
void raw_hid_receive(uint8_t *data, uint8_t length) { if (data[0] == 'A') { // Stops any effect rgb_matrix_mode(RGB_MATRIX_NONE); rgb_matrix_set_color(0, 255, 0, 0); } }
Not… quite what I expected… but something worked. What about this?
void raw_hid_receive(uint8_t *data, uint8_t length) { if (data[0] == 'A') { // Stops any effect rgb_matrix_mode(RGB_MATRIX_NONE); rgb_matrix_set_color(0, 255, 0, 0); rgb_matrix_set_color(1, 0, 255, 0); rgb_matrix_set_color(2, 0, 0, 255); } }
Nothing worked. Why…
Well, here I was actually stunned for a while, because I didn’t realise that setting the matrix mode to NONE actually doesn’t change it to none, instead it reverts to the “solid color” effect, which is conveniently set to hue=0 (red) by default. lol
Okay, okay, I’ll do everything the proper way…
bool rgb_matrix_indicators_user(void) { rgb_matrix_set_color(0, 255, 0, 0); rgb_matrix_set_color(1, 0, 255, 0); rgb_matrix_set_color(2, 0, 0, 255); return false; }
As per RGB Matrix docs, proper way is to define a callback in keymap.c and call any RGB Matrix-related functions there.
And it works! Though now I’ll have to store RGB values for each LED in an array, set it from HID packet receiver and read it every time the RGB Matrix callback is called,
// R,G,B for each LED // hdr support when? uint8_t remote_colors[RGB_MATRIX_LED_COUNT * 3]; void raw_hid_receive(uint8_t *data, uint8_t length) { if (data[0] == 'A') { // LED 0, R=255 remote_colors[0] = 255; // LED 1, G=255 remote_colors[4] = 255; // LED 2, B=255 remote_colors[8] = 255; } } bool rgb_matrix_indicators_user(void) { for (uint8_t i = 0; i < RGB_MATRIX_LED_COUNT; i++) { rgb_matrix_set_color( i, remote_colors[i * 3], remote_colors[i * 3 + 1], remote_colors[i * 3 + 2] ); } return false; }
How the life looks after one day on A:
Well, just reacting to a single command for boring action is… boring, so I’ll need a simple protocol that I can then use from client software.
bool remote_mode = false; uint8_t remote_colors[RGB_MATRIX_LED_COUNT * 3]; void clear_colors(void) { for (uint8_t i = 0; i < RGB_MATRIX_LED_COUNT * 3; i++) { remote_colors[i] = 0; } } void set_color(uint8_t index, uint8_t r, uint8_t g, uint8_t b) { remote_colors[index * 3] = r; remote_colors[index * 3 + 1] = g; remote_colors[index * 3 + 2] = b; } void raw_hid_receive(uint8_t *data, uint8_t length) { switch (data[0]) { // Enable case 0x00: clear_colors(); remote_mode = true; break; // Disable case 0x01: clear_colors(); remote_mode = false; break; // Write a single LED case 0x02: // data[1] = index if (data[1] >= RGB_MATRIX_LED_COUNT) data[1] = RGB_MATRIX_LED_COUNT - 1; set_color(data[1], data[2], data[3], data[4]); break; // Write a series of LEDs case 0x03: // data[1] = start index // data[2] = count // data[3 + i*3 (+1/2)] = R/G/B values for each LED if (data[1] >= RGB_MATRIX_LED_COUNT) data[1] = RGB_MATRIX_LED_COUNT - 1; if (data[2] + data[1] > RGB_MATRIX_LED_COUNT) return; for (uint8_t i = data[1]; i < data[1] + data[2]; i++) { set_color( i, data[3 + i*3], data[3 + i*3 + 1], data[3 + i*3 + 2] ); } // So elaborate... yeah I didn't even end up using this one. // Might also be broken, who knows. break; // Clear case 0x04: clear_colors(); break; default: return; } } bool rgb_matrix_indicators_user(void) { if (!remote_mode) return true; for (uint8_t i = 0; i < RGB_MATRIX_LED_COUNT; i++) { rgb_matrix_set_color( i, remote_colors[i * 3], remote_colors[i * 3 + 1], remote_colors[i * 3 + 2] ); } return false; }
and a ✨fancy✨ Rust implementation of it:
fn main() -> color_eyre::Result<()> { let api = HidApi::new().unwrap(); let device = api .device_list() .filter(|x| x.usage_page() == 0xff60 && x.usage() == 0x61) .next() .wrap_err("No matching device found.")? .open_device(&api)?; Packet::Enable.write(&device)?; for i in 0..16 { Packet::SingleWrite { i: i, r: 255, g: 0, b: 255, } .write(&device)?; } Ok(()) } enum Packet { Enable, Disable, SingleWrite { i: u8, r: u8, g: u8, b: u8 }, // was too lazy to implement the series write Clear, } impl Packet { const REPORT_SIZE: usize = 32; pub fn write(&self, device: &HidDevice) -> color_eyre::Result<usize> { let mut send_buf = [0u8; Self::REPORT_SIZE + 1]; match self { Packet::Enable => { send_buf[1] = 0x00; } Packet::Disable => { send_buf[1] = 0x01; } Packet::SingleWrite { i, r, g, b } => { send_buf[1] = 0x02; send_buf[2] = *i; send_buf[3] = *r; send_buf[4] = *g; send_buf[5] = *b; } Packet::Clear => { send_buf[1] = 0x04; } } Ok(device.write(&send_buf)?) } }
…which also should just set the first 15 LEDs to purple. So, will it work?
Sorry, YouTube embeds have too much tracking for my liking, so there will only be links.
Hope you will check them out anyway :)
It may seem blue, but it’s actually purple (I’m truly sorry, the photographer in me is on life support), and it actually works. Also, don’t ask how I pressed Enter with both hands full. You don’t want to know.
Showing them images
Now for the most interesting part, the part that probably made you click on this article - using this thing I built as a display. My newfound RGB matrix needs some “source material”, so I’ll just use my monitor’s contents.
This means I’ll have to do screen capture. I’m on Linux, KDE Plasma, Wayland city, so it’s usually even more pain, because portals and PipeWire and stuff, but Rust community saves the day once again, because someone wrote a screen capture library called XCap!
Let’s see… and by see, I mean try capturing something (and also measuring the performance, we don’t want a 3 FPS keyboard, do we?)
let monitor = Monitor::all()? .into_iter() .filter(|x| x.is_primary().unwrap()) .next() .wrap_err("No primary monitor found.")?; for i in 0..100 { let start = std::time::Instant::now(); monitor.capture_image()?; println!("Took {:?}", start.elapsed()); }
~/bell7 ❯ cargo run
Running `target/debug/bell7`
Took 781.034807ms
Took 864.623518ms
Took 836.140517ms
Took 812.079465ms
Took 787.699925ms
Took 765.217787ms
Took 777.90531ms
Took 765.086713ms
Took 797.440072ms
Took 821.824797ms
Ouch. It doesn’t look good. Well, I used the screenshot API, but there is also a second way to do things in this library - video recorder, which is its solution to live screen capture, that might be more fitting here.
// Example from XCap's README use std::{thread, time::Duration}; use xcap::Monitor; fn main() { let monitor = Monitor::from_point(100, 100).unwrap(); let (video_recorder, sx) = monitor.video_recorder().unwrap(); thread::spawn(move || loop { match sx.recv() { Ok(frame) => { println!("frame: {:?}", frame.width); } _ => continue, } }); println!("start"); video_recorder.start().unwrap(); thread::sleep(Duration::from_secs(2)); println!("stop"); video_recorder.stop().unwrap(); }
~/bell7 ❯ cargo run Running `target/debug/bell7` start frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 frame: 1920 stop
13 frames rendered in 2 seconds, which means… 7 FPS? Well that’s not gaming at all.
Ah, right, I should probably use a release build. Makes sense.
~/bell7 ❯ cargo run --release Running `target/debug/bell7` start # ... 106 other lines of FRAME frame: 1920 stop
54 FPS (107 frames in 2 seconds), now we are talking. I wonder if I could continue using the screenshot API…
Now to send data through HID… let’s not overcomplicate things and just put everything in the video recorder’s receive thread (spoiler: it led to everything being even more complicated).
Frames we receive from XCap are stored in 8-bit RGBA (I wonder what alpha is used for?), one pixel after another, in the raw field of Frame struct.
PIXEL 0 PIXEL 1 PIXEL 1920 0 1 2 3 4 5 6 7 7680 7681 7682 7683 +---+---+---+---+ +---+---+---+---+ +-----+-----+-----+-----+ | R | G | B | A | | R | G | B | A | ... | R | G | B | A | ... +---+---+---+---+ +---+---+---+---+ +-----+-----+-----+-----+
Utilizing this knowledge…
let monitor = Monitor::all()? .into_iter() .filter(|x| x.is_primary().unwrap()) .next() .wrap_err("No primary monitor found.")?; let (recorder, sx) = monitor.video_recorder()?; std::thread::spawn(move || { fn tc(sx: std::sync::mpsc::Receiver<xcap::Frame>) -> color_eyre::Result<()> { let api = HidApi::new().unwrap(); let device = api .device_list() .filter(|x| x.usage_page() == 0xff60 && x.usage() == 0x61) .next() .wrap_err("No matching device found.")? .open_device(&api)?; Packet::Enable.write(&device)?; let mut frame_count = 0; loop { match sx.recv() { Ok(frame) => { frame_count += 1; Packet::SingleWrite { i: 0, r: frame.raw[0], g: frame.raw[1], b: frame.raw[2], } .write(&device)?; println!("frame: {}", frame_count); } _ => break, } } Ok(()) } tc(sx).unwrap(); }); println!("start"); recorder.start().unwrap(); std::thread::sleep(std::time::Duration::from_secs(2)); println!("stop");
Ah, beautiful code, 6 levels of indentation, all just to send the first pixel. And, something actually works!
(color is off on the image but) Gray/very light blue? Apparently that’s how 0x1b1d1d looks? Whatever, let’s just fill the whole keyboard.
I’ll use a 16x6 area in the top-left of my screen, which is 96 pixels, but it will be limited on each row by the numbers of keys on the actual row on the keyboard. Of course, the better solution would be to estimate the closest key to a given pixel, but that’s for some other time.
std::thread::spawn(move || { fn tc(sx: std::sync::mpsc::Receiver<xcap::Frame>) -> color_eyre::Result<()> { let api = HidApi::new().unwrap(); let device = api .device_list() .filter(|x| x.usage_page() == 0xff60 && x.usage() == 0x61) .next() .wrap_err("No matching device found.")? .open_device(&api)?; Packet::Enable.write(&device)?; // scientifically determined by counting keys let rows = [14, 15, 15, 14, 13, 10]; let mut rows_accumulated = [14, 15, 15, 14, 13, 10]; for i in 0..rows.len() { rows_accumulated[i] = rows[i]; if i != 0 { rows_accumulated[i] = rows[i] + rows_accumulated[i - 1]; } } let mut frame_count = 0; loop { match sx.recv() { Ok(frame) => { frame_count += 1; let pixel_length = 4usize; let row_length = (frame.width as usize) * pixel_length; for i in 0..81 { let (row_index, _) = rows_accumulated.iter().enumerate().filter(|(_, v)| **v > i).next().unwrap(); // Index on the monitor row let mut ri = i; if row_index > 0 { ri = i - rows_accumulated[row_index - 1]; } Packet::SingleWrite { i: i as u8, r: frame.raw[row_length * row_index + ri * pixel_length], g: frame.raw[row_length * row_index + ri * pixel_length + 1], b: frame.raw[row_length * row_index + ri * pixel_length + 2], } .write(&device)?; } println!("frame: {}", frame_count); } _ => break, } } Ok(()) } tc(sx).unwrap(); });
If you feel like my code is getting worse with every snippet… it’s not just a feeling. But look!
ITS WHITE-ISH, IT JUST LOOKS BLUE ON PICTURES, TRUST ME.
By the way, this is what happens if you don’t consider the alpha channel:
But, still… something is wrong. It doesn’t really properly update. Stuff does change, sometimes, but not when I change it.
Well, I had my suspicions before, but running in debug mode confirmes them (because frames are rarer and USB communication actually can keep up): XCap generates more frames than I can keep up sending through HID. Fear not, because another despicable piece of code to fix this issue will be written… now!
let mut last_frame: Option<Frame> = None; loop { match sx.try_recv() { Ok(frame) => { last_frame = Some(frame); frame_count += 1; } _ => { if last_frame.is_none() { continue; } let frame = last_frame.unwrap(); let pixel_length = 4usize; let row_length = (frame.width as usize) * pixel_length; for i in 0..81 { let (row_index, _) = rows_accumulated.iter().enumerate().filter(|(_, v)| **v > i).next().unwrap(); let mut ri = i; if row_index > 0 { ri = i - rows_accumulated[row_index - 1]; } Packet::SingleWrite { i: i as u8, r: frame.raw[row_length * row_index + ri * pixel_length], g: frame.raw[row_length * row_index + ri * pixel_length + 1], b: frame.raw[row_length * row_index + ri * pixel_length + 2], } .write(&device)?; } println!("frame: {}", frame_count); last_frame = None; }, } }
Skipping frames if we can’t keep up, how original.
Now, to do funnies with this setup… we can display anything…
Well, here are two things people do when they get some piece of hardware they can show stuff on - they either run DOOM on it, or play Bad Apple.
I’m too lazy to run DOOM, so how about some Bad Apple?
Nothing fancy is needed here, just rescaling through ffmpeg and putting mpv at the right position on the screen.
# Fun fact: Bad Apple MV was originally shared on Nicovideo: # https://www.nicovideo.jp/watch/sm8628149 # Fun fact 2: MV actually has the song cut - original song is 5:19 long ~/Downloads ❯ yt-dlp https://www.youtube.com/watch?v=FtutLA63Cp8 # ... ~/Downloads ❯ ffmpeg -i 【東方】Bad\ Apple!!\ ＰＶ【影絵】\ \[FtutLA63Cp8\].webm -vf "scale=16:6,setsar=1:1" badapple.mp4 # ... ~/Downloads ❯ mpv badapple.mp4
Bell7’s source code is available on my Forgejo instance!
(mini)Games?
Now, I could just make funny games in Godot and render them in 16x6, capture them using the same method, but… what about doing everything inside Godot itself?
Godot 4 has a pretty useful thing called GDExtensions, which allow you to use and run native code from shared libraries inside of Godot. And there are Rust bindings for it! So what I’m going to do is write the minigames I want AND the keyboard display code in the same place.
how 2 rust gdext
I think Rust books are great (I really am a Rust shill, aren’t I?)! I’m not saying it just to say it though, there is actually a godot-rust one, so it’ll all be easy to setup if you want to follow along.
Here, I’ll actually skip to the interesting part: how to send the image from Godot to keyboard? Obviously, sending pixel data is not a problem anymore, but there is no GodotCap as far as I’m aware.
My idea is to just shrink the game’s Window to 16x6 and each frame read pixels from it that will be then sent to the keeb. Window is a Viewport, and Viewports have a “screen” texture you can just get and read from. Pretty neat, right?
One script on a Node is basically enough for this purpose:
use godot::prelude::*; #[derive(GodotClass)] #[class(base=Node)] struct KeebSender { base: Base<Node> } #[godot_api] impl INode for KeebSender { fn init(base: Base<Node>) -> Self { Self { base, } } fn ready(&mut self) { godot_print!("ah"); } }
I’ll also make a quick 2D scene filled by PINK (aka placeholder texture)
Default SceneTree’s root is always a Window, so we can get that, get the texture out of it, get an image out of it (basically an interface to texture’s data) and read the first pixel (what a mouthful):
If you are unfamiliar with how Godot works: Most Godot games use a high-level system of nodes, that are stored in a SceneTree, where they do the actual computation. SceneTree is Godot’s primary main loop (the thing that sends updates every frame and a lot of other useful stuff, like input).
Official docs explain these concepts in more detail.
fn ready(&mut self) { // Mostly self-descriptive code :) // Lots of unwrap()s tho, scary!! let window = self.base().get_tree().unwrap().get_root().unwrap(); let texture = window.get_texture().unwrap(); // Copies the full texture each call: let image = texture.get_image().unwrap(); let pixel = image.get_pixel(0, 0); godot_print!("PIXEL 0 COLOR: {:?}", pixel); }
Hm? Pretty sure window’s not black.
Godot docs for ViewportTexture:
Note: When trying to store the current texture (e.g. in a file), it might be completely black or outdated if used too early, especially when used in e.g. Node._ready().
To make sure the texture you get is correct, you can await RenderingServer.frame_post_draw signal.
No way, they read my mind! Well, if it’s a signal, might as well move everything into it.
use godot::{classes::RenderingServer, obj::WithBaseField, prelude::*}; #[derive(GodotClass)] #[class(base=Node)] struct KeebSender { frame: u32, base: Base<Node>, } #[godot_api] impl INode for KeebSender { fn init(base: Base<Node>) -> Self { Self { frame: 0, base } } fn ready(&mut self) { RenderingServer::singleton().signals().frame_post_draw().connect_other(self, Self::on_frame_drawn); } } #[godot_api] impl KeebSender { #[func] fn on_frame_drawn(&mut self) { self.frame += 1; let window = self.base().get_tree().unwrap().get_root().unwrap(); let texture = window.get_texture().unwrap(); let image = texture.get_image().unwrap(); let pixel = image.get_pixel(0, 0); godot_print!("[FRAME {}] PIXEL 0 COLOR: {:?}", self.frame, pixel); } }
Now, you say what you want, but I’m saying that this is insanely clean.
And really, the only things that needs to be done now, is to copy the connection code into ready and send data from on_frame_drawn.
#[derive(GodotClass)] #[class(base=Node)] struct KeebSender { frame: u32, device: Option<HidDevice>, base: Base<Node>, } #[godot_api] impl INode for KeebSender { fn init(base: Base<Node>) -> Self { Self { frame: 0, device: None, base, } } fn ready(&mut self) { let api = HidApi::new().unwrap(); let device = api .device_list() .filter(|x| x.usage_page() == 0xff60 && x.usage() == 0x61) .next() .wrap_err("No matching device found.") .unwrap() .open_device(&api) .unwrap(); Packet::Enable.write(&device).unwrap(); self.device = Some(device); RenderingServer::singleton() .signals() .frame_post_draw() .connect_other(self, Self::on_frame_drawn); } } #[godot_api] impl KeebSender { #[func] fn on_frame_drawn(&mut self) { if self.device.is_none() { return; } let device = self.device.as_ref().unwrap(); self.frame += 1; let window = self.base().get_tree().unwrap().get_root().unwrap(); let texture = window.get_texture().unwrap(); let image = texture.get_image().unwrap(); // scientifically determined by counting keys let rows = [14, 15, 15, 14, 13, 10]; let mut i = 0; for (y, range) in rows.into_iter().enumerate() { for x in 0..range { let pixel = image.get_pixel(x as i32, y as i32); Packet::SingleWrite { i: i as u8, r: pixel.r8(), g: pixel.g8(), b: pixel.b8(), } .write(&device) .unwrap(); i += 1; } } } }
If only I didn’t need all these maths for the screen capture example…
This video is actually pretty good at showing just how unaligned the keyboard LEDs are. Can’t blame them though, I don’t think Keychron had displays in mind when designing this keyboard.
Now, as we are in Godot, I probably should do some gamedev? Although, what games can you fit into a 16x6 area?
The game I decided to make is… columns! I don’t think there is much interesting to say about the columns’ game code, but I did publish it anyway on my Forgejo instance.
Okay, there is one fun thing to say: due to the game being 5x12 I had to debug it by zooming into it (on KDE it’s an accessibility option.)
Here is, what might be, the first Columns gameplay on a keyboard:
Conclusion
Was this useful/made any sense? No. Was it all fun? Yeah! I also cleaned my keyboard, because I had to pull all keycaps off it *multiple times.
How convenient!
Also had to update victorian (blog engine) because it had no footnote support up to this point. In fact, I found out I didn’t even have working italics!
This engine is such a pile of duct tape and glue…
Though, I think this post might serve as a good introduction to keyboard modding for some. Whatever is the reason you read to the end, I hope you had as much fun as I did. Maybe consider subscribing to my RSS feed 🥺👉👈?
Footnotes:
Python packages and QMK CLIs and stuff like that on Arch Linux are recommended to be installed using pacman: pacman -Syu qmk.
VIA(L) has it’s own fork of QMK and it also uses RawHID, so we can’t use that.