Meshcore is having a week

&& [ mesh, programming ] && 1 comments

I clicked a link in my local meshcore Discord while sipping coffee this morning as was greeted with this:

Description

Look, I’m not really a radio person. I was attracted to Meshcore because frankly, I feel like the internet has mostly gone to shit and off-grid mesh networks provide a convenient, technical form of escapism from those depressing thoughts. I was up and running with a Seedstudio solar node and a companion app for about 2 weeks before one of them died. Which died? Who knows!

During those two weeks I was able to send and receive messages from people hundreds of miles away, with no help from the internet at all. It’s very cool, I see the attraction.

But my interest in Meshcore at a technical level has diminished.

“Open source” drama

Exactly seven days ago, this post from meshcore.io - Why the split? started to make the rounds. Not just on the meshnet, but places like hacker news and lobste.rs. See, there was no meshcore.io prior to this post. It was created by members of the community in response to AndyK acting very, very badly. First, he filed for a trademark on the Meshcore name in the UK, which many people saw as a betrayal to the community. Secondly it turns out he’s a dirty vibe-coder, and that made people mad. So they forked.

A win for open source right?

Except that the people in charge of meshcore.io (if I understand this correctly) are the ones that provide the official meshcore app - which is closed source. Womp.

Exploitable firmware

A few days later we get this amazing post from Alainx277 that’s a pretty damning critique of the Meshcore code (the open source part). It’s a very good read, but the TL;DR is that the source is a nightmare, full of unchecked buffers and includes a bug that is very simple to exploit that can remotely wipe any repeater. Ouch.

I’m not an embedded engineer so I can’t cast stones here - but I did spelunk through the code fairly recently in an attempt to write my own client library called Meshcorrode and even I could tell something was up. I abandoned the project pretty quickly.

Exploitable software, too

So what’s the deal with the tasteful website with the middle finger? It looks like some enterprising members of the Meschore community decided to stick it to Andy - the vibe-coded closed source app guy (the bad one, not the good closed source app guys) by unleashing an LLM on the binary and reverse engineering the product activation code. Turns out a product key is a simple hash of the device ID it’s running on. They published their findings and created a simple html page where anyone can generate a product key trivially.

I’ll be back

You know what, I’m good on Meshcore for a bit. The community needs to figure their drama out, and all the official software needs to be open source - full stop.

I still believe that mesh networks will play an important role in the future, and I want to hack on them. I just don’t think Meshcore is it right now.

Gelly 1.2 and The Most Cursed Cache Bug

&& [ linux, programming, rust, jellyfin ] && 1 comments

Gelly is a music player for Jellyfin and Subsonic servers. I just released version 1.2, which includes support for saving favorites. A favorite is just a boolean flag on an item, nothing more. When the user clicks a cute heart icon or whatever, we flip that boolean, make a HTTP request, and move on. Should be easy to implement right?

WRONG

Implementing the feature

At first things were going really well. There is something therapeutic about hammering out CRUD code and some light UI wiring. You don’t have to think too hard, but you are making real improvements. Nerd Show was live on SomaFM. I was jamming, the kid was asleep and the universe was right.

Slowly I started noticing something. Songs which I had thought I had marked as favorite, were no longer marked as so after a recompile or restart. Not all songs - just some of them, and not the same songs consistently.

A slight mis-step

The implementation was boringly standard. Make a POST request to favorite, optimistically update the UI, check the result. For a while however, I was asking Jellyfin for the full favorite list after each call. This way I could atomically swap the client’s cache, as well as pick up changes from other clients. Since the favorite list is small, this is fine.

But Jellyfin didn’t appear to be working atomically. Fair, the API doesn’t claim to be. The second GET after the favorite POST would (sometimes) be missing recently favorited songs. Maybe the operation to flip that boolean is just so taxing that the Jellyfin devs decided to put in a task queue or some other mysterious C# thing. Seems strange, but I can work around that. With my tail between my legs, I removed the full fetch and went fully optimistic.

This is where I began to lose my mind

Making the client completely optimistic worked… while the app remained open. As soon as I’d restart and do a full sync from the server there would be one or two songs in the wrong state. When marking a favorite I’d see the successful POST, I’d give the server minutes to do whatever forsaken thing it needed to catch up, only then I’d refresh but still I’d get inconsistent state, as if the POSTs would silently fail for some songs but not others.

To make things much, much worse after hours of staring at the app, clicking “favorite” on songs I was rapidly beginning to hate, I noticed something peculiar: it only occurred on the first song in an album. How does that make any sense?

What in the actual #$%^

Meanwhile, on the complete other side of the codebase

Gelly supports playback reporting to Jellyfin.

All this does is POST a song ID to the server every 5 seconds while a song is playing. If this seems totally unrelated to the favorite status of a track, you are correct! It has nothing to do with it!

If you are starting to put it together, I salute you.

Double your cache easily

At this point I resorted to desperate internet searches: “Jellyfin favorite persistence”, “Jellyfin user state inconsistent”, “Plz why is Jellyfin like this”. Eventually I hit paydirt:

[Bug] 10.11: Favorites get lost during playback #14981

I read this bug. I frowned, and then laughed because it explained my problem so perfectly.

Apparently there’s a cache in Jellyfin for “user data”, which includes favorites, but for whatever reason isn’t invalided when a favorite is set. The playback reporting endpoint reads from this cache, updates something on it (like the play count) and then writes it to the DB. The result: stale favorite status is written to the DB, and there ain’t nothing you can damn do about it.

This means it’s essentially impossible to favorite the currently playing song! Any other song, all good!

The kicker: Gelly only sends a playback report every 5 seconds. So depending on the timing, favoriting the playing song could work, assuming I changed song/restarted app fast enough. FML.

Remember when I said this bug only appeared to happen with the first song of an album? That was wrong, but it just happened that I was normally testing by starting an album from the start.

The fix: just ship it broken lol

This bug doesn’t leave me with many good options as far as I can tell. I could:

  1. Disable reporting. Which also kills play counts, breaking a feature that has been working fine until now.
  2. Not support favorites at all, until the issue is fixed. This sucks, because it works on Subsonic.
  3. Keep favorites, but disable the favorite button only for the currently playing track, no matter where it appears. Besides being a nightmare to implement, this would result in a total UX clown show. Kind of funny, but no.
  4. Fix the bug in Jellyfin ¯\(ツ)

I decided the best course of action would be actually to go back to doing this the wrong way: fetch the full list of favorites after each POST and propagate the state. I’d rather the favorite be “undone” by the server. While frustrating, it’s less frustrating than thinking a favorite was saved only to learn it wasn’t next time you restart the app. Plus this should gracefully upgrade when the server is fixed.

Some other stuff from 1.2

Since this post is basically a long-winded release notes for 1.2, here’s some other stuff:

  1. Crossfading background blur. This was actually pretty hard to figure out. Normally this would be achieved using a GTKPicture as an overlay in the GTK widget hierarchy, but the player bar is a GTKBottomSheet that reealllly wants to paint itself the way it wants. Had to go lower level to bend it to my will, the actual fading logic was interpreted from Gapless.

  2. Bottom bar redesign. Got that modern top-border-is-the-progress bar thing going on now.

  3. A bunch of other stuff. See the Release Notes for more.

Thanks!

Jellyfin Makes a Good Audio Server

&& [ linux, self hosting, music ] && 0 comments

Music Servers

More and more people are turning to self hosting music as an alternative to streaming from Spotify. This is a great way to both support artists as well as reclaim some sovereignty over omnipresent rent-seekers.

The first question every intrepid hoster must ask themselves is of course: “Which server to choose”?

Do a little searching online and the established wisdom is pretty well defined: use something Subsonic compatible for music. This includes the likes of Navidrome, Gonic and LMS.

What I don’t see said very often is that Jellyfin, as well as being great for general media, is pretty good at music too.

I am a relative newcomer to this space. I only started self-hosting ~2 years ago and I started with video. Naturally that meant Jellyfin. Once I uploaded a couple of music albums though and realized it worked great, I became fully immersed: digging out old hard drives and ripping as many CDs as I could find. The immediate benefit was that I could do all this with a single server and set of clients. No need to run both Jellyfin and another music server if Jellyfin was working fine!

I even began working on my own music client: Gelly which started as a Jellyfin client but an OpenSubsonic backend was contributed quickly. This meant that I needed to try out some Subsonic servers!

I will admit that this entire time I’ve felt that by using Jellyfin for music, I was making a concession for the convenience of running a single server. Well, after testing out both Navidrome and Gonic to ensure Gelly was fully compatible with them, my mind has changed. I don’t think using Jellyfin is a concession at all.

Here are a few things I found that Jellyfin actually does better than any of the Subsonic servers I tried:

  1. Lyrics: Jellyfin comes with a plugin pre-installed. Simply enable it and it will run a periodic task to fetch missing lyrics. It just works, all my music has lyrics now. No need for a 3rd party program or plugin.
  2. Audio Normalization: Again, just works on Jellyfin. No plugin required, Jellyfin just calculates it. It’s still not clear to me how to get normalization/ReplayGain working on, for example, Navidrome. From what I can tell you need to run another program.
  3. Transcoding: Jellyfin supports advanced containers and HLS. The practical impact of this is you can still seek songs when streaming transcoded streams for Jellyfin. Seeking is not possible when transcoding, at least with Navidrome.

There is one place where Jellyfin is definitely inferior to the alternatives though: resource usage. Jellyfin is a hog. It’s a sprawling C# codebase nearing 30k commits. Currently on my home server it’s using almost 1GB of memory! Some would call it bloated. But if that bloat is providing things like Lyrics, ReplayGain, and transcoding, all of which should be top of mind of music enthusiasts, is it that bad? For me, the trade-off is worth it.

You should of course choose whichever server fits your needs best. They are trivial to test using podman/docker and a read-only mount of your music collection. The speed and efficiency of some of the Subsonic servers is impressive. However, don’t be too quick to pass on Jellyfin, especially if you also plan to self-host other media. There might not be as many clients yet, but they are improving. Check out Finamp for mobile and Jellyfin-Roku if you have one of those TVs. And of course the best client of them all: Gelly 😉 which will continue to support all the Subsonic servers.

Zed and Waypipe for Remote GUI Development

&& [ linux, rust ] && 0 comments

My laptop is old and under-powered and crappy and I love it. It’s a 10 year old Thinkpad T470s. Super light, great keyboard, battery lasts for ages (replaceable!) - but it’s dog slow. On the other hand, my desktop is a beast.

So what to do when I’m on the road for the holidays and want to ignore my family by hacking on my current project Gelly: a GTK + Rust application? Working directly on the Thinkpad is akin to computer torture: a clean build takes upward of 5 minutes and rust-analyzer alone maxes out the laptop’s 16 GB of ram.

Zed has a really nice remote development experience meaning I can do all the actual editing on the desktop (thanks to Tailscale) - that includes offloading rust-analyzer. I have a simple Zed task to run just dev to do the build. Now for the tricky part: running the application.


If I were doing web development (unfortunately a reality for the day job) this would simply be a matter of pointing my browser to the desktop’s address. However launching a GUI application on the remote desktop doesn’t help much if I can’t see the display. Enter waypipe: a project most easily described as X forwarding for Wayland. With Waypipe, one can launch a remote application over SSH like so:

waypipe ssh user@remote gnome-calculator

And it will forward the window to the local Wayland display! It’s fast and responsive and acts like a normal application window. Much more convenient than a full remote-desktop session.

Now with a single command in Zed (alt + t) I can compile and launch the application and the window appears on my laptop. It’s the same exact workflow I have on my desktop, but on my laptop, but actually it’s on the desktop still! Loving this setup.

Here’s the code for the just recipe:

dev-remote host:
    #!/usr/bin/env bash
    set -euo pipefail
    echo "Building locally..."
    cargo build
    echo "Launching on remote display..."
    DEV_HOST=$(hostname -f)
    BINARY_PATH="{{justfile_directory()}}/target/debug/gelly"
    WAYLAND_DISPLAY_VAR="${WAYLAND_DISPLAY:-wayland-0}"
    ssh {{host}} "WAYLAND_DISPLAY=$WAYLAND_DISPLAY_VAR RUST_LOG='debug,glycin=off,glycin_utils=off' waypipe -n ssh $DEV_HOST $BINARY_PATH"

The 7 Greatest Desktop Environments and Window Managers of All Time

&& [ linux ] && 0 comments

Linux is all about freedom, customization and choice. There are many people who will tell you this, and then also tell you what the best choice is and why you should definitely agree with them.

Following in that tradition, I’m going to share what I think are the seven best window managers and Desktop Environments (DEs) of all time. This is not about which one you should be using now but instead which ones you should have been using back then if you were an old geezer like me.

7. GNOME 2.x

Description

GNOME. Old faithful. It’s never been the most exciting choice, but it’s always been there. Many will say the GNOME desktop peaked in the late 2.x days when you didn’t need an extension for tray applets (they still aren’t dead!) and you could have actual themes. It was a solid desktop and it lives on as the MATE project.

6. Enlightenment

Enlightenment was such a delightfully weird desktop with the best eye candy at the time. It seemed to attract the most creative theme developers and had some pretty cool/bespoke UI patterns. If I remember correctly, it was even mostly usable.

Description

While most graphical software for Linux builds on top of either QT or GTK, Enlightenment stands out for having it’s own set of libraries called the Enlightenment Foundation Libraries which definitely helped it stand out from everything else at the time.

5. Awesome

Awesome WM was the first tiling window manager that I was able to use for an extended period of time. It started as a fork of DWM, but with sane defaults so that noobs didn’t have to struggle for weeks before being able to use their computers. In many ways it was the Hyprland of yesteryear. Also, you could configure it with Lua.

Description

I remember loving the super+space shortcut to cycle through window layouts and the cool icon that would change with them. Something that still no other tiling WM has been able to replicate.

4. Hyprland

Description

Hyprland is the AwesomeWM for the Pewdiepie generation. There’s nothing I can say here that hasn’t already been said and reacted to on Youtube. Try it out if you haven’t, despite the hype it is actually pretty good.

3. KDE 3.x

KDE 3.x was the perfect desktop for those escaping Windows. Familiar from the UI/UX, but configurable. It was still busy and quirky like KDE still is to this day but somehow much more cohesive than later versions ever managed to become.

Description

It still comes to mind as the de-facto Linux experience of the early 2000s. It’s even the desktop Zuck was portrayed as using to write Facebook in the movie The Social Network.

2. Fluxbox

Fluxbox (and openbox) have unfairly been forgotten to time. The boxes were a safe place to go if you wanted something a little edgier than GNOME/KDE but didn’t want to give up your mouse.

Description

Fluxbox had great themes and even provided niceties such as (gasp) a window list and dock.

It was the desktop for script kiddies.

1. i3

Tiling window managers had been a thing for a while before i3, but i3 made them a thing. i3 was so good I remember there being a mini Cambrian explosion of software specifically designed to work with i3: launchers like rofi, bars like polybar, notification daemons like dunst, that themselves were forked time and time again. Now there’s a massive ecosystem around WMs like Hyprland where the lineage can be traced directly back to i3.

Description

i3 really did have a massive influence on how people think of alternative window managers on Linux and helped make the new crop of them possible.

Arch Linux is for Rust Lovers

&& [ linux, rust ] && 0 comments

Arch Linux is a great distribution for people that love Rust and Rust tools, by the way.

Description

Two days ago I re-commissioned an old Thinkpad T470s and because Fedora 42 just happened to have released that morning, I thought I’d give it a try. The installer was great. The experience installing my tools afterwards, not so much. About 30 minutes, two enabled COPR repos and one git clone and cargo install later, I was already downloading the Arch Linux ISO. It just isn’t worth it, the software selection in the official Arch repos is just too good.

I am not talking about the AUR here. Only official repos.

Here is a short list of awesome Rust programs that are a single pacman -S away:

  • bat - cat with features
  • eza - ls with icons
  • fd - find with arguments that actually make sense
  • fish - the best shell, now written in Rust
  • helix - a modal text editor we’d all be using if we weren’t already addicted to VIM keys.
  • ripgrep - grep for dummies
  • ruff - The essential linter for any Python dev
  • starship - Great 0 config shell prompt
  • uv - If you write any Python you know this
  • wezterm - A nice GPU accelerated terminal with tons of features.
  • zed - IMO the best GUI editor right now
  • zoxide - Like z but maintained

Not written in rust, but too great not to include:

  • fzf - versatile fuzzy finder
  • ghostty - amazing terminal written in Zig

Shout-out to alerque and orhun between them they seem to maintain about 95% of the aforementioned packages.

The rust package is pretty good too

While installing Rust via rustup (also available in the Arch Linux extra repo) is the recommended way to download the toolchain, I’ve found that as long as you don’t need nightly or special targets for anything the Arch rust package works really well. You can grab rust-analyzer while you’re at it. Now you can have pacman handle your rust toolchain, and not worry about running rustup.

That one Time we Snuck Into Romeo Pier

&& [ other ] && 0 comments

Old Romeo Pier was demolished in 2018.

Pier

Two decades before that a group of bored teenagers from in and around Half Moon Bay, CA decided it would be a good idea to sneak out onto the long closed to the public pier at night.

I was one of those teenagers that ventured out there in April of 2004. At that age I used to carry a camera with me at all times (according to the intact metadata it was a Casio EX-Z40).

Outside

I do remember being pretty freaked out. The walk out to the building at the end of the pier was long and extremely dangerous. The decking was rotting and full of holes that could have easily sent one of us into the harbor. And it was dark - we didn’t use flashlights for fear of being seen. Ever heard that teenagers are dumb?

To this day the inside of the building is still the creepiest place I have ever been. The floor was covered with bird shit and dust. It smelled like the ocean mixed with mildew. The sound of the water moving underneath the floor combined with chains clanking with the tide really added to the horror film ambiance. Disturbingly there was no graffiti - the presence of any would have been a comfort.

Outside

And then there was the meat hook. Really? A meat hook hanging from the ceiling? Even at the time I remember thinking that was too cliche to be real. It looks like I even took a photo of it. (Upon closer inspection in 2025 it doesn’t appear to actually be a meat hook.)

Outside

Anyway, I did some searching and really couldn’t find any photos of this place, so I thought I’d share mine. These likely aren’t representative of the Romeo Fish Company during it’s heyday, but at least it’s something.

Outside

This Historic Building Documentation Report from the county of San Mateo contains a good history of the pier.

The Technium - A Great But Little Known Podcast

&& [ code ] && 0 comments

Recently I stumbled upon the 9Front website while doing a bit of research on Plan9. This only increased my interest. Seriously, check out the 9Front website. It’s pretty wild.

But the dog needs walking, so that day I decided to do a search for Plan9 in the podcast app to see if I could keep the mood going.

That’s where I stumbled upon the Technium Podcast a show where two dudes drink something and talk about new and old tech. Not exactly a novel concept, but I was almost immediately impressed. Their episode on Plan9 was well-researched and entertaining. The level of technicality is pretty high, something that is very hard to find in podcasts. If you aren’t already a programmer, some of their shows might be difficult.

Technium

Other episodes I enjoyed were on the APL language and Smalltalk

Sadly they seem to have stopped doing the show - their last being titled “LLMs eat software development” which is a bit ominous. I looked them up on YouTube, only about 400 subscribers. They deserve many more!

View .fit Files in the Terminal With f2i

&& [ code, rust, astronmy ] && 0 comments

Ever wanted to preview astronomical .fit files directly in your terminal without the need for DS9? Neither have I, but no matter - now you can.

f2i

It’s also just a crazy fast thumbnailer. Python is great but Rust beats it out if you can get it to work for you. In this case, ndarray and cfitsio were up to the task.

Github

Messing with AI Bots for Fun with django-llm-poison

&& [ python, code, django ] && 4 comments

The internet is filling up with AI slop and you and I dear reader, are unwilling accomplices to this rapid decline. Big tech may have created the models but the models are trained on our words. Social media posts, forum rants and of course blogs like this one, all hoovered up into the LLM.

At least in some cases, we have a choice of whether to continue to feed the machine.

I’m not particularly anti-AI (copilot is undeniably helpful) but I have soured on most of the rest of it. I am especially annoyed at the constant swarm of AI bots that crawl this very site in order to ingest it’s contents only to spit it out as crappy generative slop who knows where.

My answer is a middle-finger of sorts to these bots. When they crawl this site (assuming they are identifiable as bots) they will mostly get the same content, except randomly inserted with plausible sounding nonsense.

To demonstrate view this very post with bot-mode enabled.

Bot content

I’ve packaged it up as a reusable Django app django-llm-poison so that others running Django powered websites can use it as well.

It works by generating Markov chains from the content on the site. When a bot requests content the response is the same but with every few sentences replaced by Markov nonsense (I am aware of the irony of using very primitive generative “AI” to combat current AI). In this way, the quality of training data provided by this site is greatly diminished while remaining the same for human visitors.

Of course this site is an insignificant blip in the vast sea of information, but nonetheless, I derive a certain satisfaction for this small act of techno-disobedience.