I have been working on Marble's webhook system again.
The first version shipped sometime last year. The goal was simple: when something happened in a workspace, Marble should be able to tell another system about it. A post gets published, a category changes, media gets deleted, and a webhook can carry that event somewhere else.
We built that first version with QStash from Upstash. They had sponsored Marble, and QStash was genuinely a good way to ship the first version. It gave us a queue-like HTTP delivery system without having to build one ourselves. Upstash also published a detailed post about that setup here: Building a webhook system with QStash.
That version was good enough to get webhooks into the product. But eventually I ran into the part that was not good enough: when a webhook did not arrive, I could not show enough evidence of what happened.
That sounds like a minor issue until a user asks about it.
If someone expects a webhook and it does not show up, the useful questions are not abstract. They are very specific:
Did Marble create the event?
Did Marble try to send it?
Which URL did we send it to?
What status code did the endpoint return?
Did the request time out?
Was it retried?
Can the user see any of this without asking me?
The uncomfortable answer was: not well enough.
Some parts of the old dashboard flow were basically fire-and-forget. The webhook might have worked, or something might have failed before it got far enough to leave a useful trail. Either way, the product did not give the user much to inspect. That is a bad feeling because webhooks are one of those features where trust comes from receipts. It is not enough to say, "we sent it." You need to be able to show the delivery.
So I moved the system.
Marble already runs its API on Cloudflare Workers, and the MCP server is there too, so I added a separate jobs worker and moved webhook processing into Cloudflare Queues.
The new shape is roughly:
Now Marble stores the event, creates delivery rows, records attempts, retries failures, signs the payload, and keeps enough state for the dashboard to show what happened.
The system has a few guardrails to keep delivery predictable:
If an event is retried, Marble reuses the existing delivery for that endpoint instead of creating the same delivery twice.
Before sending, a worker marks the delivery as
sending. If another worker sees the same delivery after that, it skips it instead of sending the webhook twice.Each webhook request has a timeout, so one slow endpoint cannot keep a worker waiting forever.
Each attempt is recorded, including successful responses, failed responses, timeouts, and other network errors.
None of this is especially novel. But it changed how the feature feels to me. The system now leaves evidence behind.
I am writing a more detailed breakdown on the Marble Blog about the actual migration, the queue setup, and how the new worker pipeline works. This note is mostly about the product lesson I took from it.
That a feature can technically work and still feel unreliable if users cannot see what happened.