TL;DR
Ayder is a single-binary event streaming system written in C that exposes an HTTP API so curl can act as a client. It combines Raft-based replication and append-only files for durability, claims sub-4ms P99 latencies in a 3-node cluster benchmark, and reports crash recovery in about 40–50 seconds for 8M offsets.
What happened
A new open-source project called Ayder provides an HTTP-first, single-binary event log and message bus implemented in C. It exposes an HTTP API for producing, consuming and managing topics and partitions; the project’s examples show curl used directly as the client. Durability is delivered by sealed append-only files plus Raft-based replication across 3/5/7 node clusters, and the system supports consumer groups, per-partition offsets, a KV store with CAS and TTL, and stream-processing primitives (filters, aggregations and windowed joins including Avro+Protobuf cross-format joins). The repository includes Docker Compose for a quick start with Prometheus and Grafana. Benchmarks provided by the author report sustained ~50K msg/s in a 3-node sync-majority test and sub-4ms P99 client latency; a SIGKILL recovery test with 8 million offsets completed in about 40–50 seconds with no data loss.
Why it matters
- HTTP-native interface lets users interact with the broker using simple tools like curl rather than language-specific client libraries.
- Raft-based sync-majority replication aims to offer stronger durability guarantees than systems that rely on async replication.
- Single-binary, C-based design reduces external dependencies and avoids JVM/ ZooKeeper operational complexity.
- Reported fast crash recovery and compact deployment model could simplify operations and upgrades for small to medium clusters.
- ARM64 benchmark results suggest the code runs efficiently on consumer hardware as well as cloud VMs.
Key facts
- API is HTTP-based: produce, consume, commit offsets, topic management and health endpoints are exposed over port 1109 by default.
- Durability modes: 'sealed' writes append to AOF and survive crashes; 'rocket' is an in-memory fast path not persisted.
- Replication via Raft supports 3/5/7 node clusters with sync-majority writes (e.g., 2/3 acknowledgements).
- Production-like benchmark (3-node DigitalOcean, 64B payload, wrk2 at 50K req/s): throughput ~49,871 msg/s; client P99 3.46ms; server handler P99.999 1.22ms.
- Max-throughput wrk run reported up to ~93,807 msg/s; an ARM64 laptop benchmark showed ~106,645 msg/s (3-node cluster simulated).
- Crash recovery test: follower SIGKILL during writes, restart and auto catch-up produced a fully healthy cluster in ~40–50 seconds for 8M offsets with no reported data loss.
- Feature set includes consumer groups with explicit commits, per-partition append-only logs, retention controls (delete-before, TTL/max-bytes), and a KV store with CAS and TTL.
- Quick start options: clone repo and run with Docker Compose (includes Prometheus and Grafana) or build a single ayder binary. Build deps listed: libuv 1.51+, OpenSSL, zlib, liburing.
What to watch next
- Production adoption and long-term stability of the project: not confirmed in the source
- Third-party security audits and hardening for exposed HTTP endpoints: not confirmed in the source
- Availability of official client libraries or ecosystem integrations beyond curl/HTTP: not confirmed in the source
Quick glossary
- Raft: A consensus algorithm for replicated state machines that provides leader election and log replication with majority-based safety.
- Append-only file (AOF): A durable storage format where new records are appended to the end of a file to preserve write order and enable crash recovery.
- Consumer group: A set of consumers that coordinate to read partitions of a topic and track per-group committed offsets for progress and fault tolerance.
- Idempotency key: A client-supplied identifier used to detect and suppress duplicate requests so a single logical operation is applied at most once.
- Base64 encoding: A binary-to-text encoding scheme often used to transmit binary payloads safely over text-based protocols like HTTP.
Reader FAQ
Does Ayder require a JVM or ZooKeeper?
No — the project is written in C as a single binary and does not depend on JVM or ZooKeeper.
Can I use curl as a client?
Yes. The API is HTTP-native and examples in the repo show producing and consuming using curl.
How durable is data in Ayder?
Durability is provided by sealed writes appended to AOF plus Raft sync-majority replication; sealed writes are persisted and survive crashes according to the project's documentation.
Is Ayder production-ready?
not confirmed in the source
Ayder HTTP-native durable event log / message bus — written in C A single-binary event streaming system where curl is your client. No JVM, no ZooKeeper, no thick client libraries….
Sources
- Show HN: Ayder – HTTP-native durable event log written in C (curl as client)
- HTTP-native durable event log written in C (curl as client)
- Logging request/response messages when using HttpClient
- Hacker News
Related posts
- FastScheduler: Decorator-first Python task scheduler with async support
- Report: Google may launch the Pixel 10a in retail as soon as mid-February
- Mercedes pauses rollout of Drive Pilot Level 3 ‘eyes-off’ driving system