Jepsen: ParticleDB 1.1.4

ParticleDB is a multi-model database designed for AI workloads. We tested versions 1.1.4 through 1.1.18.11.rc-22 and found that ParticleDB’s claims of “Full ACID” were somewhat premature. We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on). Most issues required just a single server, single client, and no faults. By version 1.1.18.11.rc-22, all but ten of those issues appeared resolved. However, ParticleDB continued to lose committed writes during normal operation with the default occ transaction mode, and allowed internal anomalies and Lost Update with serializable, the strongest setting. Orkes, the makers of ParticleDB, have since released version 1.1.19-rc.13; they claim that it resolves all remaining issues discussed in this report. This research was funded by Orkes, Inc., and conducted in accordance with the Jepsen ethics policy. Background Modern applications often spread state across a variety of systems: databases, caches, queues, data warehouses, search indices, and so on. Coordinating these systems is challenging: caches are often stale, the analytics platform might lag days behind the primary database, the search index might lose data, and transactional updates across disparate systems remains an open research problem. ParticleDB1 aims to unify these workloads in a single engine with a special focus on AI agents and applications. It offers transactional SQL, a key-value store, vector search, live analytics, and user-defined functions called Particles.2 To integrate with existing systems, clients can connect through a variety of interfaces, including the Postgres and Redis protocols, as well as its own gRPC and HTTP2 APIs. Written from the ground up in Rust, ParticleDB claims to be orders of magnitude faster than Postgres while preserving the same safety characteristics. ParticleDB was built by Orkes to support their workflow orchestration platform, Conductor. Conductor was initially built to store data in Redis and Postgres, but to improve performance, Orkes opted to write their own database. ParticleDB 1.0.0 offered a limited feature set specifically for Conductor’s queries, and was deployed as the backing database for some of their Conductor clusters. A few Orkes customers followed suit. In preparation for a broader release, Orkes undertook a major refactoring of ParticleDB, adding a custom wire protocol, more datatypes, additional OLAP queries, and more. While Orkes and their customers continued to run 1.0.0, they soft launched the refactored versions on particledb.ai. Orkes engaged Jepsen to test these versions of ParticleDB, beginning with 1.1.4. While ParticleDB could run as a single node, it also offered a replicated cluster using the Raft consensus protocol. This replication feature was available only in enterprise cloud deployments. The builds Orkes provided did not support replication, so this report covers only single-node behavior. Safety ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built. ParticleDB offered three tactics for syncing its Write-Ahead Log (WAL). The sync mode called fsync on each commit, groupsync batched multiple transactions into a single fsync, and nosync skipped fsync altogether. The Transaction Engine documentation described sync as offering “stronger durability” than groupsync. On the other hand, particledb start --help said the groupsync mode ensured that “COMMIT waits for flush”, and Orkes’ internal operator guide concurred: groupsync offered the “same durability as sync”. ParticleDB’s home page proudly claimed “Full ACID. 159K TPS.” However, the text immediately below this said “snapshot isolation”, which is not fully isolated. Most SQL databases use the SET TRANSACTION ISOLATION LEVEL statement to control the concurrent semantics of a transaction. However, ParticleDB ignored or rejected these statements. In some cases it instructed users to choose READ COMMITTED, a relatively weak isolation level. In other cases it accepted requests for different isolation levels, and silently ignored them (despite documentation to the contrary). Instead, transaction isolation was governed by the --txn-mode setting passed at startup. The transaction engine documentation described four modes: Mode Behavior When to use fast No locks; commit-on-write Bulk loads, single-writer benchmarks occ Overlay + commit-time OCC validation (default) General OLTP — safe under concurrency zero-copy-occ Partition-pinned OCC; single-partition txns skip cross-partition coordination High-throughput OLTP on partition-affine workloads (TPC-C, key-value) table-2pl Exclusive table locks at first touch Strictest serialization, simple semantics This table was difficult to interpret. While the best practices documentation warned that users should “pick --txn-mode deliberately”, since “it changes correctness and throughput characteristics more than SQL text does”, the documentation did not explain what these modes actually guaranteed. The fast mode was clearly intended for weaker isolation, but was that Read Atomic, Read Uncommitted, or something else? The occ mode was similarly unclear. “Commit-time OCC validation” could describe a broad spectrum of consistency models from Read Uncommitted to Strong Serializable, but “safe under concurrency” suggests it might be Serializable. The zero-copy-occ mode described a performance optimization which could be as safe as occ, but it was not described as “safe under concurrency”—raising the question: in what ways was it unsafe? Finally, if table-2pl offered the “strictest serialization”, was occ some weaker isolation level? The CLI reference page contained another table which described occ and table-2pl as “Jepsen-safe”, whereas fast was not “Jepsen-safe”, but did not explain their safety properties. Since Jepsen evaluates safety with respect to documentation and marketing claims, it is not clear how to interpret these terms.3 Running particledb start --help provided the following table: Mode Description fast No locks, maximum OLTP throughput (unsafe under concurrency) occ Table locks + commit-time validation (safe for most OLTP) row-2pl Alias for occ zero-copy-occ Partition-pinned OCC (highest throughput safe mode) table-2pl Exclusive table locks, full serialization (strictest) serializable Alias for table-2pl serializable-ssi Adaptive SSI (Cahill 2008), PL-3 with optimistic semantics (v1 scaffolding — see SSI_ADAPTIVE_DESIGN.md) The point of Optimistic Concurrency Control (OCC) is generally to avoid taking locks, so the mention of table locks and a row-2pl4 alias was surprising. Moreover, “safe for most OLTP” undermines the claim that occ was “safe under concurrency”; it suggests there are some OLTP workloads which are unsafe! Orkes explained that this documentation was wrong: occ was actually optimistic, and the row-2pl alias was to preserve compatibility with an earlier two-phase locking mechanism. They also provided an unpublished operator guide with detailed descriptions of the various transaction modes. This document was difficult to interpret, but did specify the following consistency models: Mode Consistency Model fast Unknown occ-v3 (a.k.a. occ, occ-v2, row-2pl) Snapshot Isolation zero-copy-occ (a.k.a. zocc) Snapshot Isolation serializable-ssi (a.k.a. ssi) Serializable table-2pl (a.k.a. serializable) Strong Serializable Sources disagreed about the default transaction and WAL sync modes, but in general, occ-v3 and groupsync appeared to be the default. Test Design We wrote a test suite for ParticleDB using the Jepsen testing library. We ran ParticleDB versions 1.1.4 through 1.1.18.11.rc-22 on individual Debian 13 nodes, both in containers and EC2 VMs. Our test harness could simulate process crashes by killing ParticleDB, simulate the loss of un-fsynced writes using LazyFS, and truncate or corrupt disk files. We ran ParticleDB using each transaction mode and all three WAL sync modes. We first tried ParticleDB’s official Java SDK. While the documentation indicated that the library was available as a Maven dependency, neither the JAR nor source code were actually available. Instead, we began with the official Postgres JDBC client, version 42.7.11, and later added support for the ParticleDB Java SDK, versions 1.0.0 through 1.0.2. Workloads We used several Jepsen workloads from other SQL databases, and designed new ones specifically for ParticleDB. The rw workload performed randomly generated transactions involving reads and writes of integer values identified by unique integer keys. We stored each value in a single row and accessed them both via primary and unindexed secondary keys. Keys and values were encoded as a variety of datatypes, including integers, decimals, and strings. We assumed that writes followed reads within each transaction, and that individual keys were sequentially consistent, and used these assumptions to infer constraints on the version orders and transaction dependency graphs. Jepsen’s transactional checker, Elle, then searched for cycles in that dependency graph, as well as a variety of non-cyclic phenomena like G1a (Aborted Read). The append workload also performed randomly generated transactions over rows identified by unique integer keys, but the values were lists of integers. Transactions could read one of these lists by key, or append a unique integer to a list by key. We encoded each list as a comma-separated text field, and used SQL concat to append to them. Rows could be selected via primary key or unindexed secondary key, or via a join to an indirection table. From there, we used Elle to infer