Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,554 words · 26 segments analyzed
By James Blackwood-Sewell on April 2, 2026 For most of its history, full-text search has been synonymous with one library: Apache Lucene. Written in Java and battle-tested over two decades, Lucene powers Elasticsearch, Solr, and much of the search infrastructure the industry relies on today. It's the kind of project that makes you think the problem is solved, that there's no room left to rethink the fundamentals. Then in 2017, a Rust library called Tantivy appeared on GitHub. Built by a single developer who wanted to test his understanding of search engines and his new favorite language, it was small, fast, and unapologetically modular. Within a few years it had grown into the engine behind Quickwit, ParadeDB, LNX, and a growing number of search products, and had sparked a genuine performance collaboration with the Lucene team itself.
That developer is Paul Masurel.
After creating Tantivy, Paul co-founded Quickwit in 2020 to build a cloud-native log search engine on top of it. Quickwit was acquired by Datadog in 2024, where Paul now works on search infrastructure at massive scale. We sat down with him to talk about the origins of Tantivy, the philosophy behind its design, what it's like competing (and collaborating) with Lucene, and the lessons he's learned along the way.
Origins You've spent much of your career rethinking how search infrastructure is built.
How did you first get interested in search systems?
My history with search engines started at a small French enterprise search company called Exalead. I was a front-end engineer at the time. I like to think my frustration grew from not being part of the core team. Long-fermented frustration is an underrated driver in my opinion.
Building Tantivy Paul’s frustration soon led to a job as a backend engineer for a search product at Indeed in Japan, where he worked on the core search engine itself.
But the idea of building something from scratch kept pulling at him. The catalyst came on a long-haul flight in 2016. The first version was “a bit silly” and only took a couple of months of spare time, but it was enough to prove the idea had legs.
What made you decide to build a search engine from scratch, and did Rust shape its architecture?
I read the Rust book during a flight from Tokyo to Paris in 2016. Being very familiar with C++, all of the ideas were very enticing to me. I quickly worked through the exercism.io Rust track and then wanted to test the language on a real-life project — something with IO, error management, multithreading. What project should I have picked?At the time, I was working at Indeed in the search quality team. Our search engine was based on Lucene 2.4. Building a search engine was a perfect way for me to test my understanding of search engines and try out Rust on a real-life project.Rust did impact the way the code is organized, but I wouldn't say it shaped the architecture.
Lucene is really the inspiration there.
Paul didn't set out to build a Lucene replacement. He set out to build something small enough that developers could actually own it.
He has described becoming more productive in Rust within two weeks than he had been after five years of C++, and experiencing "a degree of confidence that my code was not buggy, that I had never experienced in any other language".
Architecturally, Tantivy follows Lucene’s model: it is a library, not a server. It handles indexing, compression, and search, but leaves distribution and orchestration to whatever system embeds it.
Were there particular trade-offs or priorities you focused on?
Originally I wanted to keep the project small but modular. Batteries included, but minimal.My target was companies for which search was so central to their product that they would eventually build their own engine instead of using off-the-shelf solutions like Elasticsearch. Usually such companies would have to use Lucene — and therefore Java. I thought Tantivy could be a good alternative for them.For this reason, I wanted Tantivy to be small and modular as opposed to featureful. No matter how sophisticated Tantivy might become, these users would rather be able to plug in their own tokenizer and their own query parser.Today, Tantivy users come in all sizes and shapes. We still sometimes refuse PRs that add niche features in order to keep the library simple, and we prefer PRs that make it possible to implement those features outside of Tantivy. Generally speaking though, this principle isn't as strong as it used to be. The Benchmark Game It quickly became apparent that Tantivy was not only competitive with other search engines, in many cases it outperformed them.
Under the hood, Tantivy uses finite state transducers for its term dictionary, SIMD-accelerated compression for its inverted index, and a memory-mapped I/O layer that keeps resident memory remarkably low. The result is a library that can handle indexes larger than available RAM without breaking a sweat.
Jason Wolfe, Paul's manager at Indeed, created Search Benchmark, the Game, a reproducible benchmark suite that showed Tantivy was often 2x faster than Lucene. Over the years the benchmark sparked a genuine back-and-forth between the two projects. Adrien Grand, a Lucene committer, published "Why is Tantivy Faster than Lucene?"
and a follow-up analysis, and Lucene has since landed patches that close the gap across most areas.
Can you see this game of cat and mouse between Tantivy and Lucene continuing? Let me first talk a little bit about that benchmark. We wanted it honest and reproducible, and designed it to help us find out where Tantivy's performance was lacking.Lucene developers approached it with genuine curiosity. Adrien Grand (at Elastic at the time) and Mike McCandless (Amazon) used it to investigate how to improve Lucene's performance. We kept a channel of communication open for the benefit of both projects. Some patches inspired by Tantivy made it into Lucene. After that, Adrien kept finding new optimizations to improve Lucene's search performance. He shared his progress with us and even left tickets in Tantivy's GitHub. We're trailing behind on implementing them. Today, Lucene outperforms Tantivy in many places. Overall, both projects benefited from the collaboration.Now I'll hijack this question for a second and jump on my soapbox. Some people get the wrong idea about the nature of competition in software. Of course, Quickwit and Tantivy are projects competing with Elasticsearch and Lucene. But competition does not have to translate into a feud — especially in open source.Through Tantivy, I've interacted with many companies. A good number of them turn the "fake it until you make it" saying into a culture of pettiness and hypocrisy. I truly appreciate how special Lucene and Elastic are, and how lucky we were to compete with them.
The Open-Source Life Paul's respect for Lucene is striking, especially given how tribal the open-source search space can get. But open source comes with its own challenges. As Tantivy's adoption grew, so did the weight of maintaining it.
Maintaining an open-source project at Tantivy's scale isn't easy.
What have been the hardest challenges in guiding its growth? Reviewing contributions — and sometimes simply saying no to them — is, I think, the hardest part. I review a lot of code, both open source and proprietary. It's very difficult to decide whether a PR should be merged, should be improved, or should be dropped. A lot of contributors are excited about the idea of contributing to open source and can submit features that don't solve any actual problem they have, should be kept external to Tantivy, introduce too much complexity, or are just not of sufficient quality.Every time I review code, I'm torn between: The responsibility to not let the project drift into a terrible state Wanting to please people Quickwit's interests The nihilist's thesis: code quality is subjective, after all Managing my time and the energy required for context switching Since Quickwit's acquisition, it's even more difficult for me to find the time and energy to review PRs. Like many developers, I struggle with having to deal with several problems in parallel, and my work at Datadog is already very challenging in that regard.
Despite that tension, the ecosystem around Tantivy has grown into something Paul never designed for. The library-only architecture that he borrowed from Lucene turned out to be its greatest asset: because Tantivy handles indexing and search but stays silent on how to shard, replicate, and coordinate across nodes, downstream projects can wrap it in radically different ways. What started as a library for companies who wanted to build their own search engine is now the foundation for products with very different goals: scale-out log search, typo-tolerant lookup, edge databases, and transactional search inside Postgres.
Tantivy now serves as the foundation for Quickwit, LNX, ParadeDB, Turso, and others. Have you seen contributions or design ideas flow back upstream? Someone contributed support for geo search seemingly out of nowhere. This is something we had wanted to add for a while. The PR is large but its quality is very impressive. I hope I will eventually find time to get it merged.ParadeDB has also been actively contributing great PRs and deep ideas lately. While the story of Tantivy started with you on a plane, many others have been involved over the years.
Who else has helped shape your vision?