The 2019 Vector Database Explosion: How Pinecone, Milvus, and Weaviate Changed the Game
Why 2019 was the pivotal year when vector search transitioned from research libraries to managed services.
Here's a weird thing about the vector database industry: it basically didn't exist before 2019. Sure, we had FAISS (2017) and Annoy (2013) and a handful of other libraries that could do approximate nearest neighbor search. But "vector database" as a product category? That's a 2019 invention. And understanding WHY it happened when it happened tells you a lot about where we are today.
I've been building vector search systems since before they were cool—back when you had to explain to everyone what an "embedding" was and why you'd want to store a million of them. So watching the 2019 explosion happen in real-time was fascinating. Three companies—Pinecone, Milvus, and Weaviate—made roughly the same bet at roughly the same time. That's not a coincidence. That's a signal.
The World Before 2019: Libraries, Not Databases
Let me paint you a picture of what building a vector search system looked like in, say, 2018. You had FAISS, which was amazing for what it did—Facebook's research team had figured out how to search through billions of vectors faster than anyone thought possible. But FAISS was a library, not a database. The difference matters.
With a library, YOU are responsible for everything. Storing the vectors? Your problem. Loading them into memory? Your problem. Keeping them synchronized across machines? Definitely your problem. Handling updates without rebuilding the entire index? Ha! Good luck with that. FAISS didn't even have native update operations until much later—and even now, updates are basically "delete everything and start over."
The 2018 Vector Search Stack (DIY Edition)
The cognitive overhead was immense. You weren't just building a feature—you were building infrastructure. And if you wanted to scale beyond a single machine? Or handle real-time updates? Or survive a server failure? Each requirement added weeks of engineering work. Most teams just... didn't. They kept their vector search small and static, because the alternative was building a distributed database from scratch.
2019: The Year Everything Changed
So what happened in 2019? Why did three separate teams independently decide "this is the year to build a vector database as a service"? A few forces converged:
1. Embeddings Went Mainstream
BERT dropped in late 2018, and suddenly everyone could generate useful embeddings without a PhD in NLP. Before BERT, getting good text embeddings required training custom models on massive datasets. After BERT, you could download a pre-trained model and get state-of-the-art sentence embeddings in an afternoon. The demand side exploded.
Think about what this means. In 2017, the only people generating millions of embeddings were researchers at big tech companies. By 2019, any startup could spin up a semantic search feature. The market went from hundreds of potential customers to thousands basically overnight.
2. Cloud Infrastructure Matured
Kubernetes hit 1.0 in 2015, but by 2019 it was actually usable for mortals. EKS, GKE, AKS—the managed Kubernetes services made it feasible to run stateful workloads in the cloud without a dedicated infrastructure team. Building a distributed database was still hard, but building a distributed database that could run on any cloud? That became possible.
The cloud providers also got much better at high-performance networking and storage. NVMe SSDs became the default. Network latencies dropped. The raw infrastructure to build a fast, distributed vector database finally existed.
3. The Business Model Made Sense
Here's the thing about database companies: they're insanely valuable. MongoDB, Snowflake, Databricks—multi-billion dollar companies, all of them. Investors in 2019 understood the database playbook. Build an open-source or managed database for a new workload, capture market share while the workload grows, and monetize through cloud services.
Vector search in 2019 looked exactly like document databases in 2009 or time-series databases in 2014. New workload, growing fast, terrible tooling, clear monetization path. The opportunity was obvious to anyone paying attention.
The Three Pioneers: Different Bets on the Same Wave
What's fascinating is how differently these three companies approached the same opportunity. They all saw "vector database" as a category, but their implementations and strategies diverged significantly.
Pinecone: The Managed-First Approach
Pinecone, founded by Edo Liberty (who ran Amazon's AI Labs) in early 2019, made a radical bet: fully managed from day one. No open-source core. No "download and run yourself" option. Pure cloud service.
The reasoning was straightforward: if the value proposition is "don't think about infrastructure," then why give people infrastructure to think about? Liberty had seen at Amazon how much friction the open-source-to-cloud path created. By going managed-only, Pinecone could optimize the entire stack without worrying about on-prem compatibility.
The trade-off? Lock-in. You can't migrate away from Pinecone without rewriting your integration. For many teams, that trade-off was worth it. For others—especially in regulated industries—it was a non-starter.
Milvus: The Open Source Play
Milvus took the opposite approach. Open-source from the start, Apache 2.0 licensed, designed for self-hosting. Founded by Zilliz in 2019, Milvus bet on the MongoDB playbook: capture developer mindshare with open source, build enterprise features on top, monetize through a cloud service (Zilliz Cloud) later.
The technical architecture reflected this. Milvus was built as a cloud-native database from day one—disaggregated storage and compute, Kubernetes-native deployment, horizontal scaling. You could run it on your laptop for development and scale to hundreds of nodes in production.
The trade-off? Complexity. Running Milvus in production requires understanding distributed systems. The flexibility comes at the cost of operational overhead. But for companies that needed data sovereignty or custom deployments, Milvus was the only game in town.
Weaviate: The Semantic-First Approach
Weaviate, started in the Netherlands in 2019, made a different bet entirely. Instead of positioning as "vector database," they positioned as "semantic search platform." The difference is subtle but important.
While Pinecone and Milvus focused on the storage and retrieval layer—bring your own embeddings, we'll index them—Weaviate integrated the entire pipeline. Built-in modules for text2vec, img2vec, various embedding models. GraphQL API with semantic queries. The pitch wasn't "store your vectors" but "build semantic applications."
The trade-off? More opinionated. If Weaviate's semantic model matches your use case, it's faster to get started. If you need custom embeddings or unusual workflows, you're fighting the platform. The blessing and curse of integration.
What They All Got Right
Despite their differences, these three companies shared some fundamental insights that separated them from the library-based approach:
Real-Time Updates
All three supported adding and updating vectors without rebuilding indexes. This sounds basic, but it was revolutionary compared to FAISS's append-only model. Your data could be live, not static.
Metadata Filtering
Vectors alone aren't useful. You need to filter by category, timestamp, user ID. All three understood that vector search needed to compose with traditional filtering. "Give me similar items from the last week" is the real query, not just "give me similar items."
Horizontal Scaling
FAISS on a single machine can handle tens of millions of vectors. But when you need hundreds of millions? Billions? You need distributed search. All three designed for sharding and replication from day one.
Standard APIs
REST APIs, gRPC, language-specific SDKs. No more writing custom wrappers around C++ libraries. No more wrestling with Python bindings. Just make HTTP calls and get results.
The Aftermath: A Cambrian Explosion
The success of these three companies in 2019-2020 triggered a wave of imitators. By 2023, you could count dozens of vector database offerings: Chroma, Qdrant, Vespa (repositioned), LanceDB, Marqo, Vald, and more. Every database vendor added vector capabilities—PostgreSQL got pgvector, Redis got vector search, Elasticsearch added dense vectors.
The category went from "niche" to "everyone needs one" in about three years. ChatGPT's launch in late 2022 was rocket fuel—suddenly every company wanted to build AI applications, and vector databases were the assumed infrastructure layer.
"2019 was when we stopped explaining what vector databases were and started explaining why ours was better. The education phase ended. The competition phase began."
What They Missed (And We're Fixing)
Here's where I get to complain about things. Because while these companies did incredible work creating the category, they also made some choices that... let's call them "interesting."
1. Performance Left on the Table
Most vector databases use basic HNSW or IVF implementations. The index structures are well-understood, but the implementations are often... suboptimal. OpenMP parallelism where TBB would be 2-3x faster. Naive I/O patterns that trash the cache. Memory allocations in hot paths.
Our FAISS Extended work showed that you can get 2-5x speedups just by fixing the parallelism and I/O patterns. The algorithms are fine. The engineering is where the performance is hiding.
2. Update/Delete is Still Painful
Real-time updates exist, but they're often slow or come with caveats. "Updates are reflected within 10 seconds." "Deletes require compaction." "High update rates impact query performance."
Compare that to a traditional database where updates are immediate and atomic. We've gotten so used to the limitations that we forget they're limitations. FAISS Extended's in-place updates show that fast, atomic vector updates are achievable—we just have to actually build them.
3. Hybrid Search is an Afterthought
Everyone now supports "hybrid search"—combining vector similarity with keyword matching. But it's usually implemented as two separate searches merged at the end. Query the vector index, query the keyword index, merge results, hope for the best.
True hybrid search—where vector and keyword signals are combined during the search, not after—is still rare. Our MLGraph architecture does this natively because we designed for it from day one. The difference in result quality is significant.
What This Means for 2025 and Beyond
The vector database market is maturing. The "can we build one?" question is answered. Now it's "which one is best for my use case?" And increasingly, the answer isn't a standalone vector database at all—it's vector capabilities embedded in existing infrastructure.
PostgreSQL with pgvector. DynamoDB with vector support. Supabase. PlanetScale. The established databases are adding vectors, and for many use cases, that's good enough. The pure-play vector databases need to offer something the extensions can't—whether that's better performance, better developer experience, or specialized capabilities.
That's why we focus on the hard problems. Not "can you store vectors?" but "can you search 50 million vectors in under 10 milliseconds while streaming updates at 10K/second?" Not "can you filter by metadata?" but "can you combine semantic search with code AST analysis and BM25 ranking in a single query?"
2019 created the category. The next five years will define who survives it.
Building on the Giants' Shoulders
We built FAISS Extended and MLGraph because we needed capabilities the 2019 generation didn't provide. Real-time updates. TBB parallelism. Hybrid search with AST awareness. If you're hitting the same limitations, let's talk.
Further Reading
Written by David Gornshtein, CTO at WEB WISE HOUSE LTD. David has been building vector search systems since before they had a marketing department. He remembers when "similarity search" meant brute-force distance calculations, and he's only a little bitter about how easy everyone else has it now.