Firat Atagun

UPDATE: Turns out that the following post is invalid/inaccurate.

Don’t use MongoDB
=================

I’ve kept quiet for awhile for various political reasons, but I now
feel a kind of social responsibility to deter people from banking
their business on MongoDB.

Our team did serious load on MongoDB on a large (10s of millions
of users, high profile company) userbase, expecting, from early good
experiences, that the long-term scalability benefits touted by 10gen
would pan out. We were wrong, and this rant serves to deter you
from believing those benefits and making the same mistake
we did. If one person avoid the trap, it will have been
worth writing. Hopefully, many more do.

Note that, in our experiences with 10gen, they were nearly always
helpful and cordial, and often extremely so. But at the same
time, that cannot be reason alone to supress information about
the failings of their product.

Why this matters
—————-

Databases must be right, or as-right-as-possible, b/c database
mistakes are so much more severe than almost every other variation
of mistake. Not only does it have the largest impact on uptime,
performance, expense, and value (the inherit value of the data),
but data has *inertia*. Migrating TBs of data on-the-fly is
a massive undertaking compared to changing drcses or fixing the
average logic error in your code. Recovering TBs of data while
down, limited by what spindles can do for you, is a helpless
feeling.

Databases are also complex systems that are effectively black
boxes to the end developer. By adopting a database system,
you place absolute trust in their ability to do the right thing
with your data to keep it consistent and available.

Why is MongoDB popular?
———————–

To be fair, it must be acknowledged that MongoDB is popular,
and that there are valid reasons for its popularity.

* It is remarkably easy to get running
* Schema-free models that map to JSON-like structures
have great appeal to developers (they fit our brains),
and a developer is almost always the individual who
makes the platform decisions when a project is in
its infancy
* Maturity and robustness, track record, tested real-world
use cases, etc, are typically more important to sysadmin
types or operations specialists, who often inherit the
platform long after the initial decisions are made
* Its single-system, low concurrency read performance benchmarks
are impressive, and for the inexperienced evaluator, this
is often The Most Important Thing

Now, if you’re writing a toy site, or a prototype, something
where developer productivity trumps all other considerations,
it basically doesn’t matter *what* you use. Use whatever
gets the job done.

But if you’re intending to really run a large scale system
on Mongo, one that a business might depend on, simply put:

Don’t.

Why not?
——–

**1. MongoDB issues writes in unsafe ways *by default* in order to
win benchmarks**

If you don’t issue getLastError(), MongoDB doesn’t wait for any
confirmation from the database that the command was processed.
This introduces at least two classes of problems:

* In a concurrent environment (connection pools, etc), you may
have a subsequent read fail after a write has “finished”;
there is no barrier condition to know at what point the
database will recognize a write commitment
* Any unknown number of save operations can be dropped on the floor
due to queueing in various places, things outstanding in the TCP
buffer, etc, when your connection drops of the db were to be KILL’d or
segfault, hardware crash, you name it

**2. MongoDB can lose data in many startling ways**

Here is a list of ways we personally experienced records go missing:

1. They just disappeared sometimes. Cause unknown.
2. Recovery on corrupt database was not successful,
pre transaction log.
3. Replication between master and slave had *gaps* in the oplogs,
causing slaves to be missing records the master had. Yes,
there is no checksum, and yes, the replication status had the
slaves current
4. Replication just stops sometimes, without error. Monitor
your replication status!

**3. MongoDB requires a global write lock to issue any write**

Under a write-heavy load, this will kill you. If you run a blog,
you maybe don’t care b/c your R:W ratio is so high.

**4. MongoDB’s sharding doesn’t work that well under load**

Adding a shard under heavy load is a nightmare.
Mongo either moves chunks between shards so quickly it DOSes
the production traffic, or refuses to more chunks altogether.

This pretty much makes it a non-starter for high-traffic
sites with heavy write volume.

**5. mongos is unreliable**

The mongod/config server/mongos architecture is actually pretty
reasonable and clever. Unfortunately, mongos is complete
garbage. Under load, it crashed anywhere from every few hours
to every few days. Restart supervision didn’t always help b/c
sometimes it would throw some assertion that would bail out a
critical thread, but the process would stay running. Double
fail.

It got so bad the only usable way we found to run mongos was
to run haproxy in front of dozens of mongos instances, and
to have a job that slowly rotated through them and killed them
to keep fresh/live ones in the pool. No joke.

**6. MongoDB actually once deleted the entire dataset**

MongoDB, 1.6, in replica set configuration, would sometimes
determine the wrong node (often an empty node) was the freshest
copy of the data available. It would then DELETE ALL THE DATA
ON THE REPLICA (which may have been the 700GB of good data)
AND REPLICATE THE EMPTY SET. The database should never never
never do this. Faced with a situation like that, the database
should throw an error and make the admin disambiguate by
wiping/resetting data, or forcing the correct configuration.
NEVER DELETE ALL THE DATA. (This was a bad day.)

They fixed this in 1.8, thank god.

**7. Things were shipped that should have never been shipped**

Things with known, embarrassing bugs that could cause data
problems were in “stable” releases–and often we weren’t told
about these issues until after they bit us, and then only b/c
we had a super duper crazy platinum support contract with 10gen.

The response was to send up a hot patch and that they were
calling an RC internally, and then run that on our data.

**8. Replication was lackluster on busy servers**

Replication would often, again, either DOS the master, or
replicate so slowly that it would take far too long and
the oplog would be exhausted (even with a 50G oplog).

We had a busy, large dataset that we simply could
not replicate b/c of this dynamic. It was a harrowing month
or two of finger crossing before we got it onto a different
database system.

**But, the real problem:**

You might object, my information is out of date; they’ve
fixed these problems or intend to fix them in the next version;
problem X can be mitigated by optional practice Y.

Unfortunately, it doesn’t matter.

The real problem is that so many of these problems existed
in the first place.

Database developers must be held to a higher standard than
your average developer. Namely, your priority list should
typically be something like:

1. Don’t lose data, be very deterministic with data
2. Employ practices to stay available
3. Multi-node scalability
4. Minimize latency at 99% and 95%
5. Raw req/s per resource

10gen’s order seems to be, #5, then everything else in some
order. #1 ain’t in the top 3.

These failings, and the implied priorities of the company,
indicate a basic cultural problem, irrespective of whatever
problems exist in any single release: a lack of the requisite
discipline to design database systems businesses should bet on.

Please take this warning seriously.

We have vertical scalability that we throw bigger, powerful boxes to the problem and pray that it scales.

When we don’t have any more capacity for a single big box, we need another of single big box, and then we have to worry about consistency, latency, and replication, fault tolerance and so forth. We want to optimize our replication and communication, so we want to disable logging, journals just to improve the performance, which are not desirable at all. Sooner, we will run into many problems with managing these boxes.

Then we come to a point where we want to take off some load from the database, enter distributed caching. Caching doesn’t fully solve our problems because we are still dealing with Relational data model, joins, schema changes, normalization and queries.

We come to a point in which we want our application and data storages to be massively scalable, fault tolerant and consistent. While trying to achieve all these properties, we face with several problems.

For the last two decades RDMB became very popular for several reasons, simplicity being the major one. With the development of powerful SQL (Structured Query Language), relational databases become the center of database systems. With SQL, everyone was able to use data manipulation and data definition so easily. And SQL became ANSI standard.

Enter transactions. For RDMS with SQL there was a need for transactions and in my opinion transactions is a very important property for a data store to be aware of states. With transactions, most people will think about commit, rollback and ACID terms. Commit is if everything goes well with an operation then we can safely store the data in the storage, if something fails and we have an inconsistent state, we would like to rollback in order to avoid inconsistent state of data. ACID is rather a large topic. We are talking about Atomicity, Consistency, Isolation, and Durability of the state. Atomic means, all or nothing, so we either commit the data or we don’t. Consistent means, the data should be in one and only one state. It cannot be in multiple states, users of this data cannot see multiple states or version of the data. Isolated means, one and only one client of the data can operate on the data. You can’t have multiple clients working on the data at the same exact time, which would lead the data to be in a inconsistent state. Every operation on the data should be isolated from each other. Durable means that once the transaction is committed, the data would be in the same state as long as it is not modified by another transaction. Being said about transactions and ACID properties, it is rather easy to implemented transactions on single application compared to distributed systems. When you want to implement transactions across distributed system, then you have to consider the whole system, which introduces transaction managers, synchronizations etc. Distributed transactions introduce several complexities, and fault tolerance should be implemented very carefully.

In order to scale database, Sharding is introduced which is dividing up the data into meaningful clusters based on common id. For example, by initial letter of a person, which you can have 26 servers or so from A-Z, you can distribute your data accordingly to all the servers. Sharding is like partitioning your data based on a common key. Selecting a common key is very important. There are also other ways of sharding such as feature or functional sharding, in which you shard your databases based on functionality and feature. For example, you can store user data in one database and in another database you can store some product data etc. Moreover, there is key based sharding as I described above, these can be tweaked as for what you need. Hashing also can be used in this scheme. Another way to shard is to use a Look up tables, hash table, or dictionary, disadvantage of this (if there is a single lookup table), this table would be the bottleneck, and single point of failure. However, these days there are several fault tolerant distributed hash tables.

Sharding introduced shared nothing architectures, with sharding the entities within the shards are not aware of each other, and operate independently. Shared nothing architecture becomes very popular in the last few years or so. With shared nothing there is no dependency between entities of your system. There is no central control or master nodes. All of the nodes are same.

Cassandra is a decentralized, distributed, fault tolerant, elastically scalable, tunable consistent and column oriented database. It was designed based on Amazon Dynamo and Google Big Table models. Cassandra is decentralized and distributed so that client of the Cassandra is not aware of anything. So Cassandra acts as a single entity if it is distributed or not. There is no central authority, no master nodes. This is a great benefit in terms of failures, so there is no single point of failure, if one of your Cassandra server dies, the whole system keeps performing as if nothing happened. This is often called server symmetry, all the nodes are symmetric. Because Cassandra nodes are identical and there is no central authority this greatly increases availability due to decentralized model.

High availability and fault tolerance is a must have aspect of a distributed system, whatever is happening at the system, it should be seamless to the clients of the system. High availability is satisfying the requests of the clients. Moreover, in daily computing, you face with several problems from a system bug, to a failure to system/hardware failure; it is really very hard to know when there will be any kind of problem within a system. These problems should be considered and handled very carefully. Under any kind of errors, the system should still be available and satisfy its job.

Consistency is one of the most important parts of any distributed systems. While working with big data and distributed data stores, you will have to deal with consistency. There are flavors of consistency such as strict consistency, casual consistency and eventual consistency. Strict consistency is when you want to get the most recent data from any node of your distributed system. This is great! However, you will have to synchronize your data across all the servers before you can actually satisfy the request. In this case you have a problem with latency, and synchronization of you data across your cluster. So there is a trade off. Strict consistency is mostly used with financial applications and systems. When your data is very important you most likely want to go with this consistency level. Moreover, synchronization across all the nodes of your system is another hard problem you will have to face. Strict consistency uses global clock across your distributed system and works based on the global clock/timestamp. Latency makes this problem really hard. Casual consistency on the other hand offers weaker consistency. We are still trying for consistent data across our distributed system but this time instead of timestamp and global clock, we are interested in the events. So we act upon the occurrence of the events, rather than global clock. Eventual consistency, dictates that the data will propagate across all the servers at some point, we don’t know when, but will do. This is the weakest type of consistency. Basically, you write some data to a data store and it will be available in all the nodes after some time.

CAP theorem! According to CAP theorem you can only implement two of Consistency, availability and partial tolerance. You cannot have all three properties at the same time in a system. You can either implement Consistency, Availability or Consistency, Partial Tolerance or Availability and Partial Tolerance. This theorem has been proven by Nancy Lynch et al. At the very center of this theorem there is data replication and high availability and consistency comes in when you want to replicate your data across the nodes of your system. Consistency is relative to replication factor, in Cassandra consistency is tunable consistency, which means you can configure your instances in such a way that number of nodes that your data will replicate will be configurable. You can increase or decrease your consistency level and eventually data will be in consistent state across your servers. If you configure your data to be consistent across all your nodes than you will suffer from high availability, because the data will be written to all the nodes in system, it won’t be available right away. If you configure your data replication to be consistent in a small set of nodes, than you will have higher availability. I really like the fact that you configure consistency level with Cassandra. This is a great benefit.

This article (“Starbucks doesn’t use two phase commit“), is really a great to two phase commit.