I have a small set of replicas of three mongod servers (16 GB of RAM, at least 4 processor cores and real hard drives) and one dedicated arbiter. Replicated data currently contains about 100 million records. Almost all of this data is in the same collection with an index on _id(autogenerated Mongo ID) and date, which is the native moment of the Mongo date. Periodically, I delete old entries from this collection using a date index, something like this (from the mongo shell):
db.repo.remove({"date" : {"$lt" : new Date(1362096000000)}})
It works, but it works very slowly. One of my nodes has slower I / O than the other two, having only one SATA drive. When this node is the main one, deletion is performed at a speed of about 5-10 documents / sec. Using rs.stepDown (), I downgraded this slower primary level and made the choices get primary with better I / O. On this server I get about 100 documents / sec.
My main question is: should I be worried? I don't have numbers before I introduced replication, but I know that deletion was much faster. I am wondering if synchronization of a replica set causes I / O to wait, or some other reason occurs. I would be completely pleased to temporarily disable synchronization updates and indexes until the delete statement ends, but I don't know how to do this at this time. For some reason, when I disconnected two of the three nodes, leaving only one node and the arbiter, the remaining node will be demoted and entries are impossible (shouldn't the arbiter solve this?).
To give you some idea of overall performance, if I drop and recreate the date index, it takes about 15 minutes to scan all 100M documents.