Safer MongoDB Archiving with mongo-archiver — Part 2: Install and Run Your First Safe Archive

Series: Part 2 of 4 · Part 1: Why Your MongoDB Needs an Archive Strategy

In Part 1, we talked about why keeping every MongoDB document in your primary database forever isn’t always a great idea, now let’s actually archive some data.

The goal of this part is simple: we should be able to install mongo-archiver, make sure it can connect to our running MongoDB server/cluster (if hosted on atlas), see the actual data we intend to archive and then copy those documents to an archive database.

Like the last part said “copy”, we’re not deleting anything from the source (primary) database yet, so if something goes wrong, the original documents are still exactly where they were. That’s the approach I recommend when you’re trying an archiving process for the first time.

Requirements

You’ll need:

• Node.js 20 or recent version
• A running MongoDB server you can write to (a local MongoDB instance is perfectly fine)
• A collection containing documents with a createdAt field (of course timestamps are very important).
• A destination database for the archived documents running on your local MongoDB instance

For this walkthrough, I’ll use:

Source DB: source_db
Archive DB: archive_db
Collection: transactions

Obviously, you can replace these with your own database and collection names.

Step 1: Install mongo-archiver

If you’re installing the published package, it’s just:

npm install -g mongo-archiver

Then check that it was installed:

mongo-archiver --version
Ouput: 1.0.2

You should see the version of the package you installed.

If you’d rather run the project from source:

git clone https://github.com/Olusoladeboy/mongo-archiver.git

cd mongo-archiver

npm install

npm run build

npm link

Once that’s done, mongo-archiver should be available from your terminal.

Step 2: Make sure everything is reachable

Before we start moving documents around, let’s make sure MongoDB is actually reachable.

That’s what doctor is for.

mongo-archiver doctor \
  --source-uri mongodb://localhost:27017 \
  --source-db source_db \
  --dest-uri mongodb://localhost:27017 \
  --dest-db archive_db

The command checks the things we care about before starting a job: if we can connect to both source and destination database, also if we can create and use the local job-state directory. By default, that job-state directory is .mongo-archiver, which is usually in your current working directory

Ideally, you’ll get a successful check for all three but if this step fails, you should stop here and fix the connection first, there’s no point starting an archive job when the credentials, URI, permissions, or network connection aren’t working.

Step 3: Do a dry run first

This is probably the most important thing in the whole process, we shoudn’t be guessing how many documents we are about to archive, hence we should always do a dry-run.

mongo-archiver dry-run \
  --source-uri mongodb://localhost:27017 \
  --source-db source_db \
  --dest-uri mongodb://localhost:27017 \
  --dest-db archive_db \
  --collection transactions \
  --older-than 180d

After doing this, nothing actually gets copied or deleted, the command simply looks at the source collection and tells you what would happen. The output includes the number of documents that match and the MongoDB filter that was resolved from your arguments.

For example, –older-than 180d effectively produces a filter along these lines:

{
  "createdAt": {
    "$lt": "<cutoff-date>"
  }
}

What does –older-than actually mean?

It’s a shortcut for finding older documents. Some examples includes:

180d   180 days
12h    12 hours
30m    30 minutes

So –older-than 180d means: find documents where createdAt is older than 180 days and there’s one important catch, your documents need to actually have a createdAt field containing a MongoDB ISO date.

For example:

{
  "_id": "...",
  "amount": 5000,
  "createdAt": "2025-01-10T12:30:00.000Z"
}

If your application uses something different, such as created_at or timestamp, don't force –older-than to fit, you can provide your own MongoDB query instead but we’ll go into custom queries in Part 3.

For now, if your documents have createdAt, –older-than is the easiest way to get started.

Step 4: Run your first archive

If the dry run looks correct, we’re ready to actually copy the documents.

Run:

mongo-archiver archive \
  --source-uri mongodb://localhost:27017 \
  --source-db source_db \
  --dest-uri mongodb://localhost:27017 \
  --dest-db archive_db \
  --collection transactions \
  --destination-collection transactions \
  --older-than 180d \
  --batch-size 50

This is the first command in this tutorial that actually writes data, but there’s something missing –delete-source, because we aren’t deleting anything yet.

The archive process will:

1. Find the documents matching the filter.
2. Read them in batches.
3. Write each batch to the destination.
4. Preserve the original _id( for data consistency).
5. Verify the transferred batch.
6. Leave the source documents untouched.
7. Continue until all matching documents have been processed.
8. Write a report for the job.

We’re using a batch size of 50 here simply because it's a nice size for a local demonstration, the default is 500, which may be more appropriate for larger workloads.

Why preserving _id matters

One thing worth calling out is the _id , because archived documents should keep their original MongoDB _id. That gives us an important property when running the same archive again, the destination writes are performed as upserts using the document’s _id.

So if this document already exists in the archive:

{
  "_id": "abc123",
  "amount": 5000,
  .....
}

Running the same archive again shouldn’t create another copy with a new _id. Instead, the existing document is updated and that’s useful when an archive job needs to be retried.

Step 5: Check the archive report

When the job finishes, you’ll get a report similar to:

Archive Report
────────────────────────────────────────

Collection:     transactions
Job ID:         job-2026-08-06T10-15-00-123Z-ab12cd34

Status:         SUCCESS

Archived:       90 documents
Transferred:    1.2 MB
Time:           2s
Average:        45 docs/sec

Verification:   Passed
Deleted:        No

Errors:         0
Retries:        0

────────────────────────────────────────

The exact numbers will obviously depend on your data, there are a few things I always pay attention to here.

Status: You want SUCCESS

Archived: This tells you how many documents were processed, compare this with your dry-run result.

Verification: You want it to be PASSED. The archive isn’t considered successful simply because MongoDB accepted the writes, the transferred data is checked as part of the process.

Deleted: For this tutorial, you should see Deleted: No which is exactly what we want, because we didn’t include the –delete-source flag.

And finally, keep the Job ID, we’ll use it later when working with verification, restore, and resume operations.

Step 6: Check MongoDB yourself

I always recommend doing a quick manual check after an archive. Open mongo shell with the command mongosh then check the source:

mongosh
use source_db
db.transactions.countDocuments()

And then check the archive:

use archive_db
db.transactions.countDocuments()
db.transactions.findOne()

For a fresh archive database, you should see the number of archived documents reflected in the destination, more importantly, the source count should still be unchanged, that’s because we haven’t enabled deletion.

You can also compare a document from the source and archive to make sure the _id and fields were preserved.

Step 7: See the job you’ve just created

mongo-archiver keeps information about archive jobs locally.

Run:

mongo-archiver list-jobs

You’ll find the job state under: .mongo-archiver/jobs/<jobId>. The job directory contains files such as:

manifest.json
checkpoint.json

manifest.json contains the information about the completed job, including things like the filter, counts, status, timings, and verification information while checkpoint.json tracks the progress needed to resume a job if necessary.

This local state is one of the reasons you shouldn’t casually delete the .mongo-archiver directory after running an archive.

A few options worht knowing about

There are other options or flags you might want to check out:

–batch-size : defaults to 500, you can lower it if you're on a smaller machine or working against a busy cluster (smaller batches mean less load per cycle).

–verify-sample-size : defaults to 10, you can bump this up if you want more documents checked during verification.

–destination-collection : defaults to the same name as your source collection, you can change it if you want the archive to live under a different collection name.

–job-state-dir : defaults to .mongo-archiver. In production, point this somewhere more durable so job state doesn't disappear if the local directory gets wiped.

A few problems you might run into:

“Missing required configuration”: This usually means one of the required values wasn’t supplied. Check that you’ve provided the source URI, destination URI, database names, and collection, depending on how you’re running the tool, configuration can come from command-line arguments, a config file, or environment variables, so double-check whichever method you’re using.

The dry run says 0 documents: This could easily mislead you that the tool is broken, or something is wrong somewhere, but I’d rather make sure you walk through the obvious things first: do the documents actually have createdAt? Also, is it stored as a proper MongoDB Date, not a string? Is 180 days actually the right age for your test data? or are you connected to the database you think you're connected to, and using the right collection? Honestly, most times it’s usually the filter, there’s probably no documents to archive, within that date range.

The destination already contains documents: That’s not necessarily a problem because the archive writes using _id, existing documents get updated rather than duplicated. Still, I'd always recommend starting with a dry run so you know exactly what you're about to process before you commit to it.

Permission errors: The source needs to be readable, the destination needs to be writable, so if you’re using separate MongoDB users per environment, confirm the account you’re using actually has the permissions it needs. This is exactly why running doctor first before any other commmand is recommended, it catches this kind of issues before you get into the archive process.

Don’t enable deletion yet

At some point you’ll notice the –delete-source flag sitting there, well I don’t think you should use it yet. What we’ve proven so far is something more important, you can safely copy the right documents into the archive without touching the source at all and that’s because before deleting anything, we want to verify the archive, understand how restore works, and confirm the whole process behaves the way we expect.

That’s what Part 3 is for.

What’s next

In Part 3, we’ll go beyond the basic flags: building a configuration file, writing custom queries, verifying an archive job, practicing a restore, and finally looking at what it actually takes to safely turn on –delete-source.

The important word there is safely. Deleting the source should never be the first step in an archive workflow.

Previous: Part 1: Why Your MongoDB Needs an Archive Strategy
Next: Part 3: Config, Verify, Restore, Delete

Before you go

• Please take a moment to like the post and follow the writer!
• Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute here.

---

Safer MongoDB Archiving with mongo-archiver — Part 2: Install and Run Your First Safe Archive was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.