Engineering · From LinkedIn

Safer production data scripts

For production fixes, I prefer being boring and safe.

One thing I have learned while working on production databases is this:

Migration scripts and backfill scripts should be treated with a lot of respect. Earlier, I used to think a fix script is just about writing the right query and updating the data.

But in production, even a small mistake in the query can impact a lot of records.

So now, whenever I have to run a migration, backfill, or any data correction script, I try not to combine everything in one script like:

find data → process data → update data

That feels risky to me, especially when the data is critical.

Instead, I prefer a staged approach.

First, I write only the find queries or aggregation pipelines.

Then I verify the count, check sample records, and confirm that this is exactly the data I want to work on.

After that, I export the selected data into CSV or JSON.

Then the actual modification script works on that confirmed dataset, usually by using stable identifiers like _id, instead of running broad queries again.

This gives me more control.

It also helps in reviewing what is going to change before the actual update happens.

A safer production data script, in my opinion, should have:

  • clear target data
  • dry-run mode
  • before/after values
  • rollback possibility
  • batch processing
  • post-update verification
  • logs of what was skipped and what was updated

The biggest lesson for me is that writing production scripts is not only about MongoDB queries or Node.js code.

It is about reducing blast radius.

It is about being able to answer: “What exactly will this script change?”

and also: “How can we recover if something goes wrong?”

This kind of carefulness may look slow in the beginning, but it saves a lot of stress when working with real production data.

← All short posts