🐛 How Property-Based Testing Uncovered a Hidden Bug During Our MongoDB 8.0 Upgrade

Discovering the power of random test generation and why it's a game-changer for robust software testing 🎲

When we embarked on upgrading our MongoDB from 6.0 to 8.0, we expected the usual challenges: dependency updates, breaking changes, and compatibility issues. What we didn't expect was to discover a hidden bug that had been lurking in our codebase for years - all thanks to a testing technique called property-based testing.

This blog post shares our journey of discovering property-based testing, how it helped us find a real bug, and why we believe every engineering team should consider adopting this approach.

❓ What is Property-Based Testing?

Property-based testing is a testing methodology that generates random test cases based on properties or invariants that your code should always satisfy. Instead of writing specific test cases, you define:

1. Properties: Rules that your code should always follow

2. Generators: Functions that create random but valid test data

3. Test Framework: Tools that run hundreds or thousands of random tests

The key insight is that by testing with random data, you can discover edge cases and bugs that you might never think to test manually.

🔍 Our Discovery Journey

The Setup

Our project uses a dual-database approach:

• HDB: An in-memory hash database for fast testing

• MongoDB: The production database

We have property-based tests that ensure both databases produce identical results for the same operations. This is implemented using clojure.test.check, a property-based testing library for Clojure.

The Unexpected Failure

During our MongoDB 8.0 upgrade testing, we ran our test suite and encountered this failure:

FAIL in (some-test) (some_file.clj:66)
expected: {:result true}
actual: {:shrunk {:total-nodes-visited 275, :depth 55, :pass? false, :result false, ...}}

At first glance, this looked like a random test failure - something that might be non-deterministic or related to the MongoDB upgrade. But property-based testing gave us something valuable: a minimal, reproducible test case.

The Magic of Test Shrinking

Property-based testing frameworks don't just find failures; they shrink them to the smallest possible test case that still reproduces the bug. Our framework found this minimal failing case:

{:ops [[:put {:_id 0, :vec [], :val " ", :intval 0, :A ""}]
       [:query-delete {:val [:<= "["]}]],
 :read-ops [[:count {}]]}

This simple sequence of operations:

1. Insert a document with :val " " (a space character)

2. Delete documents where :val <= "["

3. Count remaining documents

Expected: Both databases should return the same count

Actual: HDB returned 1, MongoDB returned 0

🔬 The Bug Investigation

Initial Assumptions

Our first thought was that this might be a MongoDB 8.0 compatibility issue. After all, we were testing for the 8.0 upgrade, and string comparison behaviour could have changed between versions.

The Reality Check

We created a deterministic test to reproduce this exact scenario and ran it against:

• MongoDB 6.0

• MongoDB 7.0

• MongoDB 8.0

Result: The bug existed in all three versions. This wasn't a MongoDB upgrade issue at all - it was a bug in our query translation logic that had been there for years!

The Root Cause

The issue was in our query translation layer:

• Swym Query: [:<= "["] (our custom query syntax)

• MongoDB Query: {:$lte "["} (MongoDB's syntax)

;; Deterministic test for the failing case found by property-based testing
;; This test case was discovered with seed 0123456789 and helps identify
;; a bug in query translation between HDB and MongoDB
(deftest test-failing-case-query-translation-bug
  (testing "Test the failing case that was discovered by property-based testing"
    (let [ops {:ops [[:put {:_id 0, :vec [], :val " ", :intval 0, :A ""}]
                     [:query-delete {:val [:<= "["]}]],
               :read-ops [[:count {}]]}]
      (println "=== Testing Query Translation Bug ===")
      (println "Operations:" ops)
      (let [result (test-models true ops)]
        (println "Test result:" result)
        (is result "HDB and MongoDB should produce equivalent results for the same operations")))))

The problem was in how string comparison was being handled between our in-memory database and MongoDB. The comparison between a space character " " and "[" was producing different results in the two implementations.

➜  some-model git:(mongo8-compability-check) ✗ lein test some-model.some-test :only some-model.some-test/test-failing-case-query-translation-bug

lein test some-model.some-test
=== Testing Query Translation Bug ===
Operations: {:ops [[:put {:_id 0, :vec [], :val  , :intval 0, :A }] [:query-delete {:val [:<= []}]], :read-ops [[:count {}]]}
Initializing! localhost nil test
hdb scan:  {0 {:_id 0, :intval 0, :val  }}
hdb returns:  [[:count 1]]
mongodb scan:  []
mongodb returns:  [[:count 0]]
Test result: false

lein test :only some-model.some-test/test-failing-case-query-translation-bug

FAIL in (test-failing-case-query-translation-bug) (some_test.clj:83)
Test the failing case that was discovered by property-based testing
HDB and MongoDB should produce equivalent results for the same operations
expected: result
  actual: false

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Subprocess failed (exit code: 1)

⚡ Why Property-Based Testing Was Crucial

1. Uncovered Hidden Edge Cases

We would never have thought to test the specific case of comparing a space character with "[". Property-based testing generated this scenario randomly.

2. Provided Minimal Reproduction

The test shrinking gave us the smallest possible test case that reproduced the bug, making it much easier to debug and fix.

3. Demonstrated Real Value

This wasn't a theoretical benefit - we found a real inconsistency in our test setup.

📋 The Aftermath

What We Did

1. Created a Deterministic Test: We added the failing case as a regular unit test to track the bug

2. Documented the Issue: Created a JIRA ticket with full details

3. Shared with the Team: Posted about the discovery to raise awareness

4. Planned the Fix: Scheduled investigation of the query translation logic

Key Learnings

• Property-based testing is powerful: It can find bugs you'd never think to test

• Random failures can be valuable: Don't dismiss them as "flaky tests"

• Test shrinking is magical: It turns complex failures into simple, debuggable cases

🛠️ Implementing Property-Based Testing

Getting Started

If you're interested in property-based testing, here are some popular libraries:

• Clojure: clojure.test.check → https://github.com/clojure/test.check

• JavaScript: fast-check → https://www.npmjs.com/package/fast-check

• Python: hypothesis → https://hypothesis.readthedocs.io/en/latest/

• Haskell: QuickCheck → https://hackage.haskell.org/package/QuickCheck

• Java: jqwik → https://jqwik.net/

Best Practices

1. Start Small: Begin with simple properties and gradually increase complexity

2. Use Test Shrinking: Always take advantage of the framework's ability to minimize failing cases

3. Document Properties: Clearly state what invariant you're testing

4. Combine with Traditional Tests: Property-based testing complements, doesn't replace, traditional testing

5. Monitor Performance: Property-based tests can be slower, so run them appropriately

🎯 Conclusion

Property-based testing transformed our MongoDB upgrade from a routine task into a bug discovery mission. What started as a "random test failure" turned into a valuable lesson about the power of automated test generation.

The bug we found wasn't critical, but it demonstrated a real inconsistency in our testing layer that could have tripped others noticing the same behaviour in future. More importantly, it showed our team the value of property-based testing and encouraged us to explore this technique further.

Key Takeaways

• Property-based testing can find real bugs that traditional testing misses

• Random test generation is not just for stress testing - it's a powerful debugging tool

• Test shrinking makes complex failures simple and debuggable

• The investment in property-based testing pays off in bug discovery and prevention

If you're not using property-based testing in your projects, we highly recommend giving it a try. You might be surprised by what bugs you discover!

---

Have you had similar experiences with property-based testing? We'd love to hear your stories in the comments below!

📚 Resources

• Clojure Test Check Documentation

• Property-Based Testing with Hypothesis (Python)

• Fast-Check (JavaScript)

• QuickCheck (Haskell)

---