Code Smell 320 - Brushing Over Real Problems

\ TL;DR: You write tests that touch every line but verify nothing, creating false confidence in a broken system. Problems 😔 False confidence Hidden production defects Misleading metrics Wasted test effort Untested edge cases Solutions 😃 Use mutation testing Test real behaviors Write assertive tests Delete coverage-only tests Refactorings ⚙️ https://maximilianocontieri.com/refactoring-011-replace-comments-with-tests?embedable=true Context 💬 Many teams set a coverage threshold: 80%, 90%, or even 100%. When you chase that number, you write tests that call methods without checking the actual results. A test that calls calculateTax() but only asserts result is not None executes the line. It doesn't verify the tax calculation is correct. The dashboard turns green. Defects survive in production. This is vanity coverage: cosmetic metrics that hide real problems. Like brushes that make surfaces look smooth while the rot stays underneath. Mutation testing reveals the truth. When you mutate production code and no tests fail, your coverage numbers lied. Sample Code 💻 Wrong 🚫 describe('BankAccount', () => { test('deposit', () => { const account = new BankAccount(100); account.deposit(50); // Only checking it didn't crash expect(account).toBeDefined(); }); test('withdraw', () => { const account = new BankAccount(100); const result = account.withdraw(30); // No assertion about the result! }); test('transfer', () => { const source = new BankAccount(200); const target = new BankAccount(0); // Just calling the method to "cover" the line source.transfer(50, target); }); }); Right 👉 describe('BankAccount', () => { test('deposit increases balance', () => { const account = new BankAccount(100); account.deposit(50); expect(account.balance()).toBe(150); }); test('withdraw decreases balance', () => { const account = new BankAccount(100); account.withdraw(30); expect(account.balance()).toBe(70); }); test('withdraw raises on insufficient funds', () => { const account = new BankAccount(50); expect(() => account.withdraw(100)) .toThrow(InsufficientFundsError); }); test('transfer moves money between accounts', () => { const source = new BankAccount(200); const target = new BankAccount(0); source.transfer(50, target); expect(source.balance()).toBe(150); expect(target.balance()).toBe(50); }); }); Detection 🔍 [X] Semi-Automatic Run a mutation testing tool ( PIT for Java, Stryker for JavaScript, mutmut for Python). Count the surviving mutants. If coverage is high but mutants survive, you have vanity coverage. You can also search for assertion-free tests, single assertNotNull() assertions, or tests that still pass after you delete the entire production method body. Exceptions 🛑 Smoke tests that call endpoints to verify the system starts are acceptable without detailed assertions. These work when they complement a real test suite, not replace it. Tags 🏷️ Testing Level 🔋 [x] Intermediate Why the Bijection Is Important 🗺️ Your test suite must map each test to a real behavior in the MAPPER . When a test covers a line without verifying observable behavior, you break that bijection . Coverage tools only measure "lines executed." Chase the number without bijection and your suite looks complete while missing real requirements entirely. AI Generation 🤖 AI code generators sometimes produce vanity coverage. Ask one to "add tests to reach 80% coverage," and it writes tests that call methods and assert trivially true facts. The metric goes up. Nothing gets verified. AI Detection 🧲 AI can detect vanity coverage, but only if you ask the right questions. Try: "Find tests with no real assertions" or "Find tests that pass when I delete the production method body." Without those prompts, most AI tools see a green test and call it good. Try Them! 🛠 Remember: AI Assistants make lots of mistakes Suggested Prompt: Replace vanity coverage tests with tests that verify real behaviors and fail when production code is wrong Without Proper Instructions 📵 ChatGPT Claude Perplexity Copilot You Gemini DeepSeek Meta AI Grok Qwen With Specific Instructions 👩‍🏫 ChatGPT Claude Perplexity Copilot You Gemini DeepSeek Meta AI Grok Qwen Conclusion 🏁 Coverage is a signal, not a goal. When you treat it as a goal, you create vanity coverage that hides real defects . Use mutation testing to discover what your suite actually verifies. Write tests that describe real behaviors, not tests that execute lines. Relations 👩‍❤️‍💋‍👨 https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xxi https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xvi https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xxxv https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-vi-cmj31om https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xli More Information 📕 https://en.wikipedia.org/wiki/Mutation_testing?embedable=true https://martinfowler.com/bliki/TestCoverage.html?embedable=true https://stryker-mutator.io/?embedable=true Quote The most dangerous kind of waste is the waste we don't recognize. Shigeo Shingo Disclaimer 📘 Code Smells are my opinion . Credits 🙏 Photo by Jamie Street on Unsplash This article is part of the CodeSmell Series. https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-i-xqz3evd?embedable=true \