Stop Saying That

Here are some testing related terms that people need to stop using.

End to End Testing

This is my biggest pet peeve because this term means different things to different people. Those with a development background use this term to mean testing the full development stack from top to bottom. Those with a testing only background use this term to mean a user’s flow through a site (e.g. logging in and going through all of the steps to purchase an item). Rarely when speaking to each other do these two groups realize that they are not talking about the same thing.

These should be referred to as:

• DOM to Database testing - this could also be called Full Stack testing, but D2D is a better equivalent to E2E while being much more descriptive.
• Journey testing - This could also be called Flow testing, but, since it is more suited to manual testing than automated testing, Journey fits better.

There are significant advantages to focusing on the DOM to Database approach instead of the Journey approach when writing Selenium tests, so it is important not to confuse these two.

UI Testing

This term is too ambiguous.

• Is it testing just the javascript? Those are JavaScript Unit Tests.
• Is it loading the JavaScript MVC framework and running tests while mocking out the API calls to the Service layer? That is DOM testing.
• Is it testing the full stack? As discussed above, that is DOM to Database testing.

Functional Testing

This term is also too ambiguous. Every test is verifying that something is functioning as intended.

Integration Testing

Another ambiguous term. Testing anything in combination with anything else is an integration test. It is often used as if it had more specific meaning, though, and often in contradiction to how someone else is using it.

Test Script

A script has the connotation of procedural programming. It is how manual testers often think about automation: write code to accomplish each step taken by a user and put it in a single file. When Record and Playback tools generate code, that code can be considered a test script.

If the plan is a comprehensive test suite, then that means writing good maintainable code. Writing maintainable code means proper abstractions and modeling. It means doing more than merely writing a procedural recipe, it means using page objects and controllers and managing test data and configurations.

At that level it is not writing a test script, it is writing an automated test and using it in an automated test suite. The software industry needs to get better at either encouraging developers who already know good software engineering practices to write the tests, or training existing testers to become correspondingly skilled software developers. Either way, no one should be writing (or suggesting that anyone should be writing) test scripts.

Test Frameworks

This is the appropriate term in some circumstances, but it often gets used when it shouldn’t be. A few definitions for disambiguation along with some examples of applicable Ruby projects:

Test Runner - These provide structure for running tests, but by themselves do not constitute a framework. MiniTest, RSpec and Cucumber are all test runners.

Test Library - A collection of code facilitating the user to accomplish the development tasks. Selenium, Watir and Page Object are all test libraries, not test frameworks.

Test Framework - For something to be considered a “framework,” it needs to involve Inversion of Control. The framework not the user is interacting with the external libraries to perform the actions. Capybara is an example of a test framework because once the user has set the configurations, uch of what it does happens without the user needing to direct most actions.

Testing vs Checking

Just kidding. But seriously, enough already, ok?

Conclusion

Those who work with tests can stand to be a lot more clear and consistent in the terms they use. Feel free to write a comment to add an item to this list or take issue with one of mine.