The article Test Case Design Techniques to Ensure High-quality Software on ReQtest brings up the three major categories test design techniques are generally classified into. These techniques are specification-based, structure-based, and experience-based. Each of these techniques covers different testing methods we covered in CS-443, especially structure- and specification-based techniques.
Specification-based techniques are also referred to as black-box techniques, which should give you some idea of what kind of testing this category covers. Testing techniques under this category are generally written to the technical specifications and client’s requirements. These testing techniques rely more on an understanding of what a program’s intended functions are, and what different states it can find itself in performing those functions. The actual structure of the code is not considered. Using specification-based testing, you can verify that your program works the way it was intended and is written to specification.
Structure-based techniques are conversely referred to as white-box techniques. As expected by the name, these testing techniques test the actual code that is written, and requires knowledge of the code and its internal structure. Many of the testing techniques of this type involving changing conditions and values and making sure code works in multiple cases, validating each branch of the code. Structure-based techniques help highlight any glaring structural or logical issues within your code you may have overlooked.
The final category is experience-based techniques. This category is pretty broad, mostly involving techniques that rely on prior knowledge or information that couldn’t be gained just from testing. This is the type of testing you just can’t plan for. However, the more you test, the more you can apply knowledge from previous tests to fix issues before they become back-breaking.
Dividing the testing into different categories makes their usefulness easier to understand. Structure-based techniques are more difficult and require more knowledge, but are exhaustive and effective at finding issues in the code. Specification-based techniques verify that the parts of your code that matter are functioning correctly. Experience-based testing simply leverages your knowledge of an area to improve testing. All these techniques have valid uses in the world of testing.
In his post Six Things That Go Wrong With Discussions About Testing, James Bach lists six different things that can often go wrong when testers are discussing testings. His six reasons can be summarized as testers misunderstanding the purposes of testing and not following the proper procedure when it comes to testing.
Your goal when testing is to discover any vulnerabilities in your program, but also to find strengths and things your program is doing well. This covers James’ first two points. When it comes to testing, quality is much more important than quantity. It is better to have a more complete coverage of your program and all its branches than to have a bunch of tests doing the same thing and missing different parts of your program. This is why it is important to think of tests as events instead. What is important is what your test is testing and how it goes about that, and it shouldn’t be thought of as some ‘Generic X Test.’
Often times people get carried away with automated testing and rely too much on it. While it is good to use automated testing, developers have strength far beyond that, and should use automated tests as tools to better accomplish effective testing, not as automated workers that do your work for you. Thinking of automated tests in this way also distracts from the purpose of testing, and makes it easier to forget why you should run certain tests in the first place.
James’ last point is probably his most important one. Testing isn’t just some set task that can be navigated a certain way every time. What I’ve learned this year is that testing is about thinking critically about a program and the way it COULD act, and design your tests according to those parameters. Consider where things can go wrong, and try to test the strength of weak parts of your code. But most importantly, be ready to learn, as every program has a different set of tests with a different strategy that is most effective for testing. Testing is a dynamic thing, and testers must be dynamic people.
The blog post How Do I Know My Tests Add Value? by AutomationPanda discusses the value of proper testing. The author brings up the common issues that sprout from use a bug-fixed count as a metric for successful testing, and highlights why testing is important and can help improve development.
Testing is important because it validates that your program is working as it is expected to. Well-written tests with the proper amount of coverage give you information about your program and whether any changes need to be made. A passing test indicates correctness and that your program is working as expected. A failing test points out a bug that needs to be fixed and helps highlight weak parts of your code.
The absence of testing might not necessarily lead to bugs, but we all know as programmers that even when it seems your code is running exactly as you intended it to, logic errors can happen and sometimes things weren’t written with every situation in mind. When testing is in being implemented, developers are accountable for the code they write, and must think carefully about the issues that may arise.
Tracking bugs isn’t necessarily effective because it encourages testers to find issues even when there may not be any. All tests passing is still good news and a sign of progress, even if it just means you are doing everything right. Enforcing bug quotas and forcing testers to find issues means they will expend effort looking for things that aren’t there and nitpicking small issues rather than writing more tests or making sure they have good coverage, which are far more effective and finding issues in your code.
At the end of his post, the author suggests some different metrics to use: time-to-bug discovery, coverage, and test failure proportions. All of these serve as much more accurate and effective measures of determining whether your testing has value. Whatever metric you use, it is important to think about why you are testing and whether you are doing it in a way that tells you something about your program.
The article How to Conduct Effective Code Reviews by Billie Cleek covers code reviews, when to use them, and what your objectives and goals should be when working on or submitting a code review. He discuses the different roles you can take in a code review (which are almost analogous to our roles in two of my classes this semester) and what you should expect to do while in those roles in the process of a code review.
A code review is basically a conversation between developers on a proposed set of changes to a project. It can be a discussion about why a certain part of the code is the way it is, whether or not something is effective, or if certain changes need to be made and how to go about that. Code review boils down to having a constructive conversation regarding the development of your project, and what changes might need to be made.
I personally have had a lot of trouble communicating difficulties and voicing my opinion in past classes. It is hard to find your voice and be confident, stating the issues you see and opening yourself to feedback, however through code reviews everyone who participates stands to gain knowledge from their peers as well as experience in effectively communicating to your colleagues. As long as you are able to give and receive feedback in a helpful but constructive manner, you can help clean up a project, fixing errors and making it clear and understandable for viewers to read.
In a way, I feel like my software classes this year have done a lot of work in preparing me for being effective in code reviews, as well as in the workplace in general. A lot of the important skills in code reviews are just as important in group work: effective communication, making sure questions are answered, and mutually agreeing on the decisions being made are all essential to having an effective and useful code review. Building these skills in general will make you a better team member, and help you work better in a group on big projects.
In the post Mock? What, When, How? by Lovis Möller, he discusses mocking and when to use it. Mocking is a strategy used in test-driven development where a mock object is created to test the interactions and dependencies this class has with other classes. Lovis talks about looking at his own and other people’s code and determines some situations in which he would use or would not use mocking, and why he would or would not use it.
Lovis first point is that you should only mock types that are internal to your program, and avoid mocking external types. External types may have dependencies of their own that you don’t know about, and could change in a later version of the code. Avoiding mocking makes your code more adaptable for future versions. The next thing Lovis recommends is that you do not mock values. If you mock values, you aren’t actually testing any useful part of the code. Lovin states:
“Mocking is a technique that is used to make the relationships and interactions between objects visible.”
The last big example Lovis gives is to not to mock concrete classes. The example and associated code he gives makes it pretty clear, as the result depends on a method that has not been covered by the mocked methods within the code. It is easy to see how this could get out of hand if you had to covered each method of a mocked class for your mock testing.
I had a lot of trouble understanding mocking and when you would want to use and and why before reading this post. Although it is still a difficult concept to understand why we are making these mock objects, seeing circumstances in which you would or wouldn’t want to use mocking, and alternatives to mocking make it a little easier to see how it is testing a program and why it is a useful concept. The quote I selected above made this even more apparent to me. Mocking isn’t testing the implementation or your program so much as the relationships between objects and their methods and how they interact.
Michael Bolton’s post How Long Will the Testing Take? on his blog DevelopSense goes into depth on how long testing takes. Bolton claims good software testing as a process takes just as long as development, because the two are actually interwoven as processes and should happen simultaneously.
I think a lot of people brush off testing systems and software because of the idea that it takes additional time and can make the release of a project come later. However, as Bolton states, testing actually occurs simultaneously to development, as things are tested as they are implemented into the system to make sure they interact correctly. With this in mind, the program should be fully tested not long after it is completely developed. It doesn’t really cost much extra time, just the extra costs of developing tests. However, doing it this way also allows you to fix issues that come up along the way, before they become larger issues.
In some of my earlier programming classes, students were often encouraged to write one test and make sure it works before moving onto the next test. We were also encouraged to write tests after every method we made in our programs. In a way, we were clearly being prepared for this approach to development and testing, and it was clear even then why it is a good way to work on a program. It is logical, as incremental development seems a lot safer. You avoid getting overwhelmed with errors and issues that all compound on each other at once when you write your tests this way.
The blog post also covers the silliness of the question of how long testing will take. If your developers and testers are working together and doing what they’re supposed to, testing will finish as soon as development does. Although you can test the system in post-production, it is entirely optional, and is only done with good reason. So in the future, if I am ever managing the development of a system, I will be able to recognize what the testers are doing and ask more productive questions.
The article I chose to cover for CS-443 this week is a research article titled Decision Graphs and Their Application to Software Testing by Robert Gold. This article goes in-depth on one of the subjects covered in our Activities, DD-path testing, as well as other decision-based testing methods. One of the things I like about this article is that it uses a lot of mathematical notation and logic to represent how the methods work. It also has very detailed proofs to show why the necessary branch/edge coverage is reached. There are several different examples of code that have corresponding decision graphs, to show the different parts of making a decision graph.
Looking at the example of the function f1 given in the article (Figure 3), there is something this article did with DD-path graphs that we did differently in class. whereas our DD-paths were just all the nodes in between with an in-degree and out-degree of 1, when Robert combines several D-nodes into one DD-path, he includes the first node that has an in-degree of 2 and the last node which has an out-degree of 0 along with the rest of the nodes. Therefore, his DD-Path graph for f1 is a lot simpler than what we would have made doing what we did in class. I’m curious which way is better. The article also dives into a topic about path testing we discussed but did not have an example of, that is when code branches into multiple different endpoints. In order to achieve branch coverage in these examples, you have to have multiple tests with different inputs that satisfy every branch.
Overall, the abundance of examples in both code and their associated graphs, as well as the very clear mathematical notation and rigorous proofs makes this article a really interesting read if you are a Mathematics major/minor. The conclusion at the end details a major application of this are modeling programs with flow graphs to represent programs. There are many different kinds of decision based testing methods that you can use, and if you have any questions about one of them or how they work, this is an amazing resource.
For my post this week, I found an excellent blog post by Martin Fowler on his personal website that details the “Test Pyramid.” The test pyramid is a way of representing the proportion of different tests you should be using for the purposes of keeping a balanced portfolio of tests.
As you can see by the pyramid, user-interface testing is slower and more expensive, moving to service testing which finds itself in the middle and then on towards unit testing which is speedier and less costly. This would imply that in general you want to rely more on unit testing than UI or service based testing, although all of these types of tests have a purpose.
Martin talks about the disadvantages of relying too heavily on user-interface testing. The slow testing increases build times and can require installed licenses for test automation software, which costs money and limits which machine the tests can be run on. The largest con Martin details was how brittle the tests were. That is, any small change to the system could break several tests and require them to be re-recorded. All these reasons show why the pyramid is the way it is, putting UI testing on the higher, smaller end. Fowler makes a point that often the usage of the high-level tests is as a second line of test defense. These tests are able to capture a bug you might have missed or not written tests for, and enables you to write a unit test to ensure the bug is taken care of.
I think the test pyramid is an interesting idea that highlights the importance of unit testing and helps define the priorities we should have as testers. Although it does not dismiss UI or service testing, it clearly demonstrates as testers that we should primarily be using unit testing to fix bugs in the code and make sure they stay fixed. High-level testing can be used to find things that are missed, but shouldn’t be relied on to catch bugs, and they won’t necessarily stay stable when you make changes to your program.
Hello again! My name is Nicholas Coutu. I am a senior attending Worcester State University. I major in Computer Science with a concentration in Software Development and am also attempting a minor in Mathematics. I intend to use this blog to record my opinions on various articles related to my classes throughout my final year, and have further plans to use this blog for my professional career to document my work-related activities.