Pex … a small package of dynamite
The Pex documentation defines this product as and I quote:
“Pex is a new tool that helps in understanding the behavior of .NET code, debugging issues, ann creating a test suite that covers all corner cases – fully automatically. Through a context menu in the code editor, the user can invoke Pex to analyze an entire class or a single method. For any method, Pex computes and displays interesting input-output pairs. Pex systematically hunts for bugs – exceptions or assertion failures. As Pex discovers boundary conditions in code, Pex generates new tests that target these conditions. The result is a small test suite with high code coverage. Pex enables Parameterized Unit Testing, an extension of traditional unit testing that reduces test maintenance costs. Pex has been used in Microsoft to test core .NET components. Pex is developed at Microsoft Research and is integrated into Microsoft Visual Studio.”
… end quote.
Perhaps we should explore the product by demonstrating it … much better than trying to understand the above q;-) To begin, we write a silly demo class and method …
1: using System;
2: using System.Collections.Generic;
3: using System.Text;
4:
5: namespace PexDemo
6: {
7: public enum CONVERSION
8: {
9: Append = 0,
10: UpperFirst = 1,
11: UpperSecond = 2,
12: Special = 3
13: }
14:
15: public class DemoClass
16: {
17: public string ConvertData(CONVERSION convertType, string stringOne, string stringTwo)
18: {
19: string result = null;
20:
21: switch (convertType)
22: {
23: case CONVERSION.Append:
24: result = stringOne + stringTwo;
25: break;
26: case CONVERSION.UpperFirst:
27: result = stringOne.ToUpper();
28: break;
29: case CONVERSION.UpperSecond:
30: result = stringTwo.ToUpper();
31: break;
32: case CONVERSION.Special:
33: result = (stringOne[stringOne.Length - 1]).ToString() +
34: (stringTwo[stringTwo.Length - 1]).ToString();
35: break;
36: }
37:
38: return result.ToString();
39: }
40: }
41: }
Analyzing the code
Can you see any issues with the code? Well, let’s ask Pex, by running a Pex exploration:
Pex now analyses our code by running it with different inputs, showing the results of its analysis in a Pex Exploration Results window as a table as shown:
… but there is more. If we select any line, i.e. line two, we get more information:
… a stack trace.
… details of the Pex test code that raised the concern.
… but there is more. While we are tempted to take the summary/exception and error message analysis information and fix the code ourselves, we can ask Pex to assist. If we select each of the errors, i.e. row 2, 5 and 6, we should notice the “Add Precondition” feature at the bottom.
Once we have selected each row and then the “Add Precondition” condition, the code is changed as follows for us:
1: using System;
2: using System.Collections.Generic;
3: using System.Text;
4:
5: namespace PexDemo
6: {
7: public enum CONVERSION
8: {
9: Append = 0,
10: UpperFirst = 1,
11: UpperSecond = 2,
12: Special = 3
13: }
14:
15: public class DemoClass
16: {
17: public string ConvertData(CONVERSION convertType, string stringOne, string stringTwo)
18: {
19: // <pex>
20: if (stringOne == (string)null)
21: throw new ArgumentNullException("stringOne");
22: if (stringTwo == (string)null)
23: throw new ArgumentNullException("stringTwo");
24: if (3u < (uint)convertType)
25: throw new ArgumentException("3u < (uint)convertType", "convertType");
26: if (stringOne.Length == 0)
27: throw new ArgumentException("stringOne.Length == 0", "stringOne");
28: if (stringTwo.Length == 0)
29: throw new ArgumentException("stringTwo.Length == 0", "stringTwo");
30: // </pex>
31:
32: string result = null;
33:
34: switch (convertType)
35: {
36: case CONVERSION.Append:
37: result = stringOne + stringTwo;
38: break;
39: case CONVERSION.UpperFirst:
40: result = stringOne.ToUpper();
41: break;
42: case CONVERSION.UpperSecond:
43: result = stringTwo.ToUpper();
44: break;
45: case CONVERSION.Special:
46: result = (stringOne[stringOne.Length - 1]).ToString() +
47: (stringTwo[stringTwo.Length - 1]).ToString();
48: break;
49: }
50:
51: return result.ToString();
52: }
53: }
54: }
We may have to run multiple iterations (in our case two) as the Pex exploration and test analysis is updated with each code change. As we add a precondition for the code lines 45 to 47, as above, the Pex Exploration detects the IndexOutOfRange condition for line 46. After adding a pre condition for the error, the Pex exploration detects the IndexOutOfRange condition for line 47, which is resolved in the second iteration.
I would be tempted to change line:
if (3u < (uint)convertType)
to
if ( ! Enum.IsDefined(typeof(CONVERSION),convertType))
… but that’s nit-picking q;-)
The results, after a Pex exploration, change as follows:
We note that the problematic NullReferenceExceptions are replaced with the acceptable behavior of an ArgumentNullException. The red rows have all turned green, which indicates that Pex is finally happy.
Creating a Test Project
I hate to say it, but there is more. If we select all rows, a save option appears:
… when saving, we get a confirmation dialog:
… on acceptance we have a test project with [TestMethod] attributed test methods, which we can run using the VSTS Test Manager. The real value of these tests are regression testing … in other words we can quickly re-test the class, with the test data, after making changes.
Debugging the test code
If we want to debug one the the analysis results, i.e. row 2, we place breakpoint on the relevant Pex code, select row 2 and then the Debug option:
… we break at the Pex pre condition code, after which we can debug the code step by step.
Conclusion
I really love the Pex logo … as I have been interested in and have been enjoying sharks in the ocean since I was a kid … BUT, I am really excited about the power of this product. Its potential is mind blowing and I, for one, cannot wait for the next release.
See http://dotnet.org.za/willy/archive/2009/01/05/pdc-video-8-contract-checking-and-automated-test-generation-with-pex-tl51.aspx for a reference to an excellent video on this technology.