Rosario April 2008 CTP Investigation (Part 7) - Tester
Continued from http://dotnet.org.za/willy/archive/2008/06/06/rosario-april-ctp-investigation-part-6-developer.aspx.
The test environment has been plagued with issues and missing tools, such as automation of win form application testing. The good news is, that such support is coming in future CTPs and that overall productivity and feature improvements will finally allow us to focus seriously on defect management and testing of solutions.
In this post we will have a closer look at the Team Build and some of the features relating to testing, we will look at manual tests, reviewing and fixing a bug and finally verify the bug fix.
Team Build
Refer to http://dotnet.org.za/willy/archive/2008/06/02/rosario-april-ctp-investigation-part-5-build.aspx where we setup the relevant build. Let’s focus on some of the key areas of the build for this test blog:
-
Under the Compile and Test activity, we select the Get Impacted Tests activity and set the Enabled property to true.
-
Under the Compile and Test activity, we select the Publish the Test Impact Data activity and set the Enabled property to true.
-
Under the Test All Projects, we expand the Test a Project activity, select the Run MSTest for Project activity and set the PublishTestImpactInformation to true.
We run a test build just to be sure that everything is OK and that the test analysis information is published.
Manual Test
-
We first start a new application, Microsoft Codename “Camano”, and connect to the correct Team Project (LoanOriginating).
-
We then select the Testing menu ...
-
... which takes us to the Testing Activity Centre.
-
Next we select the Initial Test Pass test case and select Start Testing command in the list if Testing Activities as shown:
-
Locate the LoanExplorer.exe application in the drop, for the build we executed in the previous build:
-
Place focus back to the Test Runner and define the configuration for Recording Settings as shown:
-
Select “Start Test Case”. The triangle to the left of the exclamation mark are added automatically and used later to replay steps related to the test ... these are magic markers.
-
We then proceed as per the test steps, entering 0.8 for the APR. Take note of the green dot appearing in the Loan Explorer.
-
Proceed by entering 60 for number of payments and 30000 for the loan amount. Hit Calculate gently to work out the payment amount ... great bank, cause the loan amount appears negative, i.e. we get money for loaning money J
-
If we review the test step details for review, we note that the expected result must be 608.29. This leaves us no option but to Fail the test.
-
In addition we select Actions à Create New Bug.
-
Completing the bug form, we notice that the Repro Steps have already been completed for us ... based on the test case.
-
When we publish the test case results, which include a number of attachments as seen at the bottom of the left illustration, the test case goes grayed out ... in other words the information has been published.
-
That’s about all the tester can or should do. The next step is to review and fix the bug.
-
By the way, if you want to create a new test case, select Planning à New Test Case in the Camano application.
Review and Fix a Bug
This is the time when developers start grumbling ... bug storm ahoy.
If you are the developer assigned to the bug report from the previous step, you would proceed as follows:
-
Review the bug item
-
The Repro Steps document the exact steps as to how to reproduce the bug.
-
To make matters worse we have three attachments, whereby the video is the evidence we love the most ... it counteracts one of the normal “show me” response or the frustrating inability to reproduce as the developer machine is typically more complete in terms of installations than production or test machines.
-
The evidence “shows” for itself”...
-
When we realise we indeed have a problem, we change the Triage to “triaged” and the State to “active”. Then save!
-
Review the bug item
-
We admit defeat, checking out the relevant solution and fixing the bugs.
-
We keep an eye on the Test View window, which currently shows Not Impacted, as the test impact status.
-
We change the code from:
Public Sub CalculatePayment()
Me.Payment = Pmt( (APR / 12), _
NumberOfPayments, (LoanAmount * 1), _
0, DueDate.EndOfPeriod )
End Sub
to ... (spot the * -1)
Public Sub CalculatePayment()
Me.Payment = Pmt( (APR / 12), _
NumberOfPayments, (LoanAmount * -1), _
0, DueDate.EndOfPeriod )
End Sub
-
After saving the code, we note that the Test View Window has changed to Recommended.
-
STOP ... why have all our builds succeeded to date, why have none of the tests failed? Simple, the unit test has the same anomaly ... fix the test as well to expect a positive, not negative replacement.
-
If, and only if, we get a passing Unit Test, we can save the code, check-in pending changes and re-assign the bug work item back to tester.
Verify a Bug Fix
The tester typically proceeds as follows to verify the fix we have just made:
-
-
Open the Testing Activity Centre.
-
Select the Initial Test Pass test case.
-
Start Testing ...
-
We are faced with a similar view:
-
Instead of doing the test again and introducing possible human error, we use the power of the Test Runner. We select start test case and then note the warning, we proceed with caution.
WARNING: This step took me a while, because the UI is either not intuitive, or I was just being silly ...
I finally managed by clicking on the “Start the ...” step and then on the marker next to the 30,000 a number of times, until it ‘automatically’ highlighted the four steps and popped up the list of commands.
-
Selecting “Play selected clip”, I watched as the Test Runner completed the capture steps for me. In other words the exact same values were entered as before.
-
To conclude we select calculate and note with happiness that the expected value equals the actual value.
-
Mark the test case as complete with success, pack-up and run ... why run? Well, the product is still temperamental and I was unable to complete a second successful iteration. I am, however, ecstatic with what is coming and what the opportunities are.
Conclusion
This post was completed at home on a public holiday ... I thought I would be able to sit for a few hours and focus. Ha, ha ... I completed it in between soccer, computer games, foxing computers, dragging furniture around, tea, ... at least I completed it. I am happy I did complete this today, because the testing enhancements are exciting. The test and re-test puzzle is finally falling in place and the test case life time management is finally a reality.
Camano is really, really exciting! Manual tests are finally first class citizens and a useful weapon in the testing arsenal.
My personal rating for the Team Build improvements in Rosario at this stage is: 
/ 5 smiles. The two unhappy smiley is for the endless clicking and battles I had when replaying the test case ... not an intuitive and repeatable process yet ... very temperamental indeed, but then again it is an ‘oldish’ CTP.
Next
... part 8 will look at Team Edition for Database Professional and conclude this mini-series. The series will also be available in a combined PDF document for easier reading.