Visual Studio 2010 CTP 2 (7/7) - Testing & Conclusion
Continued from http://dotnet.org.za/willy/archive/2008/11/21/visual-studio-2010-ctp-2-6-7-the-exciting-world-of-construction-part-2-2.aspx
After getting stuck with an issue around test impact analysis, I decided to place that in a holding pattern and conclude this series of the CTP2 evaluation regardless. The test impact analysis will be discussed in a separate post ... hopefully sometimes next week, if the weekend is going to be a successful one q;-)
Testing with Camano
We again fire up Camano to take a look at the world of testing, running a manual test and creating a bug as outlined in the CTP walk-through documentation. The key point to mention is that we closed Visual Studio, threw away the developer hat and are now assuming a tester role in the self-standing Camano environment.
Setup the battle armour for the test, including:
- Action Log ... recording each user interface action that the tester performs
- Diagnostic Trace Collector ... collecting diagnostic events for the tested application
- System information ... recording system information for the computer running the test
- Test Impact Local Collector ... recording data used to determine which tests are recommended when code changes are made
- Video Recorder ... collection video recordings of the desktop for "visualisation of what really happened"
Run the manual test and save the test.
... launches the test runner ... 
The test runner allows us to (1) Test and Automate, for future replay and (2) gives us the manual test steps to complete using the (3) application to be tested.
... before we launch, we confirm settings ... 
We proceed to use the test runner. The (1) marker tells us where we are, the (2) Pass|Fail allows us to indicate success and the (3) toolbar has magical features, such as selecting and saving a portion of the screen, invaluable when analysing a bug.
... reporting a bug when we are not happy ... 
... note that test steps are automatically included in the bug report.
The developer now has no reason to say "cannot reproduce", "cannot rely that report matches the actual test" ... in short, the developer has all the ammunition needed to analyse and resolve the bug. This is a great feature and will not only reduce the tension between testers and developers, but through comprehensive and accurate issue recording and reporting, allow the developer to resolve the issues much quicker.
... working in harmony to complete the puzzle (solution).
Using Debugger History to debug after the meltdown
Now that we have a bug, we can switch our schizophrenic self from tester to developer and review the bug. Goodbye Camano, welcome Visual Studio.
First we find the bug in the list of active bugs:
The historical debugging log file gives us a great insight into what happened, what the state of the nation was at the time and in future, the product team is planning to add a lot more information to support the developer.
All in all, without going into much more detail, this is again an exciting feature, which we have covered to some extend in CTP here.
Conclusion
We have reviewed most of the features in CTP1 already, but can happily report that most issues we have had before in the previous CTP have been resolved. The architecture/modelling features are probably the most exciting to me, with the gated check-in and branch visualisation competing for second spot. Overall the latest community technology preview (CTP) has demonstrated great strides made in VSTS and we can really look forward to the next CTP, or BETA?!? As there is no go-live-license as yet, I will not make the same mistake by stating that I want and will implement this CTP2 in our research and development production environment ... "not allowed" q;-(
Some question are unanswered at this stage, i.e. migration from 2005/2008 to 2010, coexistence and performance, but there is still lots of time for that.
Summary
My 'personal' rating, based on a 5 smiley (maximum) system ... accumulated and summed up across the posts:
| Preview Walk-through document | ☻☻☻☻ | First time I am rating it four, not five, because the last walk-through was dependent on the previous. As my VM blew, then insisted on activation, I had to re-copy the 20+ gigs and restart. Then I spotted the dependency ... |
| Visual Studio Team System 2010 | ☻☻☻☻ | The CTP gremlins were with me again, 'cause Camano decided to abort and eject a few times. BUT it is a CTP and the experience, overall was very pleasant. Therefore I gave it four smileys. |
| Team Foundation Server 2010 | ☻☻☻☻☻ | TFS still purring patiently in the background, no issues to report. |
Average evaluation rating so far is 65 smiley's out of 68, i.e. 4.78/5 score ... not too shabby!
Accumulative Wish List
- Ability to insert|paste images into the description and history fields of work items.
- Ability to select a list of source files or projects in a new view (list of graphical), which allows analysts to determine the impact on the test impact if specific projects or files re touched, without having to make code changes.
Concern / Observation List
- Performance and stability of the IDE is a concern, but as we are working with a CTP and functionality is currently dominating the priority list, I am convinced that reliability, stability and performance concerns will evaporate in the next CTPs/BETAs.
- Some of the modelling tools, especially architecture explorer, are very resource hungry. My 1.5GB ram virtual environment often had the gulp ... telling me to "go for coffee, while I give the external drive a beating" ... processing different viewing modes.
- Running all of TFS 2010 and VSTS 2010 on the same virtual machine has proven to be a patience tester ... if you can give the VPC more than one processor and more than 2GB on Hyper-V, then that is the way to go. Running it in Virtual PC with 1.25GB or 1.5GB is not a pleasurable experience.
We still have one last post to go, "Visual Studio 2010 CTP 2 - A close encounter with Test Impact Analysis", which will document our excursions and findings with the test impact analysis feature.