This is the seventh post in a series where I document how to setup a Continuous Integration (CI) process using open source tools like CruiseControl.NET, NCover, NUnit, Subversion and Sandcastle together with MSBuild and VS 2005. Part 7 covers the QtpBuild and the targets and tasks used by the QtpBuild. The QtpBuild shows how to run the regression test pack using Mercury's QuickTest Professional and Mercury Quality Center.
Post Updates
- 11/12/2007: Updated the CC.Net configuration to use build queues and entity references.
- 28/09/2007: Changed the QtpBuild to make use of incremental builds to install/uninstall the application and to run the test cases, identified in a file, from within Mercury Quality Center.
MSBuild Scripts
As mentioned in Part 1, QTP runs on a dedicated QtpServer machine as the regression test pack takes quite some time to run through (3 hours in our scenario). This begs the question how to invoke Qtp remotely from the BuildServer. I considered the following options:
- Remotely invoke Qtp using psexec from our BuildServer.
- Remotely invoke Qtp using DCOM from our BuildServer. This seems possible using QTP's automation object model.
- Instead of remotely invoking QTP, install CC.NET on the QtpServer machine itself and create a project that monitors and triggers a build once the DeploymentBuild has been completed on the BuildServer.
Depending on your environment scenario you may choose either option 1 or 2 as it requires less configuration, but in the end I chose to not use a remove invocation mechanism in favour of option 3. The down side is that I have to install CC.NET on the QtpServer as well. In addition to deploying the build onto our file server, the DeploymentBuild also copies the msi onto a share for the QtpBuild to use.
Uninstall
Before running the tests the old version of our application needs to be uninstalled.
1 <Target Name="Uninstall"
2 Inputs="@(RunQtpInstallInput)"
3 Outputs="@(RunQtpInstallOutput)">
4
5 <!-- Launch the Uninstall in silent mode -->
6 <Exec Command="msiexec /x $(DOUBLE_QUOTES)$(QtpUninstallFolder)\$(MsiInstallFile)$(DOUBLE_QUOTES) /passive /l* $(DOUBLE_QUOTES)$(QtpUninstallFolder)\Uninstall.log$(DOUBLE_QUOTES)"
7 ContinueOnError="true" />
8
9 </Target>
Observations:
- Lines 2-3: We use MSBuild's incremental builds feature to ensure that we do not uninstall the application if the latest version of the application is already installed on the machine. This is accomplished by comparing the timestamp of the QtpLastInstallSucceededFile to the timestamp of the latest msi package. If the msi package timestamp is later we know that we have a new version to install and thus need to do the uninstall.
- Line 4: The old version of our application is uninstalled by launching Windows Installer with the /x and /passive parameters. The /passive parameter ensures that no user interaction is required. We also log the uninstall output to a uninstall.log file. ContinueOnError is set to true for those scenarios where the machine hasn't got an installed version of our application.
Install
Before running the tests the new version of our application needs to be installed. As mentioned earlier, the msi is copied into the QtpInstallFolder share by the Deploy target that runs as part of the DeploymentBuild.
1 <Target Name="Install"
2 Inputs="@(RunQtpInstallInput)"
3 Outputs="@(RunQtpInstallOutput)">
4
5 <Error Text="No QtpDeployFolder has been specified" Condition="'$(QtpDeployFolder)' == ''" />
6
7 <!-- Copy the latest install from the build server -->
8 <Copy SourceFiles="$(QtpDeployFolder)\$(MsiInstallFile)"
9 DestinationFolder="$(QtpInstallFolder)"/>
10
11 <!-- Launch the install in silent mode -->
12 <Exec Command="msiexec /i $(DOUBLE_QUOTES)$(QtpInstallFolder)\$(MsiInstallFile)$(DOUBLE_QUOTES) /passive /l* $(DOUBLE_QUOTES)$(QtpInstallFolder)\Install.log$(DOUBLE_QUOTES) CLEAN_DATABASE=1"/>
13
14 <!-- Copy the msi to the Uninstall folder to use for later uninstallation -->
15 <Copy SourceFiles="$(QtpInstallFolder)\$(MsiInstallFile)"
16 DestinationFolder="$(QtpUninstallFolder)"/>
17
18 <Touch Files="@(RunQtpInstallOutput)" AlwaysCreate="true" />
19
20 </Target>
Observations:
- Lines 2-3: We use MSBuild's incremental builds feature to ensure that we do not uninstall the application if the latest version of the application is already installed on the machine. This is accomplished by comparing the timestamp of the QtpLastInstallSucceededFile to the timestamp of the latest msi package. If the msi package timestamp is later we know that we have a new version to install and thus need to do the install.
- Lines 8-9: The latest msi package is copied from the server to the local machine to use for installation.
- Line 12: The new version of our application is installed by launching Windows Installer with the /i and /passive parameters. The /passive parameter ensures that no user interaction is required. We also log the install output to an install.log file. Also notice the CLEAN_DATABASE=1 statement that we pass onto the installer. As the installer is launched in passive mode, we need to set any installer options that would normally be selected by the user via the GUI now via the command line.
- Line 15-16: After installing the application, a copy of the msi is put into the QtpUninstallFolder to use for the next Uninstall.
- Line 18: We touch the timestamp of the QtpLastInstallSucceededFile to support future incremental builds.
Qtp Automation
Fortunately, the help file of Mercury's QTP product has some good examples of how to automate QTP using COM automation. I used extracts from their VBScript examples as a base for creating a RunQTP.vbs file that we use for doing our QTP automation. As QTP stores the result for every test in XML format, the idea is to create a single XML file that contains the combined results from all the individual tests and to merge this file into the CC.NET build report.
1 Dim arTestCases, qcServer, qcUser, qcPassword
2 Dim testCasesFile, resultsDir, resultsSummaryFile
3 Dim fso, folder, subFolders, testResults, testCases
4 Dim qtApp, qtTest, qtResults
5 Dim testsPassed
6
7 testCasesFile = WScript.Arguments(0)
8 resultsDir = WScript.Arguments(1) & "\"
9 resultsSummaryFile = WScript.Arguments(2)
10 qcServer = WScript.Arguments(3)
11 qcUser = WScript.Arguments(4)
12 qcPassword = WScript.Arguments(5)
13
14 Set fso = CreateObject("Scripting.FileSystemObject")
15 Set testCases = fso.OpenTextFile(testCasesFile)
16 Set testResults = fso.CreateTextFile(resultsSummaryFile, True)
17
18 ' Build an array of test cases to use
19 arTestCases = Split(testCases.ReadAll, vbCrLf)
20
21 testCases.Close
22 Set testCases = Nothing
23
24 ' Launch QuickTest
25 Set qtApp = CreateObject("QuickTest.Application")
26 qtApp.Launch
27 qtApp.Visible = False
28 qtApp.Options.Run.RunMode = "Fast"
29 qtApp.Options.Run.ViewResults = False
30
31 Set qtResults = CreateObject("QuickTest.RunResultsOptions")
32
33 ' Connect to Quality Center
34 qtApp.TDConnection.Connect qcServer, "CCNET.DEMO", "CCNet.Demo", qcUser, qcPassword, False
35 If Not qtApp.TDConnection.IsConnected Then
36 WScript.Echo "Could not connect to Quality Center!"
37 WScript.Quit 1
38 End If
39
40 testsPassed = True
41 testResults.WriteLine("<QtpResults>")
42
43 ' Iterate through all the test cases
44 testsPassed = RunTests(arTestCases, resultsDir, testResults)
45
46 ' Close the output file
47 testResults.WriteLine("</QtpResults>")
48 testResults.Close
49
50 Set fso = Nothing
51 Set testResults = Nothing
52
53 If Not qtApp Is Nothing Then
54 ' Disconnect from Quality Center
55 qtApp.TDConnection.Disconnect
56
57 ' Close QuickTest Professional.
58 qtApp.Quit
59 End If
60
61 ' Release the created objects.
62 Set qtResults = Nothing
63 Set qtTest = Nothing
64 Set qtApp = Nothing
65
66 If (Err.Number > 0) Then
67 WScript.Echo "Run-time error occured!! " & Err.Description
68 WScript.Quit 1
69 ElseIf Not testsPassed Then
70 WScript.Echo "Tests Failed!!"
71 WScript.Quit 1
72 End If
73
74 Function RunTests(ByVal arTestCases, ByVal resultsDir, ByVal testResults)
75
76 Dim success, index
77
78 On Error Resume Next
79 success = True
80
81 ' Loop through all the tests in the arTestCases.
82 For index = 0 To UBound (arTestCases) - 1
83 If Len(arTestCases(index)) > 0 Then
84 ' Display Status
85 WScript.Echo "Running Test Case: " & arTestCases(index)
86
87 ' Open the test in read-only mode
88 qtApp.Open arTestCases(index), True
89
90 ' Get the test object and set the results location
91 Set qtTest = qtApp.Test
92 qtTest.Settings.Resources.ObjectRepositoryPath = "[QualityCenter] Subject\CCNet.Demo\Object Repository\CCNet.Demo_Obj_Rep.tsr"
93 qtTest.Settings.Resources.Libraries.RemoveAll
94 qtTest.Settings.Resources.Libraries.Add("[QualityCenter] Subject\CCNet.Demo\Functions\CCNet.Demo_Functions.txt")
95
96 ' This statement specifies a test results location.
97 qtResults.ResultsLocation = resultsDir & qtTest.Name
98
99 ' Execute the test. Instruct QuickTest Professional to wait for the test to finish executing.
100 qtTest.Run qtResults
101
102 ' Display Status
103 WScript.Echo vbTab & "Test Case finished: " & qtTest.LastRunResults.Status
104
105 success = success And (qtTest.LastRunResults.Status = "Passed" Or qtTest.LastRunResults.Status = "Warning")
106
107 ' Close the test.
108 qtTest.Close
109
110 ' Strip the unrequired results and append the test results to the overal result file
111 FormatTestResults fso, testResults, qtResults.ResultsLocation
112 End If
113 Next
114
115 RunTests = success
116
117 End Function
118
119 Sub FormatTestResults(ByVal fso, ByVal outputResults, ByVal testCaseDir)
120
121 Dim found, line, fileStream
122
123 Set fileStream = fso.OpenTextFile(testCaseDir & "\Report\Results.xml")
124
125 Do
126 line = fileStream.ReadLine
127 found = (InStr(1, line, "<Report", vbTextCompare) <> 0)
128 Loop While (found = False)
129
130 outputResults.WriteLine line
131 Do
132 outputResults.WriteLine(fileStream.ReadLine)
133 Loop While (fileStream.AtEndOfStream = False)
134
135 fileStream.Close
136 Set fileStream = Nothing
137
138 End Sub
Observations:
- Lines 7-12: The script file receives the files and Quality Center (QC) credentials to operate on as command line arguments. The arguments include a file that contains the list of test cases to run; the directory to store the test results in; the results summary file that will be used to aggregate all the individual test results and lastly the QC server and QC user logon credentials.
- Lines 14-22: We use the Scripting.FileSystemObject to build an array that contains the names of all the test cases to run. As all the test cases are stored on our QC Server, we create a file that contains the server path of all the QC test cases to run, i.e. [QualityCenter] Subject\CCNet.Demo\Executable Scripts\Create Intermediaries. We store a single test case per line and we parse the file to create an array of test cases to run.
- Lines 25-38: We use the COM Automation model of QTP to start QTP and connect to the QC server. If we are unable to connect to the QC server, we exit the script with a non-zero exit code.
- Lines 41+48: The individual test results are added to a <QtpResults> tag within the QTP results summary file.
- Lines 44, 74-113: We iterate through the test cases contained within the array and run these using the QTP automation objects. For every QTP test, we open the test case, set the test result location and run the test to finally append the test result (stored in a Report\result.xml file) to the overall result summary file.
- Lines 111, 119-138: The xml file produced by QTP contains a lot of unrequired DTD information that we do not want to include in the test result summary file. We use the FormatTestResults procedure to strip this information from the file before appending the remainder of the test results to the overall test result summary file.
- Lines 66-68: If a Windows Script Error occurred, we indicate this to the calling process by exiting the script with a non-zero exit code.
- Lines 69-72: If one of the tests failed, we indicate this to the calling process by exiting the script with a non-zero exit code.
Qtp
As the RunQTP.vbs script file takes care of all of the QTP automation, all that we need to do is to invoke the script file using Windows Script Host.
1 <Target Name="Qtp"
2 Condition=" '$(CCNetProject)' != '' ">
3
4 <Error Text="No QCUser has been specified" Condition="'$(QCUser)' == ''" />
5 <Error Text="No QCPassword has been specified" Condition="'$(QCPassword)' == ''" />
6
7 <Message Text="$(NEW_LINE)Running QTP Regression Tests" Importance="high"/>
8
9 <CreateItem Include="$(QtpResultsFolder)\$(CCNetProject)">
10 <Output TaskParameter="Include" ItemName="QtpOutputFolder"/>
11 </CreateItem>
12
13 <!-- Clear existing test results -->
14 <RemoveDir Directories="$(QtpOutputFolder)"/>
15 <MakeDir Directories="$(QtpOutputFolder)"/>
16
17 <!-- Run the QTP tests -->
18 <Exec Command="cscript.exe /nologo $(DOUBLE_QUOTES)$(QtpTestCasesFolder)\$(CCNetProject).TestCases.txt$(DOUBLE_QUOTES) $(DOUBLE_QUOTES)$(QtpOutputFolder)$(DOUBLE_QUOTES) $(DOUBLE_QUOTES)$(QtpOutputFolder)\$(QtpResultsSummaryFile)$(DOUBLE_QUOTES) $(DOUBLE_QUOTES)$(QCServer)$(DOUBLE_QUOTES) $(QCUser) $(QCPassword)"
19 ContinueOnError="true">
20 <Output TaskParameter="ExitCode" ItemName="QtpExitCode"/>
21 </Exec>
22
23 <!-- Copy the QTP test results for the CCNet build before we possibly fail the build because of Qtp test failures -->
24 <CallTarget Targets="CopyQtpResults" />
25
26 <!-- Fail the build if any test failed -->
27 <Error Text="Qtp test error(s) occured" Condition=" '%(QtpExitCode.Identity)' != '0'"/>
28
29 </Target>
Observations:
- Lines 9-11: We create a unique test results output directory based on the name of the test suite that is being run.
- Lines 13-15: We clear the old test results by deleting the test results output folder.
- Line 18-21: Windows Script host is invoked to execute the RunQTP.vbs script file. The test cases file, test result folder and test results summary file as well as Quality Center credentials are passed as command line parameters. ContinueOnError is set to true and the result of the test run is stored in a QtpExitCode property.
- Line 24: The CopyQtpResults target (see below) is invoked manually to copy the test results for inclusion in the CC.NET build report.
- Line 27: We fail the build if any of the tests failed.
CopyQtpResults
1 <Target Name="CopyQtpResults"
2 Condition=" '$(CCNetProject)' != '' ">
3
4 <CreateItem Include="$(CCNetArtifactDirectory)\$(QtpResultsSummaryFile)">
5 <Output TaskParameter="Include" ItemName="ExistingQtpResults"/>
6 </CreateItem>
7
8 <Delete Files="@(ExistingQtpResults)"/>
9 <Copy SourceFiles="$(QtpOutputFolder)\$(QtpResultsSummaryFile)"
10 DestinationFolder="$(CCNetArtifactDirectory)"
11 ContinueOnError="true"/>
12
13 </Target>
Observations:
- Line 2: We test on the CCNetProject property to determine if the build was invoked via CC.NET. The CCNetProject property is one of the properties created and passed on by CC.NET when using the MSBuild CC.NET task.
- Lines 4-8: The existing Qtp results summary file is deleted.
- Line 9: The new Qtp results summary file is copied to the CCNetArtifactDirectory.
CruiseControl.NET Configuration
One dilemma you will face with the CC.NET configuration is that the current deployment of CC.NET does not include any style sheets for formatting the QTP results to display as part of the Web dashboard reports. QTP itself uses different style sheets to format its own xml test results, so one option is to tweak their style sheets to conform to the CC.NET style sheet requirements. Another option is to use the style sheets included by Owen Rogers in a post on the CC.NET Google user group - search for a post with the title "QuickTest Pro Integration". Regardless of what style sheets you choose, remember to include these style sheets into the dashboard.config to display it on the webdashboard and also in either the ccservice.exe.config/ccnet.exe.config for publishing via the EmailPublisher.
The QtpServer uses the following CC.NET configuration. The config is really self explanatory if you are familiar with the different CruiseControl.NET configuration options.
1 <!DOCTYPE cruisecontrol [
2 <!ENTITY pub "<statistics>
3 <statisticList>
4 <statistic name='Qtp Success' xpath='sum(//QtpResults/Report/Doc/Summary/@passed)' />
5 <statistic name='Qtp Warnings' xpath='sum(//QtpResults/Report/Doc/Summary/@warnings)' />
6 <statistic name='Qtp Errors' xpath='sum(//QtpResults/Report/Doc/Summary/@failed)' />
7 </statisticList>
8 </statistics>
9
10 <xmllogger />
11
12 <email from='qtpserver@yourdomain.co.za' mailhost='smtp.yourdomain.co.za' includeDetails='true'>
13 <users>
14 <user name='BuildGuru' group='buildmaster' address='developer1@yourdomain.co.za'/>
15 <user name='Developer2' group='developers' address='developer2@yourdomain.co.za'/>
16 <user name='QtpTester' group='qtptesters' address='qtptester1@yourdomain.co.za'/>
17 </users>
18 <groups>
19 <group name='developers' notification='change'/>
20 <group name='buildmaster' notification='always'/>
21 <group name='qtptesters' notification='always'/>
22 </groups>
23 </email>">
24
25 <!ENTITY links "<externalLinks>
26 <externalLink name='Wiki' url='http://wikiserver.yourdomain.co.za/wiki' />
27 </externalLinks>">
28
29 <!ENTITY build "<tasks>
30 <msbuild>
31 <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
32 <workingDirectory>C:\Projects\CCNet.Demo</workingDirectory>
33 <projectFile>CCNet.Demo.proj</projectFile>
34 <buildArgs>/noconsolelogger /v:normal /p:QCUser=fred;QCPassword=password</buildArgs>
35 <targets>Uninstall,Install,Qtp</targets>
36 <timeout>14400</timeout> <!-- 4 hours -->
37 <logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MSBuild.dll</logger>
38 </msbuild>
39 </tasks>">
40
41 <!ENTITY header "<webURL>http://qtpserver.yourdomain.co.za/ccnet</webURL>
42 <category>RegressionTesting</category>
43 <workingDirectory>_dir</workingDirectory>">
44
45 ]>
46 <cruisecontrol>
47
48 <project name="TestSuite1" queue="Qtp.CCNet.Demo" queuePriority="1">
49 &header;
50 <artifactDirectory>_dir\Builds\TestSuite1\Artifacts</artifactDirectory>
51
52 <triggers>
53 <projectTrigger serverUri="tcp://buildserver.yourdomain.co.za:21234/CruiseManager.rem" project="DeploymentBuild">
54 <triggerStatus>Success</triggerStatus>
55 <innerTrigger type="intervalTrigger" seconds="30" buildCondition="ForceBuild"/>
56 </projectTrigger>
57 </triggers>
58
59 <state type="state" />
60
61 <labeller type="defaultlabeller"/>
62
63 &build;
64
65 <publishers>
66 <merge>
67 <files>
68 <file>_dir\Builds\TestSuite1\Artifacts\QtpResultsSummary.xml</file>
69 </files>
70 </merge>
71
72 &pub;
73 </publishers>
74
75 &links;
76 </project>
77
78 <project name="TestSuite2" queue="Qtp.CCNet.Demo" queuePriority="2">
79 &header;
80 <artifactDirectory>_dir\Builds\TestSuite2\Artifacts</artifactDirectory>
81
82 <triggers>
83 <projectTrigger serverUri="tcp://qtpserver.yourdomain.co.za:21234/CruiseManager.rem" project="TestSuite1">
84 <triggerStatus>Success</triggerStatus>
85 <innerTrigger type="intervalTrigger" seconds="30" buildCondition="ForceBuild"/>
86 </projectTrigger>
87 </triggers>
88
89 <state type="state" />
90
91 <labeller type="defaultlabeller"/>
92
93 &build;
94
95 <publishers>
96 <merge>
97 <files>
98 <file>_dir\Builds\TestSuite2\Artifacts\QtpResultsSummary.xml</file>
99 </files>
100 </merge>
101
102 &pub;
103 </publishers>
104
105 &links;
106 </project>
107
108 </cruisecontrol>
Observations:
- Lines 4-6: We use the Statistics provider to include some additional statistics for every build that includes the number of QTP tests that failed, number of tests that passed and the number of tests that raised warnings.
- Line 35: The build uninstalls and installs the new version of the application that has been copied onto a share on the local machine before running Qtp.
- Line 52-57: We setup TestSuite1 to fire once a successful DeploymentBuild has been completed.
- Line 82-87: We setup TestSuite2 to fire once a successful TestSuite1 has completed.
- Line 69+98: The combined test results summary file is merged into the CC.NET build report.
Using the above mentioned configuration it is quite possible to setup additional QTP builds on the same and different machines to create a grid of machines to run through all of your test cases more quickly. Every build will run through a different set of test cases as identified by different Qtp test case files. In our setup at work we have 3 machines running through all our test cases concurrently. I leave that as a simple exercise for the reader to complete.
Next Steps
This is probably the most complex part of the whole CI process, but thankfully the COM automation interface provided by QTP makes it not too difficult to automate the whole regression test run. The next and final post will highlight some community extensions that you can use to add some further panache to your CI Build to make it even more useful. 