YahBlog - Explore with new eyes

Difference between Fiddler and VSTS WebTestPlugin

2008-03-04T00:25:54Z

I was asked by a colleague as to why I bothered writing a Fiddler IFiddlerWebTestPlugin when I could just as well write a simple VSTS WebTest based WebRequestPlugin, so I thought I might start answering the question...

In essence the difference between the two is: (One happens before code generation, the other at runtime)

With the Fiddler plug-in you get the opportunity to insert/generate code at the source level when the .webtest xml file gets saved, which gives you plenty of opportunities to inject business related rules prior to the final code generation.

With the VSTS WebRequestPlugin you are actually providing a run-time plug-in that one has to abide by certain rules imposed by the framework. So by the time your PreRequest(..) method gets called via the framework, the PerformDataBindings method has already been done, so if you wanted to change the actual content of your Request.Body.BodyString it is already too late to inject any DataSource Bindings... which is a pity... would have been nice, but I guess I can understand that by the time your plug-in gets called a lot of plumbing has already taken place by the framework, and if you now go and change the rules, it would have to start the whole cycle all over again... pity though...

Booting Services on Vista Development Laptops

2008-02-29T18:35:07Z

As an enterprise developer, I've always had SQL Server on my laptop (which is my main development machine). For years, SQL Server has been an indispensable tool for enterprise development, since the kind of application you tend to write in this line of business always has a relational database.

One well-known drawback of having SQL Server (and others for that matter) on your laptop is that it takes up resources, and it greatly increases start-up time when you turn on your computer. While you've always been able to configure resource usage while SQL Server is running (memory consumption, processor affinity, etc.), until Windows Vista, there's not been a lot you could do about boot time.

This may be old news to some, but I just recently discovered that Windows Vista has added a fourth startup type to the already existing three (Automatic, Manual, Disabled): Delayed Start. With Delayed Start, the service will start, but not as part of the boot process. After making this discovery, I changed not only all the Automatic SQL Server services to Delayed Start but also my cumbersome ATI/Acer services that hog my boot process. This has significantly improved my boot startup time. That's a nice, free productivity gain right there...

You might find this and this list interesting...

Automated Enterprise Testing Framework

2008-02-27T16:30:53Z

AETF Overview

A few years ago, whilst on a large scalability project, we developed an automated test framework that was able to simulate a large number of concurrent multi-user enterprise environment with the aid of simulation robots. Each scenario was controlled from a "Central Core Controller" which in-turn interacted with N-many "Agents" each running on many client workstations. "Agents" were responsible for loading/starting/stopping any required BOT with bindable configuration data coming from the Central Core Controller. This Controller orchestrated that all agents ran their test schedules as per pre-configured scenarios and published (trickle-fed) its metrics back to the "Central Core Controller" on completion of each run. So, at the press of a button we could launch an army of BOTs all pounding the targeted "Enterprise Services" in a realistic simulated environment. Which BOTs tackled which armies was all centrally configured by the Controlling Core Agent, based on available client machines and user-data in the central DB.

Now why, you might ask, am I rambling on about this... well, VS2008 Team Suite now comes with most of that framework functionality available to you right now out of the box. You still, however, need to do the implementation yourself, which in itself might or might not be a trivial task based on your Application Data Requirements. But the plumbing is there. In my process of learning and comparing the product with ours for possible re-use (with and without its limitations... remember it is still a maturing technology..), for my own sake, I am documenting as I go along. In the meantime I just thought I would share some of the parallels between AETF and the VSTS deployment scenarios available to you.

VSTS Overview

You can do performance and load testing with VSTS, comprising of different products. The test and load agent solution complements VSTS and includes a controller and agent, allowing for scalable test configuration, generating extra test load distributed across several machines.

The agent is used to simulate multiple clients from a single workstation. You can use multiple agents to increase the number of clients you simulate (scale out). The controller controls the test execution of the agents and collects performance counter metrics. You can share the controller among different Visual Studio workstations in two different scenarios:

· Shared Agents (sequential mode)

o The Visual Studio workstations will share the same agents, in which case, load tests coming from different workstations will not happen simultaneously; load tests will be executed on a first come basis, with remaining tests queued up by the controller.

· Exclusively Assigned Agents (parallel mode)

o The Visual Studio workstations will use their own exclusively assigned agents, and in that case load test executions coming from different workstation machines will happen simultaneously...

For doing performance and load testing following two main options:

1. Using Single Workstation (Scaled Down Load Test Framework).

2. Using Separate Load Test Controller and Agent (Large Scale Test Framework).

Using Single Workstation (Scaled Down Load Test Framework)

In the simplest scenario, you can run load tests on your local workstation, without a separate controller and agent. In this case, VSTestHost.exe will act as both the agent and controller. This scenario is a scaled down load test framework:

1. You are a developer doing preliminary testing for a component or related set of components and you want to test performance/concurrency.

2. You want to load test an application and you have a number of virtual users that you can simulate from your workstation.

3. You don’t need to share the performance testing infrastructure. Each developer or tester is running tests from their single workstation.

Using Separate Load Test Controller and Agent (Large Scale Test Framework)

If you need to simulate more users beyond what a single workstation can handle or if you need to create a shared environment for performance testing, then it will be required to implement a scalable test configuration with controller and agent(s). The Visual Studio workstation acts both as a test development platform and local load generator; the Visual studio load agent acts as a distributed load testing platform, with controller hosted by QTController.exe and agent hosted QTAgentService.exe acting as a dispatcher and process QTAgent.exe acting as the load generator engine.

Scenario 1 – Single Workstation Scenario

Key Characteristics

When to use

ü It’s a scaled down version of the load testing platform.

ü Visual Studio acts as development and load testing platform.

ü The test is hosted by local Visual Studio VSTestHost.exe process.

Ø For testing scenarios with limited scope like performance testing components or assemblies on their local workstation.

Ø For load test with limited number of virtual users.

Ø For cases when you cannot handle the high number of virtual user but you need to do some preliminary performance testing that allows finding bugs earlier in the development cycle, before handing the code to the test team.

Key Points

The client is used to develop tests, select tests to run, and view test results. The client is also used to execute the load and collect the test results.

With this configuration the execution of tests is affected by any locally running processes, which may skew performance testing results.

If the application being tested is in an isolated private VLAN, it forces Visual Studio to be installed in machine in the same VLAN that can access the application.

For data collection it is required that machine running Visual Studio to have access to machines being monitored for performance data.

It is not possible to disconnect from the machine running Visual Studio.

Visual Studio machine cannot be closed without interrupting the test.

If the test crashes VsTestHost.exe there is no restart functionality.

Using Visual Studio to generate load, depending on the number of virtual users, the think times and the transaction response times may cause processor bottleneck and skew the performance testing results.

Using Visual Studio and results database in the same machine may cause memory pressure or disk IO activity, depending on the amount of data collected and sampling rate. This may skew the performance testing results. Consider increasing memory or using better disk system.

Using Visual Studio load test an application using Connection per user mode may cause memory pressure, depending on the number of virtual users. In this case each virtual user will be represented by a running thread. Consider increasing memory to handle more users. The connection pool model conserves the resources on the load test agent by sharing connections to the web server among multiple virtual web test users, with less memory footprint. However if the user load is larger than the connection pool size, then web tests executing on behalf of different virtual users may need to wait before issuing a request when another web test is using the connection. Consider tracking the counter load test performance counter “Avg. Connection Wait Time”. This number should ideally be a less than the average response time for a page. If it is not, then the connection pool size is probably too small.

It does not allow sharing the performance testing lab infrastructure.

Scenario 2 – Local Controller, Local Results Database, and Remote Agent

Key Characteristics

When to use

ü The Visual Studio controller service and the results database run on the same machine.

ü It is not possible to disconnect from the machine running the tests, because controller service and visual studio run in the same machine.

ü Visual Studio can be closed without interrupting the test, because controller service runs in a separate process and will continue to collect performance metrics, and controlling the test execution on the agents.

ü The agent runs in a separate machine.

ü It does allow for sharing the performance testing lab infrastructure. Other Visual Studio machines can connect to the controller and execute load tests on the agents. If Visual Studio workstations share the same agents among themselves, the load test executions coming from different workstation machines will not happen simultaneously; load tests will be executed on first come basis, with remaining tests queued up by the controller. If Visual Studio workstations will use their own separated exclusively assigned agents, the load test executions coming from different workstation machines will happen simultaneously.

Ø To share machine with Visual Studio and the controller.

Ø When memory and disk are not a limiting factor when hosting the Visual Studio for development, the controller and results database in the same machine.

Ø When moving the controller to different VLANs is not a limiting factor, because Visual Studio is attached to the same machine than the controller.

Key Points

It enables you to share development machine with the controller.

Notes

Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

Scenario 3 – Remote Controller, Remote Results Database and Remote Agent(s)

Key Characteristics

When to use

ü The Visual Studio is in a separate machine.

ü The controller, agent and the results database run on the same machine.

ü Optionally other agents can run a separate machine.

ü It is possible to disconnect from the machine running Visual Studio, because Visual Studio and controller service run on separate computers.

ü Visual Studio machine can be closed without interrupting the test, because controller service runs in a separate process and will continue to collect performance metrics, and controlling the test execution on the agents.

Ø To allow mobility for machine running Visual Studio.

Ø To reduce memory and disk IO consumption on machine running Visual Studio for development.

Ø To share machine with controller and agent.

Key Points

The controller runs on separate machine and is used to manage the agents and collect the test results.

One agent runs on the same machine as the controller.

Installing an agent on the same computer as the controller can interfere with results collection. If you choose to do this, use the Agent Weighting property to reduce the amount of load generated by the agent installed on the same computer as the controller. It is recommended that you install the agents on their own computer. Anything that is processing at the same time the agent is running affects the accuracy of the test. Agent weighting reduces the impact, but inaccuracies are still introduced. The inaccuracies will be determined by the amount of resources the agent is using, processor and memory, and the sampling rate of data collection during load test

It is possible to use Visual Studio for development in a mobile workstation or laptop because Visual Studio and controller are in separate machines.

Notes

Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

Scenario 4 – Remote Controller, Remote Results Database and Multiple Remote Agents

Key Characteristics

When to use

ü The Visual Studio is in a separate machine.

ü The controller/results database and agent(s) run on the separate machines.

ü It is possible to disconnect from the machine running Visual Studio because Visual Studio and controller service run on separate computers.

Ø To allow mobility for machine running Visual Studio 2005.

Ø To reduce memory and disk IO consumption on machine running Visual Studio for development.

Ø To reduce resource consumption on agent, that could skew performance testing results.

Ø Performance data collection on same machine as the controller may cause memory and disk IO consumption. Consider adding memory and/or improving disk subsystem.

Key Points

The controller runs on separate machine and is used to manage the agents and collect the test results.

Agents run on the separate machines.

Using Controller and results database in the same machine may cause memory pressure or disk IO activity, depending on the amount of data collected and sampling rate. Consider increasing memory or using better disk system.

Notes

Consider moving the results database to a dedicated machine, to reduce memory and Disk IO consumption.

Nifty Software Guidance Wiki

2008-02-26T20:03:48Z

If you are looking for some short and to the point guidance in the field of software engineering, have a look at the "Guidance Share". J.D. Meier has spent some of his vacation time putting together a summary of some of his experiences gained in the trenches revolving around the fields of "software engineering guidance" as well as "test user experiences" in form of a Wiki.

What can be found on the Guidance Share:

Browsable nuggets for software performance and security
Durable, evolvable frames for security and performance (think of these as maps)
How Tos, Guidelines, Checklists and much more...

"My biggest lesson learned is how difficult it is to integrate information and make it useful, even when you own it. It's one thing to have a snapshot of information that's useful for a given point in time; it's another to create a stable backdrop with a firm foundation that can evolve over time. The key is factoring volatile from stable information, and enabling them to play well together." J.D. Meier.

Technologies covered:

.NET Framework 2.0 – ADO.NET 2.0 – ASP.NET 2.0
.NET Framework 1.1 – ADO.NET 1.1 – ASP.NET 1.1 – Code Access Security (.NET 1.1) – Enterprise Services (.NET 1.1) – Interop (.NET 1.1) – Remoting (.NET 1.1) – Web Services (ASMX 1.1) – XML (.NET 1.1)
SQL Server 2000

Pattern Matching Experience - The Effective Software Architect

2008-02-23T23:10:17Z

It's an ongoing journey...

By no means do I categorize myself as a top notch architect, but I do constantly aspire to be one, and I know a good one when I see one. Over the years I have had the privilege of working on many and diverse large scale systems. In the process I have had the honour of working with extremely talented individuals who not only are good developers, but also good architects and communicators. I have also had the opportunity to write and rewrite/refine my own code as well as, as the years pass by one gathers experience by greater exposure to the diverse systems out there. One can quickly start identifying the good and bad practices (good that you can enhance on; or bad that you should avoid), this besides other common sense approaches, are what gets used as content that will feed and grow your pattern matching library repertoire.

Credibility and Integrity

In order to further yourself as an aspiring architect you need to gain credibility. This can be done in many ways, but the acid test is that you must talk from a position of authority. i.e. You must have solved or understood each problem context/scenario. When you recommend an action you should know it from first hand exposure (perhaps state your sources and how it relates to the current scenario). If not the chances of your advice will fall on deaf ears, as people tend to ignore others that make un-researched statements/actions that may have led them down the wrong path. (I have this wonderful saying but find it difficult to express it in public... but here goes... "We the developers share from this common pool of pure and undisputed knowledge. Treasure it, as it is sacred... but, don't dare to piss in the pool of shared knowledge", this will be a sure-fire way of decreasing your credibility... hear cry wolf... "It is better to be approximately right than precisely wrong.", people will respect you for that.

The passion to make life easier for others

The universe of IT project development role-players tends to conspire with each other with the intention of succeeding. Typically, what's been observed are developers and designers stretching themselves in different ways. Some designers can go further in understanding the challenges developers face. And some developers can relate more effectively to the challenges designers face. This applies equally as well between Abalysts and Developers... Having tools where there’s an overlap, where there isn’t an absolute boundary between a design tool and a developer tool, is extremely important and helpful. It does not help if we have a gap between the different players in the lifecycle of any project. We need to constantly talk a common non-ambiguous language that is simple and to the point, but understood by all role-players.

Pattern Matching - based on facts

In my opinion, being an architect is really about pattern matching. It’s about being exposed to enough tools and techniques of the trade that over time you start to develop a toolkit/library of different patterns that work in different situations under different contexts. This is true of software architecture and probably other types of architecture as well. Whether you build bridges or design buildings, you’re trying to apply design patterns to a given situation. You don't want to keep on re-inventing the wheel... Stand on the shoulders of those that have come before you. You also need to be able to think laterally and out of the box. Before deciding on an approach, always compare to past solutions, then evaluate possible completely new approaches, by removing many of the known assumptions. Sometimes this can lead one to questioning the base assumptions and even come up with a simpler design.

Once upon a time there was a frog, sitting on a lily pad. The frog wanted to get to the next lily pad, but the pond was filled with crocodiles. The frog started to think out loud - I wish I knew how to get to the next lily pad - when the wise old Owl sitting above in a tree heard the frog and said - why don’t you fly? I can’t fly, exclaimed the frog. The Owl then explained to the frog that if she could dream it - it would happen. The frog believing the all knowing wise owl, decided to give it a try. Of course the frog did not fly and fell into the middle of the pond, and betwixt all the crocodiles. Looking up at the owl the frog anxiously remarked, you silly owl, you knew that I could not fly!! The owl simply hooted - I deal in concepts, you have an implementation problem.

Pay your dues

One's role becomes essentially a cross-cutting one. By that I mean understanding how customers are using multiple products together, and then asking myself what patterns I see. What’s the smallest possible thing I could suggest to a product team that they could do to re-architect their product in order to minimize seams with other products? Or from a business perspective, what’s the smallest possible thing I could overlay on these products to add value for our customers, and advantage our solutions in the market. At the very highest level, my advice to aspiring architects is, don’t jump in too quickly. You need to do your time as a programmer to understand the different patterns that are out there, and recognize the attributes of a well-architect system, in order to raise yourself to the next level of abstraction in the solutions you’re building. No task should be to trivial to be undertaken.

Experience - Staying in touch

It especially pertains to the things that nobody at the architecture level likes to think about. For example, performance characteristics, IO characteristics, reliability characteristics… You might have had experience with a system that works well given a certain level of complexity, but if used in a more dynamic environment it could be too fragile. You only learn that through experience.

It’s really an interesting combination. The easiest and most natural thing is to just talk to people in your sphere of influence, and gain exposure to different technologies as a part of your job. But in order to be successful long term, you need to stay in touch with the trends and what’s going on externally, especially staying in touch with what customers or individual users are saying and doing.

Awareness of peers and the Industry

Spend a fair amount of time reading blogs, tracking specific influencers that have very interesting voices both related to your products and completely unrelated to your products. Go to a combination of ‘head’ and ‘tail’ conferences. Attend a few where the "known" industry influencers congregate. These allow one to track major competitive issues, or at least the outward presentation of what competitors might be saying and doing. But also enjoy the tail conferences, where you can get a closer-to-the-ground perspective on what’s really happening. Meet people who are just out of school, who have startups and understand the kinds of technology choices they’re making and why. Read credible mentor blogs as well as books. Keep on working at it...

Defining characteristics of an effective software architect

At the highest level, my advice to an aspiring architect is find the right balance of focus on internal and external trends that will give you the perspective you need. The most effective architects I’ve dealt with are the ones who’ve paid their dues. These are the architects who’ve spent time in the trenches building and debugging fairly complex systems. They are not afraid of being wrong and will admit when they do not, but the are also quick learners. You can learn a lot about how things work by fixing other people’s bugs. When something fails and the person has left the company, you can learn a lot by either reverse engineering or looking at the documentation. The more systems that you can learn from the inside out, the more you can develop an understanding for bad practice and good practice design patterns. As I mentioned earlier, it’s this library of patterns in your mind that will define you as an architect.

Reader SDK Released!

2008-02-23T19:57:24Z

The Syndicated Client Experiences Starter Kit & Reader SDK (a.k.a. the same technology that is used to build the Architecture Journal Reader) has now been released on http://www.windowsclient.net.

This is a great way to build offline, immersive, and searchable client applications that can aggregate multiple feeds into a single experience.

The Architecture Journal Reader

2008-02-23T19:50:51Z

With already 14 issues and over 60,000 subscribers, you are probably already familiar with the printed and PDF versions of the Microsoft Architecture Journal. If not, check this out...

At the Strategic Architect Forum (SAF) held in Redmond November 2007 they released a new way of reading the Journal: "The Architecture Journal Reader". Based on WPF (Windows Presentation Foundation) and the upcoming NewsReader SDK, the Architecture Journal Reader delivers an offline, immersive, and searchable reading experience for all issues of the Journal, available in 4 languages! The Architecture Journal Reader automatically synchronizes with new content as it's released, enables annotations, includes a reading list of favourite articles, and even supports feedback on articles through a set of online forums.

You can install the beta version from here !

Next: We should start using the same mechanism for our local SA Architect content...

Some Vista Tips...

2007-11-18T22:29:40Z

A collection of some of the links I have looked at:

Some Vista developer resources: