Why Visual Studio Team System Isn't A LoadRunner Killer[转载]

Microsoft recently posted a job for a performance engineer, and I happened to see it. It started off with the question, "Do you have experience with any of the common performance testing applications such as LoadRunner and OcraCoke?" I thought, "huh?". For those of you not familiar with OcraCoke, that was the pre-launch code name for Visual Studio Team System. This peaked my interest further. I was interested in VSTS because several people have been asking me if I think Microsoft has come up with a "LoadRunner killer". I attended a meeting held at  a Microsoft facility and watched VSTS demonstrated. I can say without a doubt, this is NOT a LoadRunner killer. I can't see that it ever will be. Microsoft is has created a competitive product for specific cases. They are offering VSTS as an alternative to the "500 pound gorillas" load testing products. Did I mention that their license model allows unlimited virtual users? When you buy the license, its per CPU. This means as many virtual users as your server can run for one price. So why am I so confident that we won't see a mass move to this product by the Global 2000 companies in the US?

VSTS is really made with a developer's frame. of reference in mind, using Visual Studio's interface to build scripts and run tests. The LAST person I want running an integrated test on a huge project is the developer. They should performance test their code components and the code layer of the product. They should engineer performance into their widgets via unit performance testing, but I don't want them touching a system right before it goes live. Why? They have a skewed view of the testing and cannot remain neutral. Developers are only worried about their code. If they cannot figure out how to optimize their code, they will throw hardware at it before spending the time to rewrite. I've seen it time and time again. Since it's usually their neck on the line if the product gets delayed due to defects (whether quality or performance related), how many developers are going to resist the temptation to tell the CIO that they found a performance issue in the code and the product roll out needs to be delayed two more weeks? How many developers want to expose themselves or their team to the client stakeholders as the originators of the performance defects. This is why a performance engineer who is a third party needs to do this. It should be a QUALIFIED engineer who can get into the guts of the system, and who doesn't have political ties to the development staff, including reporting to the same manager. I prefer a person who represents production because that's who is going to end up with the final result and have to support and maintain it. Most QA teams report to the development manager, which is a mistake. What happens is the QA team is gagged and bound from doing their job because they are seen as holding up the project. That's why having a performance engineer report to the production director gets them out of that jam. I am seeing more and more of this in the market and I like it. On the other hand, we must not forget that performance testing is a discipline of product assurance. Too far the other direction (operations) and you're not looking at code, but CPU counters and bandwidth alone. This is also a mistake.

The biggest reason VSTS is not the LoadRunner killer is because I don't know of a single Enterprise completely encapsulated in a Microsoft world and totally compatible with a limited set of technology platforms. Once you get away from the Microsoft realm, VSTS loses most of it's potency. In fact, if you are simply using the Firefox or Opera browsers as your standard, you are out of luck. When creating web scripts, the only choices are Internet Explorer or Netscape 6. This can probably be worked around by manually modifying some headers, but that is only the beginning. What happens when you need to move into that gray area between web and "other" application (like Siebel 7.7) where there is more of a challenge because of client stuff to install and it might communicate outside of port 80? How about mySAP? Oracle financials? What if you want to test a legacy application behind Citrix? Microsoft will tell you that they have an additional feature which will allow you to expand to other protocols (and I would assume some gifted MS developers are working on this now). It's a unit testing tool, much like Nunit. The only problem is you need to be a geekazoid wedge head programmer (the kind that you leave a 6-pack of Jolt cola's for outside their door and let them work in the dark) to create the stuff needed to test against these other protocols . I don't know a lot of QA folks out there testing applications with this much time on their hands. They're already overworked and underpaid. And haven't you heard, they are the reason the projects are being held up….?!?!? It is sort of like getting LoadRunner with only the web protocol and the Winsock protocol. Everything that falls outside of pure web, is Winsock. I can hear the groans and feel the anguish even as I type this up.

The main problem Microsoft will have in developing new protocols is not limited to the technical realm. One of the real strengths Mercury has is the strong relationships at the R&D level with SAP, ORACLE, CITRIX, and others. In fact, SAP and ORACLE have standardized internally on LoadRunner. They have provided Mercury the information they need to "hook" into these custom applications and protocols and spent the last 10 years maturing and expanding these protocols so that the scripting part gets easier with each version. Because of this, you can apply the same testing methodology across projects no matter the application, protocol, or platform. Now let's think about the fact that while its entirely possible that these kinds of protocols can be developed, who's going to bet me that Oracle and SAP are going to play nice with Billy Gates and hand over the hooks into their product? Anyone? Bueller? Bueller?...

Now that I've spent the entire time whining and standing on my soapbox (again), let me say that I think VSTS has some great qualities. Just the fact that it gets all those developers THINKING about testing is a good thing. The ability to create a simple web script. directly in their tool of choice and run a test against their code - this is a VERY good thing. Being able to put a transaction around multiple page returns, adding think time, modifying in a script. view, creating parameters, and real time performance monitors are all great things. There are lots of graphs to look at an analyze after the test is completed. If you have a development shop that is COMPLETELY Microsoft centric and you are only messing with web and web services in .NET, then this could be a way for you to validate the performance of your product. It will still take a solid testing methodology to make sure you are truly eliminating the risk of production deployment. That is something Microsoft won't bring to the table either, and best left to folks like us at Loadtester. We'd love to help. On the other hand, I'm not putting away my LoadRunner skills just yet. When Oracle standardizes on VSTS, I'll gladly close my laptop and open a BBQ restaurant….on the beach!!!