Automated testing and sins

It's not often you see an article on software testing where the format makes it stand out, but that is the case on this article on automated software testing. An example of how it treats the sin of Pride is below.

Picture the scene: The wool has been pulled over the eyes of Brimstone Business Application Co’s CIO. He did a deal with a big IT supplier to have their quality-centred products added to his order. At the time it seemed like a good idea – he saved the company money and had a nice round of golf rolled-in to boot. He’d fallen into the superiority trap of believing that the most expensive or most prevalent solution would always be the best, but now was beginning to realise that this particular technology was not actually compatible with his company’s needs. He’d brought this solution in, so his pride was unable to take failure. Instead he persevered until it was too late and placed unrealistic goals on his QA team, who were then forced to revert back to manual testing. As a result the project time-lines slipped, applications went out the door late and bug-ridden, which proved expensive in re-work costs and built up a huge stack of technical debt. His department was now damned to an eternity of fire-fighting the latest problems.

The punishment in Hell will be: to be broken on the wheel.

Avoidance strategy: bigger isn’t always better. Look around when evaluating new solutions.

VSTS testing - Visual Studio Team System Test Edition

If you are using Visual Studio Team System Test Edition for testing there is plenty of advice on the Microsoft site to help new testers and advanced practioners. For a start point try out

Getting Started with Team System Testing Tools

This section takes you on a tour through the tools and windows of Microsoft
Visual Studio 2005 Team Edition for Software Testers. You will see basic aspects
of the Team System testing tools, such as how to create and work with tests, the
types of tests that are available, and how to configure the testing tools.

For a range of different testing information on VSTS try the testing type information:

Working with Unit Tests
Provides links to topics that describe unit tests
and how to create them.

Working with Web Tests
Describes how to
create, edit, run, and view Web tests.

Working with Load Tests
Describes the uses of load tests, how to edit and run them, how to collect
and store load test performance data, and how to analyze load test runs.

Working with Manual Tests
Describes how to create and run manual
tests, the only non-automated test type.

Working with Generic Tests
Describes how to create and run generic tests. Generic tests wrap external
programs and tests that were not originally developed for use in the Team System
testing tools.

Working with Ordered Tests
Describes how to create
ordered tests, which contain other tests that are meant to be run in a specified
order

VSTS testing

Software Planner newsletter

If your looking for an overview of test cases for testing the user interface you could start with the Software Planner newsletters from Steve Miller. For the last two news letters he has focused on this area. In August it was "20 Useful test cases for testing User Interfaces" and this month it is "15 Useful test cases for ensuring a consistent User Interface".

Examples of the test cases include:

Screen font type: Ensure that the screen font family matches from screen to screen. Mismatching fonts within the same sentence and overuse of different fonts can detract from the professionalism of your software user interface.

Screen font sizes: Ensure that the screen font sizes match from screen to screen. A good user interface will have an accompanying style guide that explicitly defines the font type and size for headers, body text, footers, etc.

Error logging: If fatal errors occur as users use your application, ensure that your applications writes those errors to a log file, event viewer or a database table for later review. Log the routine the error was in, the person logged on, and the date/time of the error.

Error messages: Ensure that error messages are informative, grammatically correct, and not condescending.

You can sign up to receive it every month, on this Software Planner news letter request form.

Outsourced testing webinar

There is a Free Webinar on Thursday 24 September: 14.00 BST. Its entitled "Outsourced Testing – You Can Have Your Cake and Eat It".

"There are now many organisations offering managed testing and QA services, with the promise of in-depth technical and testing skills, cost savings, flexibility, accelerated delivery, and better quality.
While these contracts are entered into with the best intentions, all too often these seemingly great deals soon begin to feel expensive, inflexible, and no longer targeted to the needs of the organisation. Control of how well solutions are being tested, and the continuity of value delivery, must essentially be maintained.
Anyone considering or using outsourced testing services will want to avoid the pitfalls experienced by others via proven methods of management and working. This webinar explores some practical management techniques for setting up, or rejuvenating testing managed service arrangements, including supplier management and measurement, and tools that make outsourcing testing work a success for the customer."

Software Planner acquired by AutomatedQA

This week saw the merger of AutomatedQA with Pragmatic Software. Both companies had good testing tools in complementary spaces. TestComplete serviced the automated functional testing space while Software Planner provided testing tool support for test management throughout the SDLC (Software Development Life Cycle). The two companies had already begun to work together to provide a more integrated solution and compete with the wider testing tool offerings from companies such as HP (the Mercury testing tools) and Micro Focus (both the Compuware and the Borland (nee Segue) testing tool set).

"For too long, Application Lifecycle Management and automated testing has been out of reach for all but the biggest software development organizations", says Derek Langone, President of AutomatedQA. "The addition of the Software Planner product suite lowers the barriers to entry of cost and complexity, so every software developer, regardless of size, can leverage the immediate advantages of ALM".

Steve Miller, President/CEO of Pragmatic Software joins AutomatedQA as Vice President of ALM Solutions. "We are excited to have Software Planner join the AutomatedQA family of outstanding quality assurance products," says Miller. "Integrating manual testing and test automation into a comprehensive software lifecycle solution will allow our clients to succeed with their software development efforts while empowering teams to become more lean and productive."

Testing DOS (Denial Of Service)

If you're looking for a testing service which simulates a denial of service attack then you could try the one identified here:

Parabon, a provider of extreme-scale grid computing, is now offering to public sector organisations and private sector companies a risk-free, no-obligation "red team" test attack to help them assess the reliability of critical network assets in the face of distributed denial of service (DDoS) attacks.

The offer features a grid-based load and performance testing application that can replicate a full-scale DDoS attack. This offer is Parabon’s response to the rising incidence of DDoS attacks against U.S. websites.

It's not clear if this service extends to non US websites (such as UK web-enabled applications and websites) but it could be worth contacting them to find out.

ALM or SDLC

Is ALM replacing SDLC in marketing terms? Someone asked me that the other day and I must admit I don't know the answer. I have seen quite a few press releases recently in which testing tools are placed in the ALM context rather than SDLC.

For example, on August 17th Sys-con had an article about Seapine Software testing tools (here) under the headline Seapine Software Announces TestTrack 2010. The introduction was:

"Seapine Software, a leading provider of global quality-centric application lifecycle management (ALM) solutions, today announced the release of TestTrack 2010. This software suite delivers new features, functionality, and enhancements to the popular TestTrack Pro and TestTrack TCM products, and introduces TestTrack RM, Seapine’s new requirements management solution that manages the complete requirement lifecycle, including planning, workflow, traceability, review, change management, and reporting. All three tools seamlessly integrate with one another to provide end-to-end traceability of requirements, issues, and tests."

ALM got a couple more mentions later on the article but no SDLC was anywhere to be seen.

And on August 19th EWeek carried an article on Thoughtworks tools under the headline ThoughtWorks Brings Agile to Application Lifecycle Management with Adaptive ALM

"ThoughtWorks Studios, a provider of Agile application lifecycle management solutions and software development tools, announces Adaptive ALM, a new Agile solution for enterprise developers building ALM systems.

ThoughtWorks Studios, a provider of Agile application lifecycle management solutions and software development tools, on Aug. 17 announced Adaptive ALM, a new Agile solution for enterprise developers building ALM systems."

A stack of mentions in a few sentences. And there were plenty more in the rest of the article. So on first assessment it looks a like this is the case. But two cases is not conclusive evidence so more examples are needed before a conclusion can be drawn.

Testing as a Service for contact centers

I have just been reading about a new service from Empirix called “Testing as a Service”. As the name suggests its a testing service, but more specifically it’s a Quality Assurance (QA) solution for contact centers (I think these are what the good old call centre used to be) installing new plaforms or upgrading/refreshing existing technology.

The Empirix Testing as a Service, is scalable: it allows contact centres of all sizes (even to large contact centers) to determine the quality of the customer experience as well as the overall performance to ensure the ROI for a contact center. The testing service is highly adaptable and covers the complete contact center infrastructure.

Testing as a Service uses Empirix’s Hammer Test Engine., for both in and out-of-service testing and in-service monitoring basis. There is also a secure, real time reporting function to monitor the testing activities of Empirix’s Testing as a Service.

For more information you can visit Empirix

Testing center of excellence

The Center of Excellence (which, as we have noted in the past, has now replaced Centre of Excellence) is a concept which has been around for some time. It has gone in and out of favour as organisations have changed and adapted to different approaches for building and delivery IT.

HP software have been using the term to cover their QA and testing offerings. HP Performance Center is performance testing software aimed at supporting a performance testing center of excellence (COE). Qualtiy Center is a test management tool for managing a QA Centre of Excellence. You can get an HP white paper on Building and Managing a Quality CoE

There is also the HP appliction security center. And Paladion have announced that they have built an Application Security Testing Centre of Excellence around this HP tool, in India. It combines an IT infrastructure with experienced security testers and best practices in using HP application security center. Its purpose it to locate and fix security vulnerabilities in computer software applications throughout the full Software Development Life Cycle (SDLC).

Acceptance Test Engineering Guide

Microsoft have released a draft version of a guide on Acceptance Testing. As well as covering the testing aspects of the acceptance process it covers the collection of data in order to making a decision during the acceptance activities. It also covers preparation for acceptance testing and the approaches to acceptance testing: specifically whether you do it in a separate stage (an acecptance testing phase) or throughout the full Software Development Lifecycle (SDLC). The latter they call Incremental Acceptance Testing.

You can download a copy of the Acceptance Test Engineering Guide and provide feedback on the same website in the discussion forum.

Software testing books

Looking at software testing books, I came across some interesting lists at SoftwareQATest.com. As well as a list on software testing books list there were lists of automated testing books, QA books and a host of other IT topics like project management and risk management. But back to the software testing list. The top five books in the list, which are the ones most recommended by the site, had a mini review. For those interested in the choices they are:

1. Lessons Learned in Software Testing, by C. Kaner, J. Bach, and B. Pettichord (2001)
2. Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyen (1999)
3. Perfect Software and Other Illusions About Testing, by G. Weinberg (2008)
4. How to Break Web Software, by M. Andrews and J. Whittaker (2006)
5. Testing Applications on the Web, by H. Nguyen, R. Johnson, and M. Hackett (2003)

If you want to see the reviews, you can visit the site. Or better still get the book and make up your own mind.

Digital Britian - Telecoms infrastructure commitments by the UK Government

Digital Britian aims to connect all of Britian so that everyone can have access to 2Mbs Broadband by 2012. Its a long report at over 200 pages. So rather than try to read it all you could just go for a summary like this one which is called a tea-break guide to this planned revolution in Telecoms.

First interesting point in that article is how low Britian is in the league table of Broadband speeds. Some countries have really embraced it but the UK languishes at 21st. Then there is some information on the way the change in telecom infrastructure will be funded. But the best bit is saved to the end. It identifies not only the best word in the report (e-readiness) but also the best sentence:

"Work is well under way to create a PSN to supersede the overlapping and duplicative patchwork quilt or departmental or sectoral networks."

Sounds a good aim to me. If you want to read the other sentences you can read the full report or the executive summary at the Deparment of Culture and Media here

And for a super brief summary the two key commitments are:

"Availability of broadband has two components: the right network today and the right network tomorrow. To ensure all can access and benefit from the network of today, we confirm our intention to deliver the Universal Service Broadband Commitment at 2Mbps by 2012. This can be delivered through upgrades to the existing copper and wireless networks. We also propose public support for the network of tomorrow so that consumers in the Final Third who will not be reached by the market can enjoy next generation broadband. This will be a longer project which involves what amounts to installing a new network. The Universal Service Commitment and the Next Generation Final Third project are separate projects and need to be addressed in turn."

Software Planner: New version of SDLC management tool

Software Planner has released a version with significant new enhancements for all stages of the SDLC (Software Development Life Cycle). New features include:

• Advanced workflow support, which allows the you to configure business rules with the Software Planner SDLC testing tool, to define and manage the status and transition of data items. The example given with the new release is creating new defects: you may want to automatically assign a status of New and set mandatory requirements for some attributes (such as name and description) when a defect is raised. Once the defect has been assigned the status would change, progressing ideally to Resolved with the tool configured to require data items such as a resolution description are input before it can be closed.
• The new release of Software Planner is integrated with Microsoft Visual Source Safe. Defects can be associated with the source code that was fixed due to the defect. This allows you to determine what source code models are affected by specific types of defects, making analysis of code impact easier to understand.
• More robust reporting, including many new defect, test case, functional specification and project management reports. The newer reports allow trending of defects, test cases and functional specifications over time, allowing teams to spot trends and determine if the quality of their software is improving as the project progresses towards production. Software Planner has also been integrated with Business Objects Crystal Reports, so you can create your own reports and post them into Software Planner for others to use. You can also import reports from other SQL based databases (like Oracle, SQL Server, etc), allowing Software Planner to double as a full reporting solution for all areas of your business.
  

Governance and Silo's

I read an article about governance and silos which I found interesting. The analyst was of the view that organisations were recreating new silos of governance rather than applying successful governance styles that are already proven.

In the article they use the example of Service Oriented Architectures (SOA):

"Well, SOA, for instance. If you look at an SOA project, it’s often defined by an elite group of architects and developers who are benefiting from all the lessons of 20 years of iterative development and they then develop their own processes, almost in this little vacuum.

Theoretically, a lot of organisations really have a lack of governance in their software development processes, but even if there is a lack of governance, there is some governance there. With the SOA lifecycle, we’ve actually spent a great deal of energy defining the stages of it and while they're parallel to what should be the SDLC - the software development lifecycle - very often they're instituted by a separate group and it’s kind of like parallel lives. So, in effect, we’re creating an additional layer of governance. Just what we need when IT organisations are trying to simplify lives for themselves."

Software Planner experiencing strong early demand

Acutest, an independent testing company, has revealed a strong demand for Software Planner in the UK. Software Planner is an award winning application lifecycle management (ALM) tool that helps organisations manage all components of the Software Development Life Cycle (SDLC). This includes managing project deliverables, requirements, defects, test cases, test execution and help desk support.

“We had anticipated there would be a real demand for Software Planner in the UK, when we formed the partnership with Prgamatic Software, the US developer of the testing management tool, to bring it to the UK market.” said Barry Varley, Managing Director of Acutest. “But we have been surprised at the how fast interest has picked up. The recession in the UK has made many organisations cautious about spending on SDLC test tools. But the immediate value that Software Planner offers has led to a host of enquiries, particularly for the SaaS version of the product. Within a fortnight of announcing the Acutest and Pragmatic Software partnership, we’d signed up our first customer for Software Planner: HMD Clinical."

“HMD clinical is a software house developing databases and applications for the clinical trial industry. We operate in an extremely regulated industry that demands high quality, traceable requirements and needed a solution that enabled us to link activities from requirements gathering through design, build, testing and release,” a spokesman for HMD Clinical said. “On evaluating many different products, we found there was no shortage of excellent project, requirements, test and defect management solutions, but most were limited to only 1 or 2 of these functions. Software Planner was the only one that provided it all in a single package. Using Software Planner will ensure that we have clear visibility of the entire system validation process."

Software Planner provides features for managing all phases of the software development lifecycle within a single integrated tool. It supports all development methodology including Agile, Waterfall, Iterative, and Spiral. It is available in two versions: an enterprise version which a client installs in their own environment and a Software as a Service (SaaS) version which is hosted by Pragmatic Software.

IEEE 829 2008

IEEE 829 2008, also known as the IEEE 829 Standard for Software Test Documentation, is an IEEE standard that specifies the form of a set of documents for use in eight defined stages of software testing, each stage potentially producing its own separate type of document.

The IEEE state that:

"Test processes determine whether the development products of a given activity conform to the requirements of that activity and whether the system and/or software satisfies its intended use and user needs.

Testing process tasks are specified for different integrity levels. These process tasks determine the appropriate breadth and depth of test documentation. The documentation elements for each type of test documentation can then be selected. The scope of testing encompasses software-based systems, computer software, hardware, and their interfaces. This standard applies to software-based systems being developed, maintained, or reused (legacy, commercial off-the-shelf, Non-Developmental Items)."

Software Testing Tool for SDLC

Acutest, a leading provider of software testing services from London UK, and Pragmatic, a leading provider of application lifecycle management (ALM) solutions from Denver USA, have formed an exclusive partnership to bring Software Planner to the UK.

Software Planner is an award winning application lifecycle management (ALM) tool that helps companies manage all elements of the Software Development Life Cycle (SDLC). This includes the management of application requirements, project deliverables, testing plans, test cases, test execution, defects and help desk support. It also provides collaborative tools and interactive reporting dashboards to support change programmes spread across teams, locations and organisations. Software Planner is currently being used by over 70,000 users.

“Software Planner has helped software companies implement better software releases since 2000 and is being used in over 28 countries”, said Steve Miller, President/CEO of Pragmatic Software. “Having a local partner in the UK that are experts in software testing and automation will provide us the ability to expose Software Planner to a larger audience in the UK – backed by a local partner that can really service their testing needs. We are very excited about partnering with Acutest and our clients will be the beneficiaries of exemplary service and support”, added Miller.

“There’s real demand in the UK for an ALM tool like Software Planner”, said Barry Varley, Managing Director of Acutest. “In these difficult economic times, organisations are looking to unleash the value they have within themselves that isn’t currently hitting their bottom line. By joining up all the management, QA and testing activities throughout the whole Software Development Lifecycle (SDLC), Software Planner helps save time and cost whilst improving risk management and governance. What’s more, both the SaaS and Enterprise version are priced appropriately for these times of recession. “

SDLC testing tool

Pragmatic Software Company, a leading provider of application lifecycle management (ALM) solutions, has partnered with Acutest as an exclusive software testing tool provider of Software Planner in the UK.

Software Planner is an award winning application lifecycle management (ALM) tool that helps programme management with all components of software development lifecylce (SDLC)including managing customer requirements, project deliverables, test cases, defects, and help desk processes. Software Planner also provides collaborative tools like document sharing, team calendars, interactive dashboards, knowledge bases and threaded discussions and integrates with most of the major automated testing tools.

"We are delighted to be partnering with Pragmatic Software to bring the Software Planner tool to the UK market: both the company and the product are very impressive," says Barry Varley, CEO of Acutest. "At Acutest we're proud of our ability to create new tailored testing service solutions, to meet the complex demands of our clients, and to deliver them quickly. Software Planner adds another very effective building block to the mix and we envisage a wide range of situations in which it will provide tremendous value."

"We are excited about having a local presence in the United Kingdom as this will provide Software Planner clients access to a knowledgeable partner that can help them maximize the use of Software Planner, says Steve Miller, President/CEO of Pragmatic Software. "When looking for a partner in the United Kingdom, we searched for a company with a solid track record of outstanding performance that is knowledgeable in testing applications under a variety of different methodologies -- including waterfall, agile, and RUP. Acutest's experience with SAP integration testing was a big plus. We are excited about offering clients in the UK an awarding winning application lifecycle management tool backed by experts in testing that service their locality.", adds Miller.

About Acutest software testing services

Acutest is a specialist software testing company based in London, UK. It provides a range of testing and quality assurance (QA) solutions to help organisations realise the benefits of technology-enabled change programmes quickly.

About Pragmatic Software

Founded in 1992, Pragmatic Software Company, Inc. based in Denver, Colorado is a software development company that provides web-based project management and team collaboration software to over 70,000 subscribers in over 24 countries. For more information on Pragmatic Software, click here

SAP performance testing

Top 5 sites on Goolge UK for SAP performance testing:

SAP performance testing guidelines
Before you can test the performance of SAP applications, set up your test environment and incorporate these guidelines to produce reliable performance ...publib.boulder.ibm.com/infocenter/rpthelp/v7r0m0/topic/com.ibm.rational.test.lt.sap.doc/topics/csapreqs.html

Recording an SAP performance test
You can record a SAP test from the SAP GUI . When you record, the recording wizard automatically starts the SAP GUI and records all the interactions that ...publib.boulder.ibm.com/infocenter/rpthelp/v7r0m0/topic/com.ibm.rational.test.lt.sap.doc/topics/tsaprecordsap.html

Performance Testing - common mistakes
what happens if you do not conduct performance testing for SAP application. comment. izhar writes: 10/12/2007 #. It is good to found the techniques u have ...it.toolbox.com/blogs/sap-on-db2/performance-testing-common-mistakes

SAP performance testing
SAP performance testing, load and stress testing services and solutions for implementation projects and SAP performance tuning and monitoring. www.acutest.co.uk/acutest/sap-performance-testing

SAP - SAP Standard Application Benchmarks
Working in concert, SAP and our hardware partners developed the SAP Standard Application Benchmarks to test the hardware and database performance of SAP ...www.sap.com/solutions/benchmark/index.epx

SAP R/3 Performance Monitoring and Tuning
SAP ERP Scalability and Performance Monitoring Testing Analyzing on Microsoft Windows 2000 Reference Step-by-Step Instructions Tips Tricks.www.wilsonmar.com/sap_perf.htm

Load testing for web apps

Last year Oracle bought e-Test (a software testing suite) from Empirix. This was in order to add load and functional testing capabilities to Oracle Enterprise Manager. The e-TEST suite was aimed at testing websites and web-enabled software applications.

Since then Oracle have rebranded the suite as the ATS (Application Testing Suite) and included it as part of the SAM portfolio (Systema and Application Management).

The performance testing tool of the suite has been rebranded from e-load to Oracle Load Testing for Web Applications. What the name change loses in brevity it gains in clarity. As the product sheet from Oracle states:

"Oracle Load Testing for Web Applications is the easiest way to validate theperformance and scalability of your Web applications and Web services. It can simulate thousands of virtual users accessing the application simultaneously and measures the effect of the load on application performance without requiring a substantial hardware investment.

The realistic usage scenarios in Oracle Load Testing for Web Applications canhandle even the most complex Web applications. By utilizing a unique virtual users capability that encompasses many parameters (including configurable browser types, connection speeds, and think times), testers can interact with the Web application just like real users will to understand exactly how the application will scale under peak load conditions. The solution’s virtual users can generate multithreaded browser requests while performing rigorous functional validationunder load conditions—validation that protocol-based legacy client server testing tools cannot provide.

Oracle Load Testing for Web Applications can also be used to test the performance of Web service interfaces by simulating thousands of concurrent clients accessing SOA-based applications.