Alapadatok
Év, oldalszám:2009, 9 oldal
Nyelv:angol
Letöltések száma:6
Feltöltve:2012. november 28.
Méret:220 KB
Intézmény:
-
Megjegyzés:
Csatolmány:-
Letöltés PDF-ben:Kérlek jelentkezz be!
Értékelések
Nincs még értékelés. Legyél Te az első!
Legnépszerűbb doksik ebben a kategóriában
Tartalmi kivonat
20 November 2009 Beta Testing Mike Booth / mxb17u 1 Contents Introduction . 3 Where does it fit into the testing cycle? . 3 What actually is beta testing? . 4 History . 4 Beta Software. 4 Beta Testers . 4 The steps of beta testing. 6 Examples of well known beta tests. 6 Microsoft . 6 MMORPGs (Massively Multiplayer Online Role Playing Games) . 7 Netscape . 7 Issues . 7 Conclusion . 8 References . 9 2 Beta Testing Introduction Many people have come across beta testing in real life situations – it could have been for one of your favourite games or it could have been for your favourite operating systems. Most people understand beta testing as a pre-release version of software - but what does it actually involve? As a definition then beta testing can be seen as “allowing the software to undergo usability testing with users who provide feedback, so that any malfunctions these users find in the software can be reported to the developers and fixed.”1 This idea has proved
highly useful in larger scale software products as it can identify areas of software improvement while the project is still in its development stage. The main point about beta testing is that, in theory, it carries more validity than other forms of testing as it is real people, in real situations using the software in a real environment. Whether this is true appears to depend on the different situations that beta testing is used in. According to Joseph S Dumas and Janice Redish’s book “A Practical Guide to Usability Testing” then beta testing “seldom yields useful information about usability. Most companies have found beta testing too little, too unsystematic, and much too late.” 2 This however is in contrast with actual company policies such as Microsoft, who when developing both their Windows 7 and Vista operating systems have used beta testing to help gain real user feedback. Where does it fit into the testing cycle? Beta Testing is performed as the last part of the
testing cycle as all software has previously undergone a full cycle of internal testing, so that the application is sufficiently stable within different environments. To aid in illustrating how beta testing fits into the testing cycle: Unit Testing Integration Testing System Testing System Integration Testing Regression Testing Acceptance Testing Alpha Testing Beta Testing 3 As you can see then beta testing is performed after Alpha testing, which is where a controlled set of users test the system. Beta testing is the step on from this where a larger set of users are given the application to use in a real environment. What actually is beta testing? History The term “Beta” test stems from IBM’s “hardware test convention”, which dates back to their magnetic tape machines used in the 1970s.3 The hardware was first tested on a smaller scale to show manufacturing feasibility, known as an “alpha” test. This was then extended to a larger scale and given to non-developers to
test the functions of the hardware. This also meant that the manufacturing of the types of scale necessary could be tested. Beta Software Beta software itself usually contains all of the functionality that will be placed in the release version but does not guarantee that the software will be error free – this is known as the beta version. This allows users to use the software and test all areas of the system and help identify any errors that have been overlooked but it also allows the developer to test how the system will integrate into a larger, unstructured environment. This is known as the supportability of the software 4 Since the software is not guaranteed to be error free then the developer will abstain from taking any responsibility what so ever for any data loss due to using the software and the user will use the software at their own risk. In addition to the software then because this will be the first introduction of the software to the user then the developer will usually
contain documentation on how to use the software or a list of features that the software has. This is not done all of the time and if the developer chooses not to include this then it is not a necessity. As well as all of this then the software should have documentation, even if this is not given to the user. This will include release notes containing information such as: • • • • • The exact version number List of changes since the previous version Descriptions of any known errors A full list of system requirements Hardware (and other software) requirements Beta Testers A big problem with beta testing is finding people that are willing to test the software. Gaining peoples feedback on a piece of software is very difficult because many just want to experience the software and not provide feedback to the developers. The job of the user is to do just that – use the software and then ideally provide feedback on their experience as well as to feedback any bugs or errors that
they encounter while using the software. Feedback that the user should provide in addition to any errors that they find could be: 4 • • Their current hardware configuration that will run the software – this is needed so that the developer knows that their software is being tested on a wide range and distribution of systems. Any improvements that could be made to the software – this is so that the developer can make any minor improvements on the software before release. There are actually 2 different types of Beta Tests: • • An Open Beta A Closed Beta An open beta is one where any person can access the software and there is no limit on the amount of people that can use the software. A closed beta is one where the developer controls the number of people that are allowed to use the software. The latter is considered more useful as open betas tend to either get too many users that you don’t have the time or the resources to be able to control the feedback data coming
back in, or too few testers and you don’t get enough feedback data. This is an observation that has extended into real beta tests – using an example of 2 large Beta tests, Microsoft Windows 7 OS Beta and the World of Warcraft MMORPG Beta then both used closed beta tests to control the flow of users. 5 6 The inherent nature of closed beta tests also gives another advantage in that most people that wish to try and take part, means that they are more likely to give feedback on the software. Since a user must apply to use the software then the developer can have the user consent to giving feedback on the system, which again increases the amount of feedback. A closed beta still has issues as many beta testers will give feedback at the start using the software but will tail off towards the end of the testing cycle. Below is a graph of feedback verses time done for a piece of software called “Softner 1.2”7 5 As you can see then for the first stage of the beta test then the
testers produced a large amount of feedback and this tails off drastically for the latter stages of the test. From this same piece of software then it was found that only 55% of the testers actually gave feedback on the software.7 How the feedback is given is now much easier thanks to advancements using the Internet, it means that most software can take advantage of any errors by directly sending feedback reports to the developers web servers. This is not however a necessity and any other messaging system can be used such as forums, blogs, instant messaging etc. The steps of beta testing 1. Define a time frame for your Beta test Usually a Beta test can’t run too long but needs to be long enough to gain sufficient feedback. Depending on the size of the piece of software then the time frame can vary. Having looked at the length of several beta tests, including Windows 7, then I can say that at least 4 weeks is needed for a moderately sized piece of software. 8 2. Decide on whether it
should be a closed beta or an open beta If it is a closed beta then you will to decide on a way to vet your users so that you get a wide variety of users in different environments. A good way is to use a subscription method to allow users who are willing to go out of their way to sign up, as it is these users that are more willing to give feedback. 3. Give a method for allowing feedback for users An atypical method is a direct messaging system built into the software (Microsoft Connect) but others are available such as a forum messaging system. 9 4. Gather the feedback and analyse for any software changes that are possible, reasonable and sensible and then adjust the software accordingly. Any errors that have been found should be fixed. 5. Release the changes and then restart the beta test until the software is up to an acceptable level of software quality. 6. Once the software has been tested sufficiently then it can be deemed “Feature complete” – that is no more features to
this version of the software should be made and the software develops into the next development stage – the release candidate. Examples of well known beta tests Microsoft One of the most well known Beta tests in recent memory was the beta for Microsoft’s Windows 7 Operating System and before that Windows Vista. These were closed betas that were performed by Microsoft to gain feedback on how their operating system worked on different compatible machines and gain general public opinion on how the operating system worked.5 One of the main issues that has been prevalent with this beta test was that Microsoft did not take into account the feedback that their beta testers. 10 6 MMORPGs (Massively Multiplayer Online Role Playing Games) Popular online games such as World of Warcraft and City of Heroes performed beta tests of their games before the actual release. 6 The main reason for this was to test their hardware servers and make sure that they could handle the amount of players
that are likely play at any one time. However even with beta tests then it is not uncommon for servers to present problems on and after the release days. Netscape Netscape came under intense criticism when they released their Netscape 6 Web Browser in 2000 which was based on the free open source software by Mozilla and was only in its beta stage.11The rush in development meant without a beta test then the software was bug ridden. Issues As previously stated, one of the main issues that plague beta tests is gaining feedback from testers. This has become less prevalent in recent years with the Internet becoming a faster response and software distribution tool. With larger beta tests such as Windows 7 OS and World of Warcraft then this is certainly the case as automatic error reports can be sent, however with smaller pieces of software then gaining user feedback is still very much an issue. To get round this then many companies are making beta testers sign agreements when using the
software. Other than this some developers are enticing testers to give more feedback by giving away a free copy of the final product.12 This however could go the other way where the developers do not take account of the tester’s feedback, a situation similar to Microsoft during its Windows 7 OS beta test. Another issue or criticism of beta testing is that while beta testing can provide feedback on improvements to the software, do the developers have time to implement these changes effectively? Since beta testing is at the end of the development cycle then it can be argued that anything other than error correction is too late on in the development lifecycle. Another issue is knowing when the software is ready for a beta test. If the developer releases the software before all the features are complete is this testing still beta testing? Due to the advancements in user feedback then it seems that beta testing is carried out earlier and earlier to allow for changes to the software. 13
For instance a new MMMORPG, APB, opened a beta test in October 2009 where the game is incomplete in terms of features. This goes against the trend of beta testing where the software is complete and it is more like companies are now using real users as alpha testers. Alternatively then software can go the other way and end up staying as a beta version while developers endlessly tinker with their products. A good example of this is applications done by Google, both Gmail and google apps stayed as beta versions for several years before they became official release versions. 14 This is mainly due to companies wanting to not take responsibility for any errors that may occur with their newly developed software. 7 A final issue is that of knowing when a beta test ends and the live version of the software should begin. With the potential for a beta version for a piece software to be close to the actual version of the software and the amount of updates that continue after release then you
can ask if the software ever really is not in a beta phase? With free software then this can be more of an issue as the line is not always clear. It is, however, easier to identify with software that costs money as the beta will end when the software is at a sufficient standard that the developer feels that they can charge money for it – anything after this is maintenance of the software. Conclusion Beta testing is a fundamental part of the testing cycle and allows developers to see how their software will interact in a real environment. However with so many issues surrounding beta testing then it appears that the idea of beta testing may be subtly changing in the coming years. 8 References 1. Wikipedia article on beta testing http://en.wikipediaorg/wiki/Software release life cycle#Beta 2. A Practical Guide to Usability Testing - S Dumas and Janice Redish 3. Wikipedia article on IBM Product Test http://en.wikipediaorg/wiki/IBM Product Test 4. Taken from SQAnet – based on
NASA’ http://www.sqanet/ 5. Windows 7 news announcing the limited beta test http://windows7news.com/2009/01/08/windows-7-beta-to-be-released-on-january-9/ 6. Gamespot giving news of the World of Warcraft beta test http://uk.gamespotcom/news/6101498html 7. Journal Entry for Softner 12 software on beta testing http://www.namesuppressedcom/syneryder/2004/betapostmortemshtml 8. Windows 7 beta test closure reminder http://windowsteamblog.com/blogs/windows7/archive/2009/02/09/reminder-generalavailability-for-windows-7-beta-ending-this-weekaspx 9. Microsoft Connect website http://beta.microsoftcom/ 10. A news article on Microsoft responding to criticisms that they do not take on beta testers feedback http://windows7news.com/2009/03/02/windows-7-beta-testing/ 11. A news article on Netscape realising their beta test early http://newszdnetcom/21009588 22-141230html 12. Starcraft II beta test opt in agreement http://starcraft.incgamerscom/gallery/data/520/Beta Profile Settings Page smalljpg 13.
A news article on beta tests in a modern age by a company called tidbitshttp://www.tidbitscom/tb-issues/TidBITS-328html#lnk5 14. A news article on google aps as a beta http://googleblogblogspotcom/2009/07/googleapps-is-out-of-beta-yes-reallyhtml 9
highly useful in larger scale software products as it can identify areas of software improvement while the project is still in its development stage. The main point about beta testing is that, in theory, it carries more validity than other forms of testing as it is real people, in real situations using the software in a real environment. Whether this is true appears to depend on the different situations that beta testing is used in. According to Joseph S Dumas and Janice Redish’s book “A Practical Guide to Usability Testing” then beta testing “seldom yields useful information about usability. Most companies have found beta testing too little, too unsystematic, and much too late.” 2 This however is in contrast with actual company policies such as Microsoft, who when developing both their Windows 7 and Vista operating systems have used beta testing to help gain real user feedback. Where does it fit into the testing cycle? Beta Testing is performed as the last part of the
testing cycle as all software has previously undergone a full cycle of internal testing, so that the application is sufficiently stable within different environments. To aid in illustrating how beta testing fits into the testing cycle: Unit Testing Integration Testing System Testing System Integration Testing Regression Testing Acceptance Testing Alpha Testing Beta Testing 3 As you can see then beta testing is performed after Alpha testing, which is where a controlled set of users test the system. Beta testing is the step on from this where a larger set of users are given the application to use in a real environment. What actually is beta testing? History The term “Beta” test stems from IBM’s “hardware test convention”, which dates back to their magnetic tape machines used in the 1970s.3 The hardware was first tested on a smaller scale to show manufacturing feasibility, known as an “alpha” test. This was then extended to a larger scale and given to non-developers to
test the functions of the hardware. This also meant that the manufacturing of the types of scale necessary could be tested. Beta Software Beta software itself usually contains all of the functionality that will be placed in the release version but does not guarantee that the software will be error free – this is known as the beta version. This allows users to use the software and test all areas of the system and help identify any errors that have been overlooked but it also allows the developer to test how the system will integrate into a larger, unstructured environment. This is known as the supportability of the software 4 Since the software is not guaranteed to be error free then the developer will abstain from taking any responsibility what so ever for any data loss due to using the software and the user will use the software at their own risk. In addition to the software then because this will be the first introduction of the software to the user then the developer will usually
contain documentation on how to use the software or a list of features that the software has. This is not done all of the time and if the developer chooses not to include this then it is not a necessity. As well as all of this then the software should have documentation, even if this is not given to the user. This will include release notes containing information such as: • • • • • The exact version number List of changes since the previous version Descriptions of any known errors A full list of system requirements Hardware (and other software) requirements Beta Testers A big problem with beta testing is finding people that are willing to test the software. Gaining peoples feedback on a piece of software is very difficult because many just want to experience the software and not provide feedback to the developers. The job of the user is to do just that – use the software and then ideally provide feedback on their experience as well as to feedback any bugs or errors that
they encounter while using the software. Feedback that the user should provide in addition to any errors that they find could be: 4 • • Their current hardware configuration that will run the software – this is needed so that the developer knows that their software is being tested on a wide range and distribution of systems. Any improvements that could be made to the software – this is so that the developer can make any minor improvements on the software before release. There are actually 2 different types of Beta Tests: • • An Open Beta A Closed Beta An open beta is one where any person can access the software and there is no limit on the amount of people that can use the software. A closed beta is one where the developer controls the number of people that are allowed to use the software. The latter is considered more useful as open betas tend to either get too many users that you don’t have the time or the resources to be able to control the feedback data coming
back in, or too few testers and you don’t get enough feedback data. This is an observation that has extended into real beta tests – using an example of 2 large Beta tests, Microsoft Windows 7 OS Beta and the World of Warcraft MMORPG Beta then both used closed beta tests to control the flow of users. 5 6 The inherent nature of closed beta tests also gives another advantage in that most people that wish to try and take part, means that they are more likely to give feedback on the software. Since a user must apply to use the software then the developer can have the user consent to giving feedback on the system, which again increases the amount of feedback. A closed beta still has issues as many beta testers will give feedback at the start using the software but will tail off towards the end of the testing cycle. Below is a graph of feedback verses time done for a piece of software called “Softner 1.2”7 5 As you can see then for the first stage of the beta test then the
testers produced a large amount of feedback and this tails off drastically for the latter stages of the test. From this same piece of software then it was found that only 55% of the testers actually gave feedback on the software.7 How the feedback is given is now much easier thanks to advancements using the Internet, it means that most software can take advantage of any errors by directly sending feedback reports to the developers web servers. This is not however a necessity and any other messaging system can be used such as forums, blogs, instant messaging etc. The steps of beta testing 1. Define a time frame for your Beta test Usually a Beta test can’t run too long but needs to be long enough to gain sufficient feedback. Depending on the size of the piece of software then the time frame can vary. Having looked at the length of several beta tests, including Windows 7, then I can say that at least 4 weeks is needed for a moderately sized piece of software. 8 2. Decide on whether it
should be a closed beta or an open beta If it is a closed beta then you will to decide on a way to vet your users so that you get a wide variety of users in different environments. A good way is to use a subscription method to allow users who are willing to go out of their way to sign up, as it is these users that are more willing to give feedback. 3. Give a method for allowing feedback for users An atypical method is a direct messaging system built into the software (Microsoft Connect) but others are available such as a forum messaging system. 9 4. Gather the feedback and analyse for any software changes that are possible, reasonable and sensible and then adjust the software accordingly. Any errors that have been found should be fixed. 5. Release the changes and then restart the beta test until the software is up to an acceptable level of software quality. 6. Once the software has been tested sufficiently then it can be deemed “Feature complete” – that is no more features to
this version of the software should be made and the software develops into the next development stage – the release candidate. Examples of well known beta tests Microsoft One of the most well known Beta tests in recent memory was the beta for Microsoft’s Windows 7 Operating System and before that Windows Vista. These were closed betas that were performed by Microsoft to gain feedback on how their operating system worked on different compatible machines and gain general public opinion on how the operating system worked.5 One of the main issues that has been prevalent with this beta test was that Microsoft did not take into account the feedback that their beta testers. 10 6 MMORPGs (Massively Multiplayer Online Role Playing Games) Popular online games such as World of Warcraft and City of Heroes performed beta tests of their games before the actual release. 6 The main reason for this was to test their hardware servers and make sure that they could handle the amount of players
that are likely play at any one time. However even with beta tests then it is not uncommon for servers to present problems on and after the release days. Netscape Netscape came under intense criticism when they released their Netscape 6 Web Browser in 2000 which was based on the free open source software by Mozilla and was only in its beta stage.11The rush in development meant without a beta test then the software was bug ridden. Issues As previously stated, one of the main issues that plague beta tests is gaining feedback from testers. This has become less prevalent in recent years with the Internet becoming a faster response and software distribution tool. With larger beta tests such as Windows 7 OS and World of Warcraft then this is certainly the case as automatic error reports can be sent, however with smaller pieces of software then gaining user feedback is still very much an issue. To get round this then many companies are making beta testers sign agreements when using the
software. Other than this some developers are enticing testers to give more feedback by giving away a free copy of the final product.12 This however could go the other way where the developers do not take account of the tester’s feedback, a situation similar to Microsoft during its Windows 7 OS beta test. Another issue or criticism of beta testing is that while beta testing can provide feedback on improvements to the software, do the developers have time to implement these changes effectively? Since beta testing is at the end of the development cycle then it can be argued that anything other than error correction is too late on in the development lifecycle. Another issue is knowing when the software is ready for a beta test. If the developer releases the software before all the features are complete is this testing still beta testing? Due to the advancements in user feedback then it seems that beta testing is carried out earlier and earlier to allow for changes to the software. 13
For instance a new MMMORPG, APB, opened a beta test in October 2009 where the game is incomplete in terms of features. This goes against the trend of beta testing where the software is complete and it is more like companies are now using real users as alpha testers. Alternatively then software can go the other way and end up staying as a beta version while developers endlessly tinker with their products. A good example of this is applications done by Google, both Gmail and google apps stayed as beta versions for several years before they became official release versions. 14 This is mainly due to companies wanting to not take responsibility for any errors that may occur with their newly developed software. 7 A final issue is that of knowing when a beta test ends and the live version of the software should begin. With the potential for a beta version for a piece software to be close to the actual version of the software and the amount of updates that continue after release then you
can ask if the software ever really is not in a beta phase? With free software then this can be more of an issue as the line is not always clear. It is, however, easier to identify with software that costs money as the beta will end when the software is at a sufficient standard that the developer feels that they can charge money for it – anything after this is maintenance of the software. Conclusion Beta testing is a fundamental part of the testing cycle and allows developers to see how their software will interact in a real environment. However with so many issues surrounding beta testing then it appears that the idea of beta testing may be subtly changing in the coming years. 8 References 1. Wikipedia article on beta testing http://en.wikipediaorg/wiki/Software release life cycle#Beta 2. A Practical Guide to Usability Testing - S Dumas and Janice Redish 3. Wikipedia article on IBM Product Test http://en.wikipediaorg/wiki/IBM Product Test 4. Taken from SQAnet – based on
NASA’ http://www.sqanet/ 5. Windows 7 news announcing the limited beta test http://windows7news.com/2009/01/08/windows-7-beta-to-be-released-on-january-9/ 6. Gamespot giving news of the World of Warcraft beta test http://uk.gamespotcom/news/6101498html 7. Journal Entry for Softner 12 software on beta testing http://www.namesuppressedcom/syneryder/2004/betapostmortemshtml 8. Windows 7 beta test closure reminder http://windowsteamblog.com/blogs/windows7/archive/2009/02/09/reminder-generalavailability-for-windows-7-beta-ending-this-weekaspx 9. Microsoft Connect website http://beta.microsoftcom/ 10. A news article on Microsoft responding to criticisms that they do not take on beta testers feedback http://windows7news.com/2009/03/02/windows-7-beta-testing/ 11. A news article on Netscape realising their beta test early http://newszdnetcom/21009588 22-141230html 12. Starcraft II beta test opt in agreement http://starcraft.incgamerscom/gallery/data/520/Beta Profile Settings Page smalljpg 13.
A news article on beta tests in a modern age by a company called tidbitshttp://www.tidbitscom/tb-issues/TidBITS-328html#lnk5 14. A news article on google aps as a beta http://googleblogblogspotcom/2009/07/googleapps-is-out-of-beta-yes-reallyhtml 9
Ahogy közeledik a történelem érettségi, sokan döbbennek rá, hogy nem készültek fel eléggé az esszéírás feladatra. Módszertani útmutatónkban kitérünk a történet térbeli és időbeli elhelyezésére, a források elemzésére és az eseményeket alakító tényezőkre is.
