Contents
Plan for Non-functional testing of evolution
The non-functional testing of evolution will include the following types of testing:
Reliability Testing
Under this category, evolution client will be tested to evaluate the ability to perform its required functions under stated conditions and set of operations for a specified period of time or number of iterations.
Scope
The scope of reliability testing is to execute basic functionalities provided by evolution client i.e. Mailer, Calendar, Task and Address Book operations. The type of account in which these operations will be performed will be Groupwise, Exchange and IMAP.
Evolution/User:Sush: LDAP address book also needs to be considered, with account of any type.
Setup
6 evolution clients will be installed on a SLP 10 test machine and will be configured in the following manner:
- 2 evolution clients will be configured for a groupwise account
- 2 clients will be configured for an exchange account
- 2 clients will be configured for an IMAP account
Test Scenarios
All the 6 clients running on different test machines will be running simultaneously performing the following basic operations (test scenarios) :
1. Mailer
- Send a mail to different contacts randomly selecting from a list of email-ids.
- Send a mail with or without attachments
- Send a plain text mail with or without attachments
- Send an HTML mail with or without attachments
- Send a mail with huge text/HTML content spanning multiple pages
- Open a large text/html mail with or without attachment
- Reply to an existing mail
- Forward an existing mail
2. Calendar
- Create a new meeting request
- Add 1 or multiple invitees to a meeting request and send the appointment.
- Add 1 or multiple attachments (different file formats) in the meeting request and send the appointment.
- Accept an appointment present in the inbox.
- Discard an appointment present in the inbox.
3. Address book
- Send a mail/meeting request autocompleting the ids from the address books.
- Send a mail/meeting request autocompleting the ids to distribution lists.
- Search for the contacts in remote address books and large local books.
NOTE:
- The attachments mentioned in the above set of operations will be selected randomly from a directory containing various sizes and types (file formats) of attachments and there can be more than one attachment present in the mail.
- At this point of time only Mailer and Calendar are being considered to execute for reliability testing, later Tasks and Address book can be added.
- There are more test scenarios but for the purpose of reliability testing we are considering only a small set of operations that are performed by a normal user quite frequently.
Execution
The execution of the above scenarios will be automated by using a shell script and the input data files. The shell script will perform the operations in the evolution client using a GUI automation tool, ltfx. The currently available shell script handles the following tasks so it needs to be modified in order to add other test scenarios explained above:
- Launch evolution after setting the debug environment variables on command line
- Automates the selection of the operation to be performed i.e. sending mail, calendar OR task.
- Adds the email-ids and performs the auto-completion.
- Adds the text in the various text fields.
- Attaches files automatically.
- Send a mail (with/without attachments to different people)
- Send a meeting request (with/without attachments)
- Create a new task (with/without attachments) and assign it.
The script will be started once and will be left to run continuously for 5 days. At the end of 5 days we will be observing the state of the system and the evolution application.
Results
The result of reliability testing of evolution will consist of the system statistics (memory and CPU usage of all evolution processes) and the statistics of the operations performed (this will include the number of operations of each type that are performed by evolution client program)
There can be various circumstances under which the statistics should be collected. Some of these are listed below (Think of some more situations):
1. If the execution of reliability test script has stopped before the specified end time we need to collect the following data:
- Number of mails/appointments sent/received by evolution through the script.
- Time for which the test was running.
- Memory snapshots of evolution and e-d-s processes at specific intervals when its performing the various operations in mailer, calendar or addressbook.
- Reason for the termination OR hang.
- Stack trace at the time of termination OR hang.
2. If the execution has completed without any unexpected errors and problems we need to collect:
- Number of mails/appointments sent/received in the specified time period.
- Time taken for sending these mails/appointments.
- System state at the end of the test execution (will include the memory snapshots and CPU utilization of all evolution processes and load average of the system.
Performance Testing
This testing will be conducted for evolution to evaluate the time taken or response time of evolution to perform it's required functions (in mailer, calendar, address book and tasks) under stated conditions in comparison with different versions of evolution.
Scope
The scope of performance testing of evolution is to execute the basic functionalities provided by each of its component i.e. Mailer, Calendar, Address Book and Tasks and record the time it takes to execute each of the functions. Execution will involve creating a specific set of input data for the operations and then measuring the performance of the system. These performance tests will be executed on 2 different versions of evolution, 2.2 (the last stable version) and 2.4 (the latest stable version). The output of performace test will be the comparison of results between the 2 versions of evolution.
Setup
There will be 6 evolution clients installed on different test machines. Out of the 6 clients 3 will be evolution-2.2 running on SuSE 9.3 and 3 will be evolution 2.4 running on SuSE Linux Professional 10. Each of the 3 evolution clients (2.2 and 2.4) will be configured with any one of the IMAP, Groupwise OR Exchange accounts.
3 Evolution-2.2 clients will be configured with IMAP, Groupwise and Exchange accounts 3 Evolution-2.4 clients will be configured with IMAP, Groupwise and Exchange accounts. <br\>Evolution/user:sush And also LDAP address books. (on SSL and non-SSL connections). <br\>For the performance testing it is better to have SSL enabled connections. Also the servers should contain huge number of realistic objects (typical organization like). I mean we should not have user objects with names user1, user2... or only first names.
Data/Load Profile
We will be starting the tests with an existing set of data (mails, appointments, contacts and tasks) in each of the different kinds of email accounts (configured in evolution). We also need to create some amount of data with which the tests will be done. The generation of test data will include the various scenarios OR situations in which this data will be handled by evolution client.
So basically we will be dividing the inputs for performance test into the following:
1. Different types of users of evolution client.
EXAMPLE :
Who is actually using the client. Is it a Home user, a student, a software developer, etc.. Each of these users will use the evolution mailer in a different pattern, so we will have different input sizes.
2. Different operations that a user can perform.
EXAMPLE :
- Send Mail
- Receive mail
- Searching for mails
- Sorting of mails
- Syncing up with the server
- Delete/Move/Copy a mail
- Having message filters.
- marking a folder (mail/contacts/calendar) for offline usage, this involves buidling the local cache.
with autocompletion and without autocompletion (By disabling autocompletion => not selecting any folder for autocompletion)
- searching for a contact.
- ... Can add more ...
3. Different scenarios when the number of operations will be more OR less
EXAMPLE :
When is the mailer being used. If its during some festival season, the number of mails flowing to and from the evolution client will be very HIGH and if its a vacation period it might be very LOW. Based on this we can generate the amount of input that we need to use.
4. Number of operations performed by a particular user.
EXAMPLE :
How many mails in a typical scenario (#3 above) are being sent, received, deleted, moved, searched, sorted by a particular user (#1 above)
Conclusion
Based on the above 4 categories we can come up with some set of data that has following characteristics:
- Email size (it will include attachments also)
- Number of emails present in folders
- Number of emails being received
- Rate of retrieval of mails
- Mail filtering
- Number of contacts.
- Number of contacts with images.
- Number of distribution lists with large number of members.
- Number of contacts with all the fields filled. (not only name and e-mail id)
- ... there can be some more ...
And this data can be used for measuring the performance of evolution client with respect to the number of operations performed in a particular scenario.
Test Scenarios
Mailer
The following measurements will be taken on evolution-2.2 and 2.4:
1. Time taken for initial loading of evolution with different number and types of mails present in the INBOX [Test with 10, 100, 500, 1000, 5000, 10000, 50000 mails in the inbox]
2. Total time taken to compose and send a new mail [with OR without attachments]
''Total time taken = Time taken for (Open compose mail window + To: + Cc: [with OR without autocompletion] + write mail text + Send the mail)''
3. Time taken to open an existing mail [with OR without attachments]
4. Time taken to move a mail from one folder to another. Evolution/user:sush time taken to move a set of mails (say 100) to another folder.
5. Time taken to save an attachment [with size of attachments varying from 1KB, 500KB, 1MB, 5MB, 10MB, 50MB, 100MB, 500MB and with different types of attachments]
6. Time taken to reply to an existing mail [with OR without attachments]
7. Time taken to reply to all to an existing mail.
Calendar
Some calendar performance testing scenarios are explained in Evolution/ CalendarPerformanceTesting.
NEED COMMENTS : I am not sure if the above mentioned scenarios can also be included in the scope of this Non-functional Test Plan.
In my opinion the following measurements can be taken on evolution-2.2 and 2.4:
1. Time taken for initial loading of calendar component with different number and types of appointments present in the calendar [Test with 10, 100, 500, 1000, 5000, 10000 appointments]
2. Time taken to create a new meeting by filling in all the details in the create meeting window.
3. Time taken to accept an existing appointment in the inbox.
4. Time taken to open an existing appointment in the calendar.
5. Time taken to add new invitees to a meeting request [with OR without autocompletion of mail ids]
6. Time taken to create a new meeting with 1 or more attachments [test with attachments of different sizes and types]
7. Time taken to display all the appointments in list view.
8. Time taken to randomly open an appointment by specifying a date/ date range,
Address Book
Some of the performance test scenarios for Address book are mentioned in Evolution/AddrBook_perf.
NEED COMMENTS : I am not sure if the above address book performance scenarios can be included in the scope of this Non-Functional Test Plan. <br\>Evolution/user:sush why they can not be? any specific reasons?
Test Scenarios addition IN PROGRESS
Execution
The execution of the above scenarios will be automated by using a shell script and the input data files. The shell script will perform the operations in the evolution client using a GUI automation tool, ltfx. The script for performing the above Mailer/Calendar/Address Book operations is not currently available so we need to write it in order to run these tests.
Results
When the various operations are being performed with the evolution client, the values of the following 2 system parameters will be captured: 1. Memory usage of all evolution-* processes. 2. CPU utilization of all evolution-* processes.
Scalability Testing
Scalability testing of the evolution client will be done together with the performance testing and will mainly involve the following:
1. Measuring the time it takes for evolution to load when the number of mails in the inbox are 100, 1000, 5000, 10000, 50000, 100000.
2. Measuring the time it takes for the address book to load when the number of contacts are gradually increased from 50 to 10000 and beyond (Evolution/user:sush with most of the contacts having images).
3. Measuring the time it takes to perform a search when there are many number of mails present in the inbox.
4. Measuring the time it takes to perform a search in the address book (Evolution/user:sush both when address book is marked for offline usage and not also for remote address books (LDAP, Exchange GAL, GW SAB) having more than 10,000 entries).
5. Measuring the time it takes to load the calendar entries from a remote server.
6. Measuring the time when the number of entries in the calendar is very high. <br\>Evolution/user:sush
7. Measuring the time it takes to perform auto-completion.
As we are mainly concerned with the performance of evolution client, the scalability tests will be limited to scaling the amount of input data, so it can be performed in conjunction with the Performance testing.