Review of PushToTest Webinar: “Create Robust Selenium Tests With Page Objects”

After attending the PushToTest-sponsored Selenium: You’re Doing It Wrong webinar in May, I (foolishly, as it turns out!) followed almost all of presenter Adam Goucher’s advice, including the part about not necessarily following all of his advice! Specifically, I continued to put Selenium API calls (such as is_text_present) within the tests of my Python page object framework, albeit only for the assertions, because I thought the assertions should exist in the tests rather than in the page objects. At yesterday’s Create Robust Selenium Tests with Page Objects webinar (also sponsored by PushToTest and also presented by Adam Goucher), I learned that keeping API calls out of the tests does not mean keeping assertions out of the tests, and I learned what could be achieved by maintaining the former. The two biggest “wins” I took away from this super useful webinar are:

  • While API calls should not appear in the tests, the assertions themselves should exist there. These assertions should take the form of calls to assertion methods located within the page objects.
  • If one adheres to this “no API calls within the tests” strategy, one can theoretically switch test frameworks without having to modify one’s tests! Since I’m still using Selenium-1 but envision transitioning to Selenium-2 within the next year, this point really hit home with me.

The webinar covered a wealth of other useful info for those of us trying to implement our automated tests using the page objects model: the concept of “page portion objects”; an explanation of the “one and only place” a sleep command should appear; a recommended order of attack for developing a test within a PO framework; lots of code samples, etc.

If you’re a Selenium test developer who’s not yet an expert on the PO model, I heartily recommend you check out the screencast of this presentation, which should be available soon on PushToTest’s ScreenCast page. You can also follow PushToTest Founder & CEO Frank Cohen on twitter.com/fcohen; he’ll send out a tweet as soon as the webinar has been published.


Those elusive Selenium/Python docs!

Just over a month ago, I started transitioning from script development in Selenium/Perl to script development in Selenium/Python. I was flabbergasted by how much time I spent trying to find documentation on the Selenium/Python API! Finally, I located this very valuable wiki page.

My first response was to copy/paste the output of dir(selenium) into a file, and search in the file whenever I needed a particular type of method. Then (following the instructions on the Wiki page), I’d do a print selenium.method.__doc__ to get the full documentation on the method of interest. But that struck me as pretty cumbersome. I longed for a lengthy page full of method name and accompanying documentation, which I could search and read virtually simultaneously.

Then, while I was puzzling over what new assignment I could issue to my beginning Python students next class, it dawned on me. They could produce my longed-for full-length documentation page!

Here’s the teensy program that generates the Selenium-1/Python API:

from selenium import selenium
for call in dir(selenium):
    str = "selenium." + call + ".__doc__"
    print call + ":"
    print eval(str)

And here’s the almost identical program that generates the (unfortunately, also teensy) Selenium-2/Python API:

from selenium import webdriver
for call in dir(webdriver):
    str = "webdriver." + call + ".__doc__"
    print call + ":"
    print eval(str)

I can’t believe how long it took before I finally spent the 5 minutes necessary to develop a tool I really needed. SIGH!


PPO FTW!

In my last post, I reviewed a PushToTest-sponsored webinar I had attended entitled Selenium (You’re Doing It Wrong). One of the slides in that presentation contained the wording: “Page Objects FTW!” I’m generally leery of acronyms containing an “F” so was relieved to learn that “FTW” expands to “For The Win.”

Since I was about to start a new QA job at the time and was particularly disinterested in having to create a framework before I could delve into developing Selenium scripts, I decided to investigate using the Python Page Objects framework published by the Selenium (You’re Doing It Wrong) presenter, Adam Goucher.

It’s been only a few weeks since I was able to download and start using the framework, but so far, I’m seriously impressed. First off, the framework did just what I hoped–it allowed me to focus on developing scripts right off the bat rather than having to first design and implement a framework. Secondly, my scripts are very clean and crisp as a result of following the page object model. All of the locators are in the page object modules. Almost all of the Selenium API calls are there too. Ditto for the various messages that a page can display in response to a user action. Ditto again for the various email subject lines that are generated by user interactions with the page. (My scripts have to retrieve these emails so that they can check and/or interact with them.)

In short, I don’t envision needing to modify a completed test case very often at all going forward! Changes to a page will instead be handled via modifying just the corresponding page object. That “FTW” acronym from the webinar is quite accurate.

And to top things off, PushToTest is sponsoring another free webinar with Adam Goucher on Wednesday, July 27th: Create Robust Selenium Tests with Page Objects. Attendees will get to hear an in-depth discussion of the Page Objects model that was just touched on in the previous webinar. I highly recommend:  (a) checking out AG’s framework on github; and (b) signing up for the upcoming PushToTest webinar. And if you fail at the latter, don’t worry–the webinar will undoubtedly be posted on PushToTest’s ScreenCast Central a few days after it happens.

Now if there were just fewer bugs in the Selenium APIs, I’d be totally over the top!


Review of PushToTest Webinar: “Selenium (You’re Doing It Wrong)”

What a provocative webinar title! And what a wonderful piece of “brain candy” this Thursday morning webinar was! The entire presentation was organized as a set of two-sentence slides. The first sentence was an example of what I termed in my notes “wrong Selenium thinking.” After a minute or two of explanation, presenter Adam Goucher would display the second sentence, the complementary “right Selenium thinking.”

My favorites of these wrong/right points from Adam’s slides:

My goal is to replace humans and push to production faster.

My goal is to let humans test higher quality builds with greater efficiency.

Adam claimed that thinking like the “wrong thinking” point above is “setting yourself up for failure.” I couldn’t agree more! Awhile back, I interviewed with a company that wanted to automate their tests primarily in order to shorten their release cycle. No mention was made of the higher quality product that could be achieved if testers were freed of mundane, time-consuming, repetitive checks that could be better done by automation. Automation was only about saving time!

My application is written in X so my scripts must be as well.

I write scripts in the language that makes the most sense.

In Adam’s view, Python and Ruby are far easier languages for test engineers to pick up if they’re
not used to a lot of programming. He stated that just because your application is written in Java doesn’t mean that your scripts have to be too. I was really cheering on this one! At a previous QA gig of mine, the developers wanted Selenium automation done in PHP because that’s what all the developers used. (What about the language all the QA engineers used? Wasn’t that more relevant?!?) And at another recent interview, the QA manager wanted the Selenium automation done in Java, because that’s what all the developers used, so they’d be able to help with issues. (Mightn’t the developers know other languages besides Java? Mightn’t better Selenium support be available from the Selenium community than from the company’s developers? Shouldn’t the skill set of the QA engineers be taken into account when determining the most suitable automation language?)

I email updates to formats and extensions.

I use a site-specific plugin.

Adam mentioned that he had added a plugin API as soon as he started maintaining Selenium IDE. He recommended bundling up one’s user extensions for IDE scripts using a Firefox extension which can be hosted inside your company’s firewall. (Note: Adam made a presentation entitled “Selenium IDE Plugins – The What, How, and Why” at the Selenium Conference in early April.) I was really inspired by this wrong/right pair. I was well aware of IDE plugins (several are available, some of which I already use, at the Selenium Downloads page), but I never thought of using one to bundle site-specific user extensions!

I felt validated by many of Adam’s other points: using a cloud service is a better way to go than using Selenium Grid for cross-browser testing; locators should be placed in a shared <thing> (user extension for IDE, ini file in Python, resource bundle in Java, etc.) rather than in a script; data should be fed in externally rather than being present in scripts; individual scripts should do only one thing; and automation code should be treated the same as product code with respect to source code check-in, back-ups, etc.

And I’m eager to investigate Jenkins CI and learn more about Page Objects, both of which were roundly praised by Adam during this presentation.

To summarize, I found this PushToTest-sponsored webinar to be validating, full of good reminders, and educational / inspiring! Not only that, but the presentation style was eminently successful! I heartily advise you check out Adam Goucher’s slides, which are available now, and check PushToTest Screencasts soon–the webinar itself should be posted there within a week.


waitForSomethingOtherThanElementPresent

Every Selenium assertion can start with assert, verify, or waitFor, according to the Selenium API docs. Yet many testers never need any waitFor assertion beyond the ubiquitous waitForElementPresent.  This post elucidates the usage of a different waitFor assertion–waitForNotAttribute – a non-intuitive name if ever there was one!

To understand how my Selenium IDE demo program works, first look at the page I “tested”–a very nice online alarm clock. Then “capture” the DOM source of the six large red digits. I did this via selecting (highlighting) the digits in Firefox, then selecting View Selection Source from the context (right-click) menu….


See those src attributes in the img tags? They indicate that I selected View Selection Source precisely at 8:09:19 PM. cb.gif is a blank, c8.gif is a large red 8, c0.gif is a large red 0, c9.gif is a large red 9, cpm.gif is a large red PM, etc. The clock updates in this page are achieved via Ajax changes to the img tags’ src attributes’ values.

My Selenium test case doesn’t really test anything–it just demos how waitForNotAttribute works….

AJAX-CHANGING-VALUES
setTimeout 65000
open http://onlineclock.net/
storeAttribute e@src minutes_digit_2
echo The second digit of minutes is: ${minutes_digit_2}
waitForNotAttribute e@src ${minutes_digit_2}
storeAttribute e@src minutes_digit_2
echo The second digit of minutes is: ${minutes_digit_2}

After opening the clock page, it uses storeAttribute to save the src attribute for the element with an id attribute of e (the second digit of the minutes displayed by the clock). That URL is then echo‘ed out to the log file. Then waitForNotAttribute is called, with the e element’s src attribute specified as the first argument. waitForNotAttribute will wait until the value of the specified attribute does NOT match the pattern specified by the second argument. In other words, waitForNotAttribute is going to wait until there’s a change to the minutes being displayed. Then, the program again captures the URL for the image file that displays the second digit of the minutes on display and echoes it out to the log file.

Note that I’ve used setTimeout with an argument of 65000 milliseconds. Since the minutes on display have to change every 60000 milliseconds, the waitForNotAttribute will always wait long enough even if the Selenium program begins executing right after a new minute starts.

And now for the demo! Don’t get impatient watching it! This demo is not about things happening–it’s about proper waiting!


Good-bye, XPath! Hello, CSS!

Last night’s San Francisco Selenium Meetup was one of the two most useful I’ve attended (and I’ve attended quite a few!). Sauce Labs’ own Santiago Suarez Ordoñez gave a talk entitled CSS vs XPath Locators: An In-Depth Comparison & Why The Former Is the Way To Go.

Santi presented four reasons why we should all be using CSS locators instead of XPath locators whenever possible. CSS locators are (1) faster (on IE), (2) more readable, and (3) used by jQuery, whereas (4) “no one else uses XPATH.” I’m not sure he’s totally right about #2 and #4, but his evidence for #1 was quite compelling and amusing!

Santi presented a list of tools which one could use to help learn/use CSS locators. I was particularly intrigued by one he wrote himself: cssify.py. Here’s a copy/paste from my MacBook’s terminal window of cssify.py converting an XPath locator I used in an example elsewhere in this blog:

%mamp ./cssify.py "//select[@name='topnav']"
select[name=topnav]
%mamp

And here’s a little Perl program called convert.pl which I whipped up after the meeting in order to convert an entire HTML test case’s XPath locators into CSS ones. It calls Santi’s cssify.py program to do the conversion.

The “if” statement is admittedly ugly! The four sets of parentheses “memorize” the various components of a line like this:

<td>//select[@name='topnav']</td>

Everything up through the opening <td> container gets “memorized” in $1. The XPath expression itself winds up in $3. Since the xpath= prefix for an XPath locator is optional, if it is present, it gets stored in $2 (but then never used). Finally, the closing </td> container goes into $4. And because the regular expression delimiter is a slash, all of the slashes in the regular expression had to be backslash-escaped.

The rest of the code is quite a bit easier to follow! Here’s a demo of what it does:

And although I didn’t show this in the demo, the foothill-converted.html test case does indeed run exactly the same as foothill.html.

Santi’s talk was supposedly a re-run of a popular one he gave at the Selenium Conference in April, but for reasons I don’t understand, it is not available via video along with the other talks on the Selenium Conference site. However, you can see his slide set now and the video of his talk will appear on the Sauce Labs blog in the near future (as is the case for all of the San Francisco Selenium Meet-Up presentations).

I’m a big believer at putting new info into action asap, so going forward, I hereby pledge to show only examples with CSS locators rather than XPath locators!


Selenium-IDE & JavaScript: Abstracting Out Locators

One of the disadvantages of using Selenium-IDE to record HTML tests is that one winds up with a lot of repetitive, hard-to-maintain code. However, the generated code can be cleaned up in a variety of ways to increase its maintainability. One of these clean-up methods is abstracting out the locators to a JavaScript extensions file.

Let’s look at a trivial example: three recorded tests, each a negative test of Yahoo’s login page.

no-password
open /
type username stumbleuponqualityassurance
type passwd
click .save
verifyText //div[@class=’yregertxt’] Please enter your password
no-userid
open /
type username
type passwd correctpassword
click .save
verifyText //div[@class=’yregertxt’] Please verify your Yahoo! ID
wrong-password
open /
type username stumbleuponqualityassurance
type passwd wrongpassword
clickAndWait .save
verifyText //div[@class=’yregertxt’] regexp:Please try again using your full Yahoo! ID..*

What’s wrong with these tests? They were easy to create, they run correctly, and they do what they’re supposed to–test scenarios in which the Yahoo! user makes mistakes attempting to login. What’s wrong is that the four locators – username, passwd, .save, and //div[@class=’yregertxt’] – are all repeated in each test. What if the source of the login page changes so that a different locator is needed for one of the fields? The test developer will have to change all three tests to use the new locator. What if there are thirty tests that use that locator instead of three?!?

One solution is to abstract out the locators to a JavaScript extensions file, such as locators.js:

var yLoginLocators = {
username : "username",
password :"passwd",
signin : ".save",
errormsg : "//div[@class='yregertxt']"
};

In the tiny code fragment above, a record named yLoginLocators is declared, with four fields, one easy-to-remember identifier per locator. The value of each field is the actual locator. This locators.js file, like all JavaScript extension files, needs to be installed in Selenium-IDE via Options=>Options=>Selenium Core Extensions; then Selenium-IDE must be restarted. Once this has been done, the “okay” tests above can be revised to use the four field names instead of the actual locators, thusly:

no-password
open /
type javascript{yLoginLocators.username} stumbleuponqualityassurance
type javascript{yLoginLocators.password}
click javascript{yLoginLocators.signin}
verifyText javascript{yLoginLocators.errormsg} Please enter your password
no-userid
open /
type javascript{yLoginLocators.username}
type javascript{yLoginLocators.password} correctpassword
click javascript{yLoginLocators.signin}
verifyText javascript{yLoginLocators.errormsg} Please verify your Yahoo! ID
wrong-password
open /
type javascript{yLoginLocators.username} stumbleuponqualityassurance
type javascript{yLoginLocators.password} wrongpassword
clickAndWait javascript{yLoginLocators.signin}
verifyText javascript{yLoginLocators.errormsg} regexp:Please try again using your full Yahoo! ID..*

Voilà! With this scheme, whether there are 3 or 30 tests that use these four locators, the test developer will only have one location in which to modify them–locators.js.


Sauce Builder: First Impressions

As a follower of Sauce Labs for the past year or so, I’ve often thought that they were catering too much to the most technically sophisticated amongst the QA crowd. So, I read with more than a passing interest their April 1st blog post by CEO John Dunham, which stated in part:

To date, we have focused our products and our communication pretty heavily toward the experienced sheer-face climbers. With Sauce Builder, we begin to expand our coverage to address the needs of the valley visitors who aspire to climb.

Here was validation of my perception and another mention of their new Sauce Builder tool, which had been announced via the previous day’s post on their blog.

So herein are my first impressions of Sauce Builder! Please keep in mind that its latest version is 0.5.4, i.e., this is clearly “early days….”

  • There is not yet a way to specify a user extensions file, which means that none of the Selenium-IDE examples I’ve demo’ed in this blog can be run! This lack of support for user extensions struck me as the most significant problem with Sauce Builder.
  • Not all of the commands supported by Selenium-IDE appear to be supported in Sauce Builder. Furthermore, the error message one gets when trying to execute an IDE-generated test case containing such a command is seriously user-unfriendly. For example, one of my IDE test cases contained a call to store–the easy-to-remember synonym for storeExpression. When Sauce Builder encountered it, I saw TypeError: handler.getCommandHandler(step.method) is undefined. Huh?!?
  • Sauce Builder didn’t seem to be all that well-tested or else a lot of bugs have failed to make the cut for fixing in the tool’s first month in the public eye. For example, I’ve observed that many new Selenium-IDE users erroneously select a test case after specifying that they want to open a test suite, or vice versa. Sauce Builder’s response to each of these two scenarios was total silence!
  • The help panes are not very helpful. For example, the help for verifyTitle states: getTitle: Gets the title of the current page. There’s no mention of the argument for verifyTitle or the fact that the argument is a pattern, not a string. Similarly, the help for verifyOrdered states: isOrdered: Check if these two elements have same parent and are ordered siblings in the DOM. Two same elements will not be considered ordered. What two elements?!? And what is a novice test developer supposed to make of the command verifyOrdered being echo’ed in the help pane as isOrdered?

Since I have zero qualifications as a user experience analyst, I can’t say whether new users will find Sauce Builder easier to get started with than Selenium IDE. I can say that as an experienced Selenium IDE user, I found attempting to use Sauce Builder a bit painful. A lot of that pain could have been avoided if Sauce Labs had done more user testing and bug-fixing before releasing it to the public. I don’t think any tool at rev 0.5.4 is ready for a big announcement.

That said, I vastly prefer looking for issues than good points! So, in my upcoming post – Sauce Builder: Second Impressions – I’ll spell out the best points re: Sauce Builder.

In the meantime, if you want to read more, in addition to the two posts from Sauce Labs which I mentioned at the beginning of this post, here are two more pages of interest:

  • The Sauce Builder docs
  • The Selenium Developers (Google Groups) discussion on Sauce Builder

Selenium-IDE & JavaScript: Data-Driven Testing

In January, I made a post on the Sauce Labs blog re: how to do data-driven testing with Selenium-IDE, using XML input. But data-driven testing with Selenium-IDE can also be accomplished via storing the test data in JavaScript objects in a user-extensions file. Here’s how to create the same test case as in my January post – checking the links in the Sauce Labs home page’s footer – only with JavaScript test data this time.

  1. Download goto_sel_ide.js from the flowControl page on the OpenQA Wiki.
  2. Create the datafile containing the footer links to be clicked and the expected titles of the respective landing pages. I named my datafile saucelabs-footer.js:
  3. var testdata = [
    {linkText:"How It Works",title:"Sauce OnDemand: Cross browser testing with hosted Selenium testing in the cloud - Sauce Labs"},
    {linkText:"Downloads",title:"Sauce Labs Downloads"},
    {linkText:"Pricing",title:"Sauce Labs Pricing For Hosted Selenium in the Cloud"},
    {linkText:"Support",title:"Support: Sauce RC (Selenium RC) - Sauce Labs"},
    {linkText:"Forums",title:"Sauce Labs Forums"},
    {linkText:"Blog",title:"Selenium Testing? Do Cross Browser Testing with Sauce Labs"},
    {linkText:"Flash/Flex Solution",title:"Automate testing of your Flex and Flash Web Apps - Sauce Labs"},
    {linkText:"Documentation",title:"Documentation - Sauce Labs"},
    {linkText:"About",title:"About - Sauce Labs"},
    {linkText:"Team",title:"The Sauce Labs Team"},
    {linkText:"News",title:"Selenium News & Events - Sauce Labs"},
    {linkText:"Webinars",title:"Webinars - Sauce Labs"},
    {linkText:"Contact us",title:"Support Contact - Sauce Labs"},
    {linkText:"Service Status",title:"Status - Sauce Labs"},
    ];

    linkText and title are field names of my choosing. You can use any names you want, and also any number of fields you want; I only needed two for this simple test case.

    Note that I have used an array of records, but other schemes are possible also.

  4. Install goto_sel_ide.js and your .js file via Selenium-IDE’s Options=>Options=>Selenium Core extensions field.
  5. Utilize your .js file via creating and executing an HTML test case similar to this one:
    setSpeed 500
    open http://www.saucelabs.com
    storeEval testdata.length; tests
    store 0 index
    while (${index} < ${tests})
    storeEval testdata[storedVars[‘index’]].linkText linkText
    storeEval testdata[storedVars[‘index’]].title title
    echo DEBUG: $index = ${index}, $linkText = ${linkText}, $title = ${title}
    clickAndWait link=${linkText}
    verifyTitle ${title}
    goBackAndWait
    storeEval parseInt(storedVars[‘index’])+1 index
    endWhile

    The storeEval command gets the number of elements in the JavaScript testdata array via the length property. That number is then used to control the number of iterations of the while loop.

    The first two storeEval commands inside the loop are the trickiest/ugliest part of the test case. The outer square brackets need to hold the value of the test case’s index variable. But within JavaScript, such variables are stored in the storedVars associative array. So storedVars[‘index’] must be used to represent the numeric index for the element of interest within the testdata array.

    The last point of interest is the final storeEval command. Its first argument requires parseInt to ensure that arithmetic addition is performed as a result of the plus-sign operator rather than string concatenation. (JavaScript uses the plus-sign for both operations.)

Why learn how to do data-driven testing within Selenium-IDE using JavaScript when one can use XML data instead? The biggest advantage I can see of the approach above is that it requires only ONE outside extension – goto_sel_ide.js. The XML method requires THREE. Beyond that, JavaScript is the primary tool for extending the power of Selenium-IDE, so it’s good to be as familiar as possible regarding how to utilize it in various ways.


Selenium-IDE & JavaScript: Generating Unique New Account Names

Most of us have to create tests for registering new users at some point. In my last QA gig, I generated account names like this:

companyName+yyyymmdd-hhmmss

I was using Selenium-RC/Perl to do this, but the same thing can be accomplished via Selenium-IDE/HTML tests with JavaScript in an external user-extensions file. Here’s a demo:

1. Paste the JavaScript code below into a file; I named mine generating-unique-acct-name.js.

function generateUniqueAccountName(prefix) {
var now = new Date()
var month = zeroPad(now.getMonth()+1); //getMonth returns 0..11 so must add 1
var date = zeroPad(now.getDate());
var hours = zeroPad(now.getHours());
var minutes = zeroPad(now.getMinutes());
var seconds = zeroPad(now.getSeconds());
var milliseconds = now.getMilliseconds();
return(prefix +
"+" +
now.getFullYear() +
month +
date +
"-" +
hours +
minutes +
seconds +
milliseconds);
}


function zeroPad(number) {
if (number < 10)
{number = "0" + number}
return String(number)
}

2. Install this extension via Selenium-IDE’s Options=>Options=>Selenium Core Extensions field.

3. Select the Reload button adjacent to the Selenium Core Extensions field (a couple times as there’s a bug which sometimes prevents it from working after just one click!).

4. Create the following two-line Selenium-IDE demo test case:

storeEval generateUniqueAccountName(“SEL”) newAccountName
echo ${newAccountName}

5. Run the test repeatedly. You should see output in your Log pane that looks something like this:

[info] Executing: |storeEval | generateUniqueAccountName(“SEL”) | newAccountName |
[info] Executing: |echo | ${newAccountName} | |
[info] echo: SEL+20110415-233042980
[info] Executing: |storeEval | generateUniqueAccountName(“SEL”) | newAccountName |
[info] Executing: |echo | ${newAccountName} | |
[info] echo: SEL+20110415-23304488
[info] Executing: |storeEval | generateUniqueAccountName(“SEL”) | newAccountName |
[info] Executing: |echo | ${newAccountName} | |
[info] echo: SEL+20110415-233044909

Note that I’ve “improved” the uniqueness of the new account names being generated by adding milliseconds to the end of the hhmmss string. If you don’t want this improvement, simply eliminate the call to getMilliseconds and the string concatenation of the value of variable milliseconds.

Finally, note that I’ve “zero-padded” all one-digit values for month, date, hours, minutes, and seconds. I leave it as an exercise for you to do this for all values of milliseconds < 1000!

Here’s a video demo…