As a follower of Sauce Labs for the past year or so, I’ve often thought that they were catering too much to the most technically sophisticated amongst the QA crowd. So, I read with more than a passing interest their April 1st blog post by CEO John Dunham, which stated in part:
To date, we have focused our products and our communication pretty heavily toward the experienced sheer-face climbers. With Sauce Builder, we begin to expand our coverage to address the needs of the valley visitors who aspire to climb.
Here was validation of my perception and another mention of their new Sauce Builder tool, which had been announced via the previous day’s post on their blog.
So herein are my first impressions of Sauce Builder! Please keep in mind that its latest version is 0.5.4, i.e., this is clearly “early days….”
- There is not yet a way to specify a user extensions file, which means that none of the Selenium-IDE examples I’ve demo’ed in this blog can be run! This lack of support for user extensions struck me as the most significant problem with Sauce Builder.
- Not all of the commands supported by Selenium-IDE appear to be supported in Sauce Builder. Furthermore, the error message one gets when trying to execute an IDE-generated test case containing such a command is seriously user-unfriendly. For example, one of my IDE test cases contained a call to store–the easy-to-remember synonym for storeExpression. When Sauce Builder encountered it, I saw TypeError: handler.getCommandHandler(step.method) is undefined. Huh?!?
- Sauce Builder didn’t seem to be all that well-tested or else a lot of bugs have failed to make the cut for fixing in the tool’s first month in the public eye. For example, I’ve observed that many new Selenium-IDE users erroneously select a test case after specifying that they want to open a test suite, or vice versa. Sauce Builder’s response to each of these two scenarios was total silence!
- The help panes are not very helpful. For example, the help for verifyTitle states: getTitle: Gets the title of the current page. There’s no mention of the argument for verifyTitle or the fact that the argument is a pattern, not a string. Similarly, the help for verifyOrdered states: isOrdered: Check if these two elements have same parent and are ordered siblings in the DOM. Two same elements will not be considered ordered. What two elements?!? And what is a novice test developer supposed to make of the command verifyOrdered being echo’ed in the help pane as isOrdered?
Since I have zero qualifications as a user experience analyst, I can’t say whether new users will find Sauce Builder easier to get started with than Selenium IDE. I can say that as an experienced Selenium IDE user, I found attempting to use Sauce Builder a bit painful. A lot of that pain could have been avoided if Sauce Labs had done more user testing and bug-fixing before releasing it to the public. I don’t think any tool at rev 0.5.4 is ready for a big announcement.
That said, I vastly prefer looking for issues than good points! So, in my upcoming post – Sauce Builder: Second Impressions – I’ll spell out the best points re: Sauce Builder.
In the meantime, if you want to read more, in addition to the two posts from Sauce Labs which I mentioned at the beginning of this post, here are two more pages of interest: