Sauce Builder: First Impressions

As a follower of Sauce Labs for the past year or so, I’ve often thought that they were catering too much to the most technically sophisticated amongst the QA crowd. So, I read with more than a passing interest their April 1st blog post by CEO John Dunham, which stated in part:

To date, we have focused our products and our communication pretty heavily toward the experienced sheer-face climbers. With Sauce Builder, we begin to expand our coverage to address the needs of the valley visitors who aspire to climb.

Here was validation of my perception and another mention of their new Sauce Builder tool, which had been announced via the previous day’s post on their blog.

So herein are my first impressions of Sauce Builder! Please keep in mind that its latest version is 0.5.4, i.e., this is clearly “early days….”

  • There is not yet a way to specify a user extensions file, which means that none of the Selenium-IDE examples I’ve demo’ed in this blog can be run! This lack of support for user extensions struck me as the most significant problem with Sauce Builder.
  • Not all of the commands supported by Selenium-IDE appear to be supported in Sauce Builder. Furthermore, the error message one gets when trying to execute an IDE-generated test case containing such a command is seriously user-unfriendly. For example, one of my IDE test cases contained a call to store–the easy-to-remember synonym for storeExpression. When Sauce Builder encountered it, I saw TypeError: handler.getCommandHandler(step.method) is undefined. Huh?!?
  • Sauce Builder didn’t seem to be all that well-tested or else a lot of bugs have failed to make the cut for fixing in the tool’s first month in the public eye. For example, I’ve observed that many new Selenium-IDE users erroneously select a test case after specifying that they want to open a test suite, or vice versa. Sauce Builder’s response to each of these two scenarios was total silence!
  • The help panes are not very helpful. For example, the help for verifyTitle states: getTitle: Gets the title of the current page. There’s no mention of the argument for verifyTitle or the fact that the argument is a pattern, not a string. Similarly, the help for verifyOrdered states: isOrdered: Check if these two elements have same parent and are ordered siblings in the DOM. Two same elements will not be considered ordered. What two elements?!? And what is a novice test developer supposed to make of the command verifyOrdered being echo’ed in the help pane as isOrdered?

Since I have zero qualifications as a user experience analyst, I can’t say whether new users will find Sauce Builder easier to get started with than Selenium IDE. I can say that as an experienced Selenium IDE user, I found attempting to use Sauce Builder a bit painful. A lot of that pain could have been avoided if Sauce Labs had done more user testing and bug-fixing before releasing it to the public. I don’t think any tool at rev 0.5.4 is ready for a big announcement.

That said, I vastly prefer looking for issues than good points! So, in my upcoming post – Sauce Builder: Second Impressions – I’ll spell out the best points re: Sauce Builder.

In the meantime, if you want to read more, in addition to the two posts from Sauce Labs which I mentioned at the beginning of this post, here are two more pages of interest:

  • The Sauce Builder docs
  • The Selenium Developers (Google Groups) discussion on Sauce Builder
Advertisements

About Mary Ann May-Pumphrey

I'm a software QA automation engineer, focusing primarily on Selenium/Webdriver automation of the front end of web apps. View all posts by Mary Ann May-Pumphrey

3 responses to “Sauce Builder: First Impressions

  • Mike V

    I really wanted to like Sauce Labs, but somehow. I kept having issues with their core product. Honestly, they should have put the “bane” of their funds into development AND support. Then done everything possible to get clients up and running through their support.
    As it is.. I tried them out.. couldn’t get things running smooth and said “forget it”.. and architected another solution. You get very few “chances” with me. When I have my job and client depending on the success of my recommendations.

  • Adam Christian

    Hi Mary Anne,

    First off I apologize for what sounds like a frustrating experience. I just wanted to quickly respond about a few things. As we are a super small team and only one of us is working on Builder — it wasn’t easy to start from scratch building a test creation IDE. We plan to support a plugin architecture. It should have been made clearer that we are not focusing on porting from IDE. However as we open sourced all this code, it’s fair to say that if you were to have logged these bugs in google code – we probably would have fixed them before you could have written this blog post.

    Since we have no QA team, it’s really hard to catch all the issues and we have really been relying on the community to help us with that. We have received an amazing amount of community feedback and help with bugs that we have worked very hard to respond to.

    In terms of the comment from Mike — we have worked insanely hard to make ondemand as reliable as we possibly can and our numbers as of late have successfully reflected all that focus. I would suggest if you tried sauce ondemand 2 months ago or more and had problems, that you take a few minutes to try the service again. I think you will be pleasantly surprised.

    Thanks for all the input, we will continue doing our best to provide a good experience.

    Adam Christian
    Developer
    Sauce Labs

  • Mary Ann May-Pumphrey

    Hi Adam! Thanks for the response. However, I’m a bit confused. I had deliberately not looked for any known bugs before I wrote my review of Sauce Builder because I wanted my findings to be as objective and “mine” as possible. Now that I’ve gone looking, here’s what I’m finding:

    –No Sauce Builder bugs appear to have been filed on the Sauce Labs site. (http://issues.saucelabs.com/query?status=accepted&status=assigned&status=closed&status=new&status=reopened&order=id)

    –Only 4 Selenium Builder bugs have been filed, none of which have been closed. (http://code.google.com/p/se-builder/issues/list?can=1&q=&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles)

    Finally, exactly a week ago, I emailed you a “laundry list” of issues I found (which I “boiled down” for my blog post), but I’ve never received a response to my email.

    In closing, I certainly do not mean to pick on Sauce Builder. And I’d be willing to create formal bug reports for the issues I put on my “laundry list.” But I’m not seeing any evidence that Builder’s bugs are getting addressed. Am I missing something?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: