- Image by justinsomnia via Flickr
One of my favorite aspects about being a tester at Atlassian is working with developers who love pushing the envelope by trying out new technology as soon as they can get their hands on it. Recently, I’ve been testing a feature that uses HTML5. Before my testing, I had heard of HTML5 as the apocalypse of testing because of the canvas tag. Canvas is, to be sure, controversial
BUT…
There is much more to HTML5 than the canvas tag.
I’m always tweeting about “dev-nial”, but if all we do, as testers, is scream bloody-murder about the canvas tag, we’re going to miss some bugs as HTML5 creeps into the web applications we test. I was pointed, in a work training session, to Google’s HTML5 Rocks web pages. This post is about the new functionality revealed in the HTML5 Rocks slides and my thoughts about testing features made with HTML5.
If you go through the slides, do yourself a favor and use a browser that has some IDE-ness to it, so you can see what happens with storage or you can inspect elements. The slides were intended to be very hands on and the site, itself, includes a playground.
Web Storage & Web SQL Database
Remember google gears? I remember when it came out and thinking that having the ability to save at least some content offline would make web stuff much more fun. Now that I live in a country with download limits, I see the ability to view web content offline from a completely different perspective. For a feature exploiting offline web storage, I would test that content is still there even if my internet connection is turned off. If you’re looking through the JS you have to test and you see “localStorage.setItem” it’s an indication that something is being stored on the client side.
Security testing is also a consideration. On the html5rocks page explaining the “web sql database” I was originally able to store an unescaped xss string in the web-storage. To be fair, on this page, the user is entering the value to be stored on their own computer, BUT this is implies that it could be possible to insert xss in a user’s web-storage without telling them. (Cheers to the html5rocks team for fixing this possible exploit…y’all might want to have a look at what happens after session timeout ;)
WebSockets bring networking up to browser-level
I’m still wrapping my head around this one, but I think it will be quite important to understand WebSockets thoroughly wherever they are used. In my brief experience with sockets, which was a few years ago, I noticed that it’s really easy to leave a socket open or to get confused about which socket is sending/receiving data. I would want to know where this is used so I can test that the sockets are open when they should be and closed when they are no longer needed.
Concurrency hits browser based testing
When I was choosing a master’s thesis, one of my possible topics was testing and concurrency. I floated this topic in conversations I had at the Seattle GTAC and got crickets (The only person who cared was a guy working at, surprise, Google). I’m guessing that since HTML5 includes “web workers” known to me in Java as “threads” concurrency will become an issue. That means race conditions, contention, etc.
Geolocation
This is something I have absolutely no experience testing. If it’s more available to developers, are they more likely to use it in applications they haven’t before? If I had to test something with this feature, I’d have to figure out how to mock a location. This would be interesting to test as it related to offline storage. If you have part of the application that relies on coordinates, but then the app goes offline, is there a way for the user to type in their coordinates?
Device Orientation
Whatever computing device you are using to read this, iPad, laptop, desktop…yes, even desktop, tilt it. If you are on a mac, you will see interesting things happen. If you are on windows, not so much. Since I’ve already noticed a difference in the behavior of the same browser on different operating systems, this is definitely something I would test across devices and operating systems. So if I’m lost in the urban jungle with my laptop, I should be able to orient myself with google maps on a mac, but if I have a Windows 7 netbook, I’ll just be screwed? [Update: I submitted this as a bug and the HTML5Rocks team pointed out that it’s because my netbook does not have an accelerometer. I’m kind of surprised that the MacBook has one. #learnsomethingneweveryday]
There are many more slides to digest at HTML5 rocks. If you look through the slides and have some thoughts about testing, I encourage you to leave a comment or, even better, write your own post. If you find a few more bugs, I’m sure Google would absolutely welcome and cherish any bugs reported. When the next version of Confluence is released, I’ll write more about my own exploits with HTML5. I was taken by surprise at how quickly it showed up for testing and very impressed by the dev who decided it was time to make use of it. If you use Confluence, I think you’re going to like the next release, and I’m not biased at all ;)
Why is canvas the apocalypse of testing? Isn’t it like any other graphics library in any other language? I can see usability and accessibility issues if buttons etc. are rendered onto canvas, but testing?
This is an issue for automated, browser-based testing. That is, automated tests using something like Selenium or Watir. You describe the scenario in your second sentence. If controls and functionality are rendered using the canvas tag, they won’t be exposed for Selenium or Watir tests. This means that if a UI is moved inside of canvas, previously written automated tests will be useless. If an application previously had 20 or 30 automated tests for an interface that are now broken because of canvas, those tests now have to be performed manually or, most likely, will not be performed due to time constraints.
I don’t mean to paint canvas as an enemy, because it’s not. It does, however, present a testability challenge. When I’ve previously heard testers discussing HTML5, it’s only ever been this particular testability issue despite the fact that there is a lot more going on with HTML5. I guess I expected that this had been discussed more online than it actually has.
Canvas is essentially the same problem for Selenium [or Watir] that Flash/Flex and Silverlight are — that is, it is a big black box that things cannot peer into directly. Thankfully, the solution is the same though I believe.
For flash/flex you have to provide ‘hooks’ (externalinterface) to pull the strings that control the stage. Canvas is ‘just’ JS and so you can provide the same sort of hooks out to things like Se to pull. And if your application happens to also use those hooks as its standard operating procedure then you are so much more off to the races.
See also http://adam.goucher.ca/?p=1049.
-adam
Thanks, Adam, for the follow-up. The post you’ve linked to is a good summary of this issue, and I’m sure others will find it helpful as well.
Nice overview of some things to consider.
Regarding web storage, one thing that quickly comes to my mind is looking for code that assumes a “set” will always succeed or that a “get” for previously stored data will always succeed. I’m not an expert on this feature, but it seems like you’d have these classic caching issues. There’s simply no gaurantee these operations will always succeed as you might assume.
Off-topic: got a new version of the HTML5 fist available that I thought you might appreciate.