Thoughts on Testing with Selenium, Python, etc

Recently, for client projects and for our own Complete UI code, we have been looking more and more towards selenium testing for functional testing of the application. However, it's interesting to see how different people approach testing, that it seems that people forget the differences between what a Unit Test is and what a Functional Test is.

A unit test is where you test a method for a desired result. This is where each method is tested to make sure that it produces the same expected result each time. When I did QA earlier, I heard developers tell me that the main purpose of Unit Testing was just to test banking software to make sure that it produced the proper result, and that because it wasn't banking software, it didn't matter. This isn't true at all, and in fact Unit Testing should be done at the beginning of a project. This is why I am partial to rspec, since it keeps things clean, and allows me to properly think about what I am trying to do with my code before I go off and do it. This saves a lot of time at the end, where looks matter.

Then there is functional testing. Normally this is testing that doesn't get done, or testing that would traditionally be done by your beta testers or some other form of unforgiving users who are not going to use your app because it does that stupid thing where it deletes the values of the fields and doesn't notify them whether their data was accepted. This is what selenium is for. Selenium is software that fills this gap, since it allows you to test the interface before and after it's styled and it forces you to think about making your site easy not only for you but for other people and robots to navigate as well.

The reason I mention robots is beacuse that's what Selenium is. It will not be able to beat the captcha, and you will be stuck testing certain things yourself. However, if you wrote unit tests for your authentication or comment systems, you would know that it does work, and this wouldn't necessarily be where you are looking to do functional testing.

When doing functional testing, the best thing to do is to try to take a black-box approach. The questions that you need to ask is this:

  • What are you testing?
  • Why are you testing this?
  • Is this the most efficient way to test this?

What are you testing?

This is important. Knowing the goal of the test is part of the battle. If you have to make sure that a site is serving something that the browser will use that you don't have access to in Selenium, like browser add-ons, then you need to re-assess how you are testing it. For example, in some cases, I write code in Python using only unittest, the Expat XML parser, and httplib so that I can make sure it's
serving XML files correctly that would otherwise be used by the web browser (or Google Earth KML).

h1 = httplib.HTTPConnection("localhost", 8080)
h1.request("GET", "/gearth_map/51/show")
resp = h1.getresponse()
self.assertEquals(resp.status, 200)
xmlData =
xparse = xml.parsers.expat.ParserCreate()
self.assertTrue(xparse.Parse(xmlData) > 0)

If we are using a browser that can read KML files, it should read the header and the header should have the right mime type. We don't test for that here, but we do test to make sure that we are
returning something that can be parsed as XML instead of bad data that can't be parsed. This perhaps isn't the best example of this practice, but is a good idea.

Remember that you are testing the site, and that you are NOT testing the browser. It's possible that the browser has a bug, but in that case, it's probably a good idea to maintain a level of browser agnosticism, and that any browser-specific stuff should be tested using jsUnit.

Why are you testing this?

Occassionally there will be tests that make no sense. They may test nothing, or they may be so trivial that testing them may be pointless, and would inflate your test results. This could also be something that has been unit tested or has been tested already. Doing the same thing over and over and expecting a different result is insane, and it's important that we keep a level of sanity. Therefore, repeating ones self when writing tests should be avoided and if you find that a step is being repeated in numerous tests, it's probably a good idea to abstract that out into its own test and to move up that test's priority since this test failing will cause all tests that depend on this step to fail as well.

Is there a better way to do this?

Selenium is supposed to be a substitute for point and click testing. However, it's a robot that will only do what you tell it to. It should not know what is on the server side, and it should not be able to get past systems that verify whether you are a robot or not, such as Captchas. This is where manual testing should happen.

Writing tests isn't as much fun as writing code. The more time that you spend writing tests, the less time you spend writing code. However, the more tests you write, the better your code tends to get. Here are some tips to ease the pain of selenium testing:

Try to use Firefox and Safari for testing as much as possible

You are testing a web application. Selenium can not tell whether a class looks right when it's styled a certain way. It is not subjective, it is objective. The reason that you should use Firefox or Safari for the primary browser to be tested for functional tests is the fact that these browsers have getElementsByClassName. This is probably the most useful Javascript method for gathering elements to be checked. Have a collection of elements in a table and you need to verify that they all showed up? Getting the length of the collection that this method gets is the easiest way to do it. Need to make sure that certain fields are displayed or not displayed? Checking to see if this method isn't nil will work. This will make life so much simpler for testing.

The lack of getElementsByClassName is Internet Explorer rather irritating, and you may want to add it yourself, either with an existing framework, or a homebrew hack. However as far as testing goes, this may cause bigger problems than you think, since your version of getElementsByClassName may contaminate other JS code and cause weird results that may not actually be errors but may be caused by you trying to be clever. Don't do this! If the app uses a framework that does this, it's fine, but otherwise try to avoid adding things that could lead to false positives.

Elements that need to be verified should have IDs or classes assigned to them

A lot of functional testing requires navigating the DOM. The less crawling through the DOM the test does, the faster the result can be arrived at, and the sooner we can get the tests done so that we can
get the actual code fixed. This also makes the code MUCH easier to be styled if the UI is deemed ugly or wrong. This also allows you to automate IE testing easier so it's easier to find elements using xpath or by using the DOM.

Keep the tests short. One Test per Class:

Python is full of many different test runners of various quality. If you need to write a log of what was done, it's a good idea to make sure that they are broken down this way, otherwise no tests will be run and you will have an empty file. Also, in python there are many different runners, such as XmlTestRunner, HTMLTestRunner and TextTestRunner that work this way. I recommend XmlTestRunner for Python because of it's compatibility with CruiseControl. This makes it easier to have Java-based tests and Python based tests to be on the same machine.

Use Selenium IDE as a guideline only:

Selenium IDE is a great add-on to Firefox. The thing is that most of the time the Python/Java/Ruby/whatever that it produces is not suitable for anything past the most basic tests. This is painfully obvious if you have something like the Nitobi Grid where IDs are dynamically generated and are progammatically selected. If you rely on this without the ability to seriously customize the tests in Python or Java, you will hit a wall hard.

Have Patience:

Selenium tests tend to take over the computer you test it on as far as workspace. This is fine. I run my selenium tests in a VM so that I can do other things and let Selenium have free reign on my browsers and screen space. If not, and you're testing a batch of processes, then I recommend walking off for a while and coming back. The reason they call this Selenium RC is because of the remote control functionality and the fact that this automation can eventually be done when you are not in the office, usually during a build, or at some point throughout the day.

Remember that it's important to be rigorous when doing Functional Tests, and that it's important to test real values. I highly recommend the ID and class advice since this will more likely not break your tests if minor design and styling gets changed. Of course, this is just advice, and this is not a magic bullet to fix your app. Take whichever advice you feel is the most applicable to your situation.