Skip to main content ↑ ↓
Topic: SimpleTest? (Read 5276 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

SimpleTest?

Is SimpleTest gone?
The domain is expired a couple of weeks ago... ???
Bugs creator.
Features destroyer.
Template killer.

Re: SimpleTest?

Reply #2 –

I've been having trouble getting it running with Netbeans. No matter what I do, it seems to say that I have zero tests.

Re: SimpleTest?

Reply #3 –

Do you mean this one?

http://simpletest.sourceforge.net/

I'd used some of its components in the past and it was pretty good.

Re: SimpleTest?

Reply #4 –

mmm... was I tricked by google? ???
I was looking at simpletest.org that was bringing me to a domain parked message...
Bugs creator.
Features destroyer.
Template killer.

Re: SimpleTest?

Reply #5 –

Simple test has been good, but I do think we should consider the move to phpunit (assuming we can get it running) ... it seems better supported as a program and is used in several of the CI projects that we use as well (which simple test is not)  Something to consider, that link makes it look easy to transfer over, maybe!

Re: SimpleTest?

Reply #6 –

I was quickly looking at phpunit, but I can't see how do tests on the generated webpages.
I'm probably missing something. somewhere.
Bugs creator.
Features destroyer.
Template killer.

Re: SimpleTest?

Reply #7 –

You want to do functional tests of the output?

Re: SimpleTest?

Reply #8 –

We are currently doing some, yes.

I don't know if they are always necessary or if we can live without.
Bugs creator.
Features destroyer.
Template killer.

Re: SimpleTest?

Reply #9 –

Yeah .. an old bump ! 

I've been playing a bit with PHPUnit to replace SimpleTest, as noted there is no "functional" tests with it, meaning a grab a web page, click on some links, get a result.  I think we can use Selenium and PHPUnits interface to that for the same thing but I don't know.  Right now we only had 1 or maybe 2 tests that used the web interface, and there is a lot of "unit" testing that should be added before that as well.

So what I have done:
Updated the travis.yml file. 
- I moved inline commands to several bash .sh files, (setup-elkarte, setup-server, setup-script, setup-results)
- setup-server.sh does the install of apache and either mysql or postgre.  It also sets the elkarte_test db for install
- setup-elkarte.sh copys the site files to the Apache directory and installs the database files for mysql or postgre
- setup-script does the calling of phpunit, for 5.4/mysql it also runs it with code coverage enabled
- setup-results will send the results of code coverage to scruitinizer if it was a code coverage run (we have a lot of work to do)

Created a composer.json file
- This is a basic composer file to define the project, the main use of it is the dev section.  This section defines phpunit as a dependency.  This file is called/run from setup-elkarte.sh to install phpunit and all its dependency's for the test run. [1] Once It was running I also added the composer lock file which keeps the running config values the same so it does not break with a new version.
- I added composer.phar to the repo, this is used by the test setup.  [2]


Updated the tests:
- The test files must  be named xxxxTest.php, with that convention phpunit will find them and run them
- Update the class names etc to use phpunit calls in place of simpletest calls
- Updated the assert statements
- Add some docblocks :P

Created a bootstrap.php
- This is called by phpunit at the start (its usage is defined in the xml files), its sort of  an index.php and sort of SSI.php, it loads enough stuff so all testcases can run.

Created phpunit-postgres-travis.xml, phpunit-sql-travis.xml, phpunit-with-coverage-mysqli-travis
- These are the config files that phpunit uses to define the configuration, they used in setup-script to provide phpunit the right config for the current run

You can see the commit results here (sorry for the cluster) ... https://github.com/Spuds/Elkarte/tree/phpunit  If there are no objections to the conversion, I'll clean up the commit and make a PR.
1 We could also add phpunit to the repo like we had done with simpletest but using composer seemed a nice way to go.↵
2 We can also remove this and download it in the setup-elkarte.sh or use the outdated one that is in the standard install.↵

Re: SimpleTest?

Reply #10 –

No objections from me! :D

Is there anything else to know about writing tests apart from copy&paste an existing one? O:-)
Bugs creator.
Features destroyer.
Template killer.

Re: SimpleTest?

Reply #11 –

There are a few minor changes, but Copy&Paste is a good start.  PHPUnit does have a richer set of assert statements as well which can/will be handy.

1) Extend PHPUnit_Framework_TestCase instead of UnitTestCase
2) Should never need SSI, any extra files you needs should just be the subs.  You can include those in the setUp method
3) globals are backup by default in the .xml configs (which means each test that is run gets the original globals).  If a subsequent test method relies on a updated global from a previous test method, then you can do one of two things
    a) declare @backupGlobals disabled in the class level docblocks. (example is hooksTest)
    b) get the value of the new global in the test method that sets it and pass it to the test method that needs it.  Also use @depends test xyz in the doc block of the method that needs testxyz results.
4) A few asserts did change
        assertEqual => assertEquals
        assertNotEqual => assertNotEquals
        assertPattern => assertRegExp
        assertIdentical => assertSame
        assertNotIdentical => assertNotSame
        assertIsA => assertInstanceOf  OR assertInternalType (like assertIsA($var, 'numeric');  => assertInternalType('numeric', $var);
        $this->pass() => $this->assertTrue(true)

Re: SimpleTest?

Reply #12 –

Cool cool! :D
Bugs creator.
Features destroyer.
Template killer.

Re: SimpleTest?

Reply #13 –

I now have "functional" tests working as well, this is done with phpunit + selenium.  This ended up being a bit tricky but everything seems to work. 

The "functional" tests are only run when the 5.4/mysql matrix test is run.  This is the same test case that generates the code coverage report.  Running the functional test requires more server setup and test time, so that's why I added all that overhead to a single test in the overall matrix.

Updates in additional to the other post are:
- Added a new setup-selenium.sh script.
   - This bash script is called from setup-elkarte.sh 
   - It downloads the standalone selenium-server .jar file and installs Firefox
   - Starts a xvfb (virtual frame buffer) with java running selenium.
   - Ensures the frame buffer server starts.
- Added the phpunit-selenium dependency to our composer.json file and updated the lock file
   - Phpunit-selenium issues commands to the the selenium server, this in turns runs Firefox in the vfb.  You are running an actual copy of Firefox, so js etc is available.  You can even write drag and drop testcases if you wanted.
- Removed composer.phar from the repo, instead it now updates composer that is part of the Travis test setup.

Function Testcases:
- Functional tests must end with Web.php, this ensures they are only loaded for the 5.4/mysql test run.
- Functional test cases are run by extending PHPUnit_Extensions_Selenium2TestCase
- The commands are what you would typically find in any DOM traversing code, just with their own names.
- Updated the functional tests we had to use the new routines
- Updated bootstrap by adding  PHPUnit_Extensions_SeleniumTestCase::shareSession(true);  Without that in place, when you submitted a form, the $_POSTed vars were empty.  Don't ask how long it took me to figure that out :'(
- Probably best to run all Web.php scripts with @backupGlobals disabled

Writing functional tests
- This demo script is the best resource I could find.
https://github.com/giorgiosironi/phpunit-selenium/blob/master/Tests/Selenium2TestCaseTest.php
You can also read https://phpunit.de/manual/3.7/en/selenium.html

Code Coverage
- I tried to enabled code coverage for the functional tests, however I'm not sure they are being included in the generated clover report.  They are noted in the test count total and in the run test report, just not sure about the coverage report.

Enjoy ! https://github.com/elkarte/Elkarte/pull/1860

Re: SimpleTest?

Reply #14 –

 emanuele likes and marks unread the topic because there is too much cool stuff going on to understand it in just one go! :D
Bugs creator.
Features destroyer.
Template killer.