Ongoing simple thing todo: QA testing December 09, 2013, 06:05:08 am Several months ago, Norv introduced some unit testing in the "system":https://github.com/elkarte/Elkarte/tree/master/testsbased on SimpleTest (nothing to do with Simple Machines).A while ago, I played a bit with them, github and Travis so that on any pull request and any merge the tests are automatically run and if there is any issue we should be able to know before merging the pull request or soon after.So, why this post?Well, at the moment we have just a small bunch of very basic tests, and since write tests is usually rather simple, it could be yet another way to start contributing! The organization of the tests is (hopefully) rather simple:tests is the directory that contains all the relevant code.[/url]simpletest contains SimpleTest code (the unit testing suite).travis-ci contains all the files necessary to initialize the travis-ci environment (creation and population of the database, setup of the files, etc.).install the tests related to installation (these are not run by travis (yet), because I wasn't able to properly set them up, so this is already another possible thing to do)sources contains all the tests related to the sources directory, the internal structure follows the one of Elk's sources directory with subdirectories like admin, controllers and subs that contain relevant tests for the corresponding parts of the code (here is where most of the tests should end up in).all_tests.php is the file that runs all the available tests (not used for travis-ci builds, but it can be used locally).run_tests.php is the file that contains the tests run at travis-ci (it doesn't contain installation-related tests).To know what a test looks like you can open one of those already present, for example:https://github.com/elkarte/Elkarte/blob/master/tests/sources/subs/TestBoards.subs.phpThis files has two tests:testBoardInfo]https://github.com/elkarte/Elkarte/blob/master/tests/sources/subs/TestBoards.subs.php#L37]testBoardInfo runs the function boardInfo (from Boards.subs.php) and verify that:the board has a namethe post count is presentthe post count is a numbertestBoardPosts that tests the function boardsPosts to verify that:a post count for the board 1 is returnedand post count returned is a number (it does that in two ways: first using the is_number php function and then asserting the result as true, and using SimpleTest function asserting directly if it is a number)[/url]If you feel like you want to give it a try, feel free, there is a lot to that is not covered by tests and more tests we have less bugs we should be able to introduce! Spoiler (click to show/hide)I will continue creating bugs even with the most accurate testing!
Re: Ongoing simple thing todo: QA testing Reply #1 – December 17, 2013, 09:12:02 am Recently added tests:basic bbc codehtml to bbcdata validatorbasic likesverification that the tables are created during installbasic registration
Re: Ongoing simple thing todo: QA testing Reply #2 – January 19, 2014, 10:22:18 am One of the last things I did last year about testing is make them independent one from the other, now each test runs as an exec command, that ensure the tests are properly self-executed (and as such self-contained).Another small goodies I added is that when the series of tests is completed (successfully or not) the travis-ci script grabs the content of the error log and outputs it to the console.At the moment it doesn't raise any error, but from time to time it is useful, for example:https://travis-ci.org/elkarte/Elkarte/jobs/17197799#L445the first error is "normal" (it's generated while testing the registration), but the other two are not, they are two legit bugs I introduced while changing something else. In my plans there is to create a test to check if the log is empty (a general one to be run at the end of everything). There is a bit to play with though, because in the log there may be legit errors. One option is in each test to cleanup any "expected" error that may popup, for example in the current case, part of the tearDown method in TestRegistration could be to fetch the error and remove it. At that point, what is left is clearly a bug and could be reported as a "build breaker".
Re: Ongoing simple thing todo: QA testing Reply #3 – January 19, 2014, 11:29:03 pm Spuds wipes sweat from brow that you did not use one of my broken PR's as the example All the Travis work is really very very cool, has saved us a bunch of errors already!
Re: Ongoing simple thing todo: QA testing Reply #4 – February 09, 2014, 05:54:51 am Teh @Spuds is shy and didn't tell people that he added another toy to the collection: https://scrutinizer-ci.com/In particular: https://scrutinizer-ci.com/g/elkarte/Elkarte/What does scrutinizer do?It checks the code looking for broken things, for example methods or functions not existing, methods or functions not used, bad documentation, bad coding style, and many other things.It runs on each pull request, so before merging there is another test that will help ElkArte to be less broken and so allow easier development and faster release cycle!
Re: Ongoing simple thing todo: QA testing Reply #5 – February 09, 2014, 09:07:15 am Quote from: emanuele – February 09, 2014, 05:54:51 amTeh @Spuds is shy and didn't tell people that he added another toy to the collection: https://scrutinizer-ci.com/In particular: https://scrutinizer-ci.com/g/elkarte/Elkarte/What does scrutinizer do?It checks the code looking for broken things, for example methods or functions not existing, methods or functions not used, bad documentation, bad coding style, and many other things.It runs on each pull request, so before merging there is another test that will help ElkArte to be less broken and so allow easier development and faster release cycle! Cool.
Re: Ongoing simple thing todo: QA testing Reply #6 – February 09, 2014, 10:23:00 am Its been a fun toy to play with and its found several legit bugs in the code that we have fixed. There are lots of the mundane documentation issues, style issues as well, but the real joy is that it does find bugs and unused code that would be difficult to find otherwise. It does take looking at each area that gets highlighted and determining why that area got flagged, some are obvious, some not so much, but in the end its creating a better product.
Re: Ongoing simple thing todo: QA testing Reply #7 – March 16, 2014, 04:37:00 pm One thing that I just discovered would greatly benefit from automated testing is the moderation area. Mainly because it's something not that used.But, in order to be able to test the basis there are several things to do, because there are tons of variants, so the easiest are:Create a user <= one test: use the internal functions to create one or mimicking the registration process and then do the test querying directly the db to check the expected values are thereCreate a topic (with a post obviously) <= second test: again use createPost or mimic the Post page and verify everything is inserted correctly into the databseReport that post <= and check the report is thereOpen/view the reportDismiss itClose itAnd this is just about reporting, but the moderation area is also about unapproved posts (and then there are posts posted by members without permission and posts that are unapproved that need to be inserted and tested for both approval and deletion), and members, and emails, etc...Well, this need is now tracked, sooner or later someone will create a test.
Modern, Free, Powerful