One of the most boring things to do is testing, and with our breakage rate we need a lot of it. :P O:-)
Well, this evening I was particularly bored and I started writing a script to do some automated testing:
https://gist.github.com/emanuele45/5630884
At the moment is still at the beginning (take in consideration I don't know Python, not joking, I'm learning it while writing that script), what it does at the moment is download a package from github, extract the zip into a folder of your choice, and run the install process using
mechanize (http://wwwsearch.sourceforge.net/mechanize/).
I think this was the easy part...
Now what I would like to have is a script that:
1) actually checks the output of the pages (at the moment is a blind install as you can see, no checks at all);
2) runs through "all" the pages of elkarte (or at least most of them) to check for undefined indexes (I have to write the code to append $db_show_debug = true; to Settings.php (should be easy: open(file, 'a') or so);
3) POST some forms, at least the most important.
If anyone wants to join the fun... :D
Some random sites I found and don't want to save in the bookmarks:
http://www.pythonforbeginners.com/cheatsheet/python-mechanize-cheat-sheet/
http://stackoverflow.com/questions/4720470/using-python-and-mechanize-to-submit-form-data-and-authenticate
http://stackoverflow.com/questions/3956280/submitting-a-form-in-mechanize
http://ubuntuincident.wordpress.com/2011/11/08/download-login-protected-pages-with-python-using-mechanize-and-splinter-part-3/
Oh, we really need testing, and any automation is more than welcome! Thanks for initiating this, I think we'd better add this wip to tools or something like that.
And we try it at home.
So! LOL, I feel the site experience is still teaching us our lesson, huh? Many changes, both structural and localized, lead to some issues caught up in the development process, but quite a number weren't. Not surprising, to be honest, refactoring and rewrites are not even supposed to be done without unit testing. Again, I don't think we had that much choice considering we have started from a codebase which doesn't quite support it (with exceptions), but it's about time to take some more steps to fix this.
I think this is more needed than ever, or, the better alternative, unit testing if we can introduce it at this stage, for more or less things, and more for later.
TBH I'm not entirely sure what I'm doing, though:
https://travis-ci.org/emanuele45/Dialogo
http://piwik.org/blog/2013/10/our-latest-improvement-to-qa-screenshot-testing/
^^ That would be painfully cool! :P
Nerds running wild :P
A couple of interesting articles:
http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/
https://sebastian-bergmann.de/archives/881-Testing-Your-Privates.html