Elk Development => Feature Discussion => Topic started by: Jorin on November 25, 2015, 01:46:45 am
Title: Backups
Post by: Jorin on November 25, 2015, 01:46:45 am
I follow your discussion about making ElkArte safer and at this point the question about backups came to mind.
Do you think every forum owner makes regularly backups? I don't think so. So why can't ElkArte offer a simple, easy to use solution for this problem? It would be great: Be the owner of a forum and know that you don't need anything else - your ElkArte installation will have everything you need. Regular backups too.
I know there's a way to make a database backup in the admin panel. But you have to start it yourself, and with SMF in the past the backup can often be incomplete. Is this bug fixed with ElkArte?
What do you think about this idea? Is there a way? Or do we have to inform others about other ways of regluar backups, such as MySQLDumper? Is there a way to make regular ftp backups too? Or is this not neccessary and you think a forum owner should know about these things and find a solution on it's own?
Title: Re: Backups
Post by: wintstar on November 25, 2015, 02:27:21 am
I like the idea. It should be possible, enter the older backups are deleted automatically from the server. The period for this should be configured individually. Or, depending on the size of the space.
Title: Re: Backups
Post by: ahrasis on November 25, 2015, 03:55:41 am
Currently, we have a backup after addon's installation with no database but overall we do let the forum owner do a full backup on their own. Tedious, I think, but this actually gives a good value to a forum software. I vote for it.
Or an addon pending this feature to be considered?
Title: Re: Backups
Post by: emanuele on November 25, 2015, 08:16:02 am
The problem with backups is the usual: the backup should reside on a "secure" part of the server (usually a directory not accessible from the web), but most people will not set up the forum like that (or may not even have the option to set it up like that) and may leave these sensitive data in a web-accessible directory, meaning anyone could download and steal them. The other option could be to send these via some secure connection to another server (but probably that means installing something like I did for my backups uploaded to google drive and so, again, we are ruling out people on shared hostings).
Backups are a touchy argument...
Title: Re: Backups
Post by: Jorin on November 25, 2015, 08:32:21 am
Yeah, I don't know if this is possible in general. I only had an idea. And it seems like not a bad one. ;)
Maybe you can write a script which sends the backup via e-mail to the admin? A function to save the backup on another folder on the server would be great. The admin could save the needed login data in the admin panel, this can be saved encoded in the database of the forum. Again, only ideas.
But speaking of backups on Google Drive... I already searched for a possibility to save my backups on my Google Drive. Can you give me a hint (maybe in a message)?
Title: Re: Backups
Post by: emanuele on November 25, 2015, 08:41:55 am
I thought I posted more, but apparently I didn't. lol http://www.elkarte.net/community/index.php?topic=2540.msg17971#msg17971
I'll try to remember to elaborate, in case you don't see anything, feel free to bump the other topic. ;)
Title: Re: Backups
Post by: Jorin on November 25, 2015, 08:48:11 am
I didn't say I'm interested. My fault. :D Now I am. ;)
Title: Re: Backup, what do you use?
Post by: radu81 on November 25, 2015, 09:25:03 am
Quote from: emanuele – The problem with backups is the usual: the backup should reside on a "secure" part of the server (usually a directory not accessible from the web), but most people will not set up the forum like that (or may not even have the option to set it up like that) and may leave these sensitive data in a web-accessible directory, meaning anyone could download and steal them.
I am agree with you, and in my opinion backups should be made on an external server. A few years ago, my shared hosting account was hacked, it was not a hack caused by insecure forum/cms, but all server was compromised. I've been lucky that they did not delete files, they only added an x.php file on all my websites with something like this echo "Hacked by ....."; just to prove they hacked those websites. All my backups were on the same ftp, so I think backups in this way are useless.
Now I use a small vps from Linode, I got daily backups of database using AutoMySQLBackup, and every week I do manual backups of FTP and DB using ssh. I know it's not a good solution, but I am going to use the Linode Backups for 2,5$/month
Title: Re: Re: Backup, what do you use?
Post by: kucing on November 25, 2015, 09:47:28 am
Well If I can request a feature or addon is auto backup to DropBox/Google Drive. I use this kind of backup (UpDraft Plus) in my WordPress blog, autosync every 4 hours (files and databases). It's been a lifesaver several times when my shared hosting provider being iffy.
Title: Re: Re: Backup, what do you use?
Post by: Jorin on November 25, 2015, 09:53:50 am
That would be real great!
Title: Re: Backups
Post by: ahrasis on November 25, 2015, 10:46:11 am
How would this handle large DB backups? For example, I have one SMF install where my morning download consists of a gz file that, as of this morning, is 215,516kb. Not sure that would email well. I'm not saying I'm against the idea. I'm simply wondering how certain issues could be handled. ;)
Yeah, that is SMF. But it could get the Elkarte treatment should OI come to support 2.1. ;D
Title: Re: Backups
Post by: kucing on November 25, 2015, 07:58:09 pm
upload to dropbox/google drive can be a solution when the filesize is big and not suitable for email. I ran wordpress backup plugin everyday for a site that almost 1GB in size including files and database. from a shared hosting.
Title: Re: Backups
Post by: ahrasis on November 25, 2015, 09:03:09 pm
Well, if we can use and send email from our forum using gmail, I think we should be able to send a large enough database file attachment to our own email. Do check on gmail limits for this which is 25MB but I think, you can overcome that with google drive trick.
Title: Re: Backups
Post by: Joshua Dickerson on November 25, 2015, 11:44:01 pm
Feature creep! It's a forum. There are excellent backup tools around. If we get in to that game, we lose focus and development of the forum grinds to a halt.
Title: Re: Backups
Post by: Jorin on November 26, 2015, 01:09:01 am
lol Yep, it is. Then let's erase the backup functionality we have at the moment. Better no backup at all than an uncomplete one. Oh, and let's erase the file edit functionality in the backend of ElkArte too. There are better tools for this kind of work. ;)
Sorry for the sarcasm. :-[ But yep, I think kind of this way. I don't think it is good to offer functionalities that don't work in a full, complete way. It can lead to serious problems if users trust in the functionalities and they don't work perfect in the end. An incomplete backup can crash a board.
If we have a backup function inside ElkArte, why can't it work like the database maintenance and offer scheduled backups? I understand the technical difficulties (at least some of them), and if there isn't a way after all or you think it is much too complicated, okay, I can respect and live with that. I can't change this then.
All I wanted is to share my thoughts about things that maybe make ElkArte better. Better in handling and/or better in UI. I don't demand things. I know you appreciate that. ;) So no bad feelings, it's just an exchange of ideas and thoughts.
Title: Re: Backups
Post by: ahrasis on November 26, 2015, 04:55:33 am
I agree with @Jorin on this. Besides, we can still attempt to create an addon for automatic database backup though it may not be a feature.
Title: Re: Backups
Post by: kucing on November 26, 2015, 07:45:15 pm
i agree, it should be an addon too.
Title: Re: Backups
Post by: Joshua Dickerson on November 26, 2015, 08:17:19 pm
I said that we should get rid of it before. I am all for that. Though, it was shot down because it would take more to get rid of it than it is just to keep it. I guess in a future version it will disappear.
I totally agree with not implementing half assed features and users' expectations.
Title: Re: Backups
Post by: emanuele on November 27, 2015, 08:45:31 am
Just to clear things up a bit and not mix stuff. 1) The database back in ElkArte is perfectly functioning. 2) To reduce at a minimum the risk of broken backups, Elk tries to "guess" (based on resources available) if the backup will complete before the backup is started. And if the server config looks problematic, is not even possible to start the backup. Under optimal conditions I was able to perfectly backup a 2 GB database. So, as far as I can tell it's far from being an half assed feature.
That said, of course any php script may suffer from many kind of problems, so it may be possible the database backup is not entirely reliable. Unfortunately it's difficult to account for all the possible combinations.
I personally wouldn't use an email to store the backups for many reasons (from the limits that may prevent the sending or the receiving of the email, making the backup as useless as a broken one, to the fact that emails are not really the safes place to store things, see the many cases of account breaches and passwords leaks).
Title: Re: Backups
Post by: Eliana Tamerin on November 27, 2015, 03:33:45 pm
Quote from: emanuele – Just to clear things up a bit and not mix stuff. 1) The database back in ElkArte is perfectly functioning. 2) To reduce at a minimum the risk of broken backups, Elk tries to "guess" (based on resources available) if the backup will complete before the backup is started. And if the server config looks problematic, is not even possible to start the backup. Under optimal conditions I was able to perfectly backup a 2 GB database. So, as far as I can tell it's far from being an half assed feature.
That said, of course any php script may suffer from many kind of problems, so it may be possible the database backup is not entirely reliable. Unfortunately it's difficult to account for all the possible combinations.
I personally wouldn't use an email to store the backups for many reasons (from the limits that may prevent the sending or the receiving of the email, making the backup as useless as a broken one, to the fact that emails are not really the safes place to store things, see the many cases of account breaches and passwords leaks).
My Gmail has 2FA security, starting with a 15 character password with varying case, numbers and symbols. I'd like to see someone hack into my email.
Title: Re: Backups
Post by: emanuele on November 27, 2015, 04:04:36 pm
Part 2 (sorry, I had to quickly stop because I noticed I was going to be late for work lol).
First: I'm not saying that because I think the idea is bad, but just to put on the table all the difficulties to overcome before having a fully-100%-working-without-any-doubt feature/addon/whatever.
After what I wrote, there is the little bit of "problems" due to php. The usual reason why SMF backup was used to fail, is because of timeouts and/or memory limits. Even though php tries to give you ways to workaround these limits, many hosts do not allow you to override default settings, so you are bound to a system that is limited and doesn't allow proper ways of backing up data.
If you have to backup few megabyes of data, it may take "some" time (difficult to estimate) to build up the stream, pack it and prepare for further handling. So, if you do that from a fake-cron like the Elk (or SMF) scheduled tasks, you are going to have bad surprises because sooner or later the backup my fail due to timeouts. In order to avoid timeout issues you should be able to run a php script as a proper cronjob, of course not all the hosts allow to manage cronjobs. So there is a limit.
Assuming timeouts are not a problem, if your php memory is limited, you'll be able to backup only so much of your database before php hits the wall of the "out of memory" error. And work around the memory limit is a bit trickier. Not impossible, just tricky.
There may be a part 3, but likely only Sunday. :P
Title: Re: Backups
Post by: meetdilip on November 27, 2015, 04:31:26 pm
2FA is a fake sense of security. Someone with a duplicate SIM card can easily get through the defence.
Title: Re: Backups
Post by: Eliana Tamerin on November 27, 2015, 05:03:27 pm
Quote from: meetdilip – 2FA is a fake sense of security. Someone with a duplicate SIM card can easily get through the defence.
No they can't. Google Authenticator app, not text messages.
Title: Re: Backups
Post by: ahrasis on November 28, 2015, 09:31:40 am
I found this old post and was thinking on using it as backup tool. I managed to get the quickstart working but I am not sure to proceed further from there.
I guess I have to create a folder most probably outside the website folder and link it to my gDrive, but I am not sure about the link part. I will keep testing this for a while and see how it goes.
Title: Re: Backups
Post by: emanuele on February 01, 2017, 02:46:26 am
Title: Re: Backups
Post by: emanuele on February 01, 2017, 07:15:25 pm
Actually... scrap what I said, I use http://www.webupd8.org/2014/09/gdrive-simple-google-drive-cli-client.html Basically I have an sh script that looks something like:
Title: Re: Backups
Post by: ahrasis on February 01, 2017, 08:22:59 pm
I already setup grive2 and only need a script to backup my database to its folder.
I was considering to use jorin's php script yesterday but yours seems better and simpler @emanuele, since it covers both forum database and files.
However I'd prefer smallest compression, so I think I'll change the zip command to tar -cjvf to turn it to a tar.bz2 file.
I am not sure whether I need to backup the whole forum though I think it is good and easy. At least I think packages/backups can be opted out by adding --exclude=/path/to/root/htdocs/forum/packages/backups.
Let's see how this goes then.
Title: Re: Backups
Post by: emanuele on February 02, 2017, 03:08:35 am
And while we are at it, on the gdrive side, I use a https://script.google.com to remove the oldest ones:
function FindEmptyFolders() { var pageSize = 200; var folders = null; var token = null; var i = null; var thefolder_name = 'backup'; var DeleteLimit = new Date().getTime() - 3600 * 1000 * 24 * 14;
Logger.clear()
var folders = DriveApp.getFoldersByName(thefolder_name);
while (folders.hasNext()) { var folder = folders.next(); // http://stackoverflow.com/questions/25749906/permanently-delete-file-from-google-drive var files = folder.getFiles(); while (files.hasNext()) { var file = files.next(); if (file.getDateCreated().getTime() < DeleteLimit) { Logger.log(file.getName()); Drive.Files.remove(file.getId()); } } }
var DeleteLimit = new Date().getTime() - 3600 * 1000 * 24 * 14;
hmm... 2 weeks I'd say (3600 1000 24 is a day worth of milliseconds, and the 14 would be the number of days).
Title: Re: Backups
Post by: Frenzie on February 02, 2017, 04:28:13 am
Quote from: emanuele – And while we are at it, on the gdrive side, I use a https://script.google.com
Huh, I had no idea that existed. Interesting.
Title: Re: Backups
Post by: ahrasis on February 02, 2017, 05:18:01 am
Very interesting indeed. I haven't setup mine yet. Will look into it one by one before proceeding.
Title: Re: Backups
Post by: emanuele on February 02, 2017, 11:57:46 am
Yep, it's pretty interesting.
Title: Re: Backups
Post by: ahrasis on February 03, 2017, 09:31:23 pm
There is an php auto backup script shared by @Jorin and improved by @emanuele if you search around the forum. Even ElkArte already has a database backup function. That (either one) can be add up as an addon to ElkArte if somebody is interested to package it. I was thinking that autotask scheduler may perform that job. The database info can be taken from Settings.php so the credentials may be dropped if it is an addon. The only thing is the secret folder / directory which I think can also be dropped IF the database is in a small size and can be easily emailed to a specified email address. But this will not be so easy and as such I don't think that I am able to do it even though I have more free time for now.
Title: Re: Backups
Post by: emanuele on February 04, 2017, 02:37:24 am
Quote from: derived – would it be feasible to have elk do at time of install/setup and tie the file to the same admin key?
I'm not sure what you mean here, could you elaborate a bit? (Maybe in another topic if it is not related to the backup discussion.)
Title: Re: Backups
Post by: radu81 on February 04, 2017, 05:27:31 am
there is also this discussion regarding backups http://www.elkarte.net/community/index.php?topic=2540.0
Title: Re: Backups
Post by: shaitan on February 05, 2017, 03:58:13 pm
I have databases 7 G , 130 G files (more when I hosted Nao :D ). I needed absolutely a secure and easy solution for backups. A software solution was not possible in my case. Many years ago, I have found a very good Hoster. (specialized for developers). It's a shared Host, include many services, no options. One is backup, on 30 days, DB, files, mails. Read only, access from ftp and ssh, duplicated on différents servers on two différents data centers. High security.
Title: Re: Backups
Post by: radu81 on February 05, 2017, 04:52:58 pm
that's a very nice and complete solution for backups ;)
Title: Re: Backups
Post by: shaitan on February 05, 2017, 05:23:52 pm
I hesitate to talk about it. I know how difficult it is to find a good web host. But I talked about it because I wanted to say that if you have a large volume to backup, the usual solutions are not suitable.
Title: Re: Backups
Post by: ahrasis on February 07, 2017, 08:12:47 pm
Ho ho... That's a lot. Google (free version) will only accommodate up to 15 G. So using LEMP + ISPConfig, I would do the discussed methods per domain basis or otherwise it won't be enough.
Title: Re: Backups
Post by: Frenzie on February 08, 2017, 02:22:08 am
Only 15 GiB? How times have changed. My first e-mail accounts were in the 5 to 10 MiB range. :P
And 150 kb of attachments max! LOL
Title: Re: Backups
Post by: hartiberlin on January 24, 2018, 02:11:56 pm
So why don´t we have a well running Mod for Backup of Elkarte, that can backup the Database and all the files ? Like in Wordpress the Plugin "BackupWordpress" https://wordpress.org/plugins/backupwordpress/
Maybe someone can port this to Elkarte or anybody can post his complete scripts, that really work ? Many thanks.
Title: Re: Backups
Post by: emanuele on January 24, 2018, 04:20:42 pm
My answer in the first page still stand as the reason I am not going to code it. ;)