ElkArte Community

General => OpenImporter => Topic started by: TE on January 29, 2016, 01:02:26 am

Title: [SMF 2.1] Importer
Post by: TE on January 29, 2016, 01:02:26 am
This importer is intendet to import a SMF 2.1 (Beta) installation to ElkArte:

Notes:
-  Currently the passwords are not converted, so you have to use the "Forgot your password?"  link ..
- Likes and Mentions are not converted, maybe I'll add them later..

importing settings... ✔
Importing members... ✔
Importing attachments... ✔
Importing avatars... ✔
Importing categories... ✔
Importing boards... ✔
Importing topics... ✔
Importing messages... ✔
Importing polls... ✔
Importing poll choices... ✔
Importing poll votes... ✔
Importing personal messages... ✔
Importing pm recipients... ✔
Importing pm rules... ✔
Importing board moderators... ✔
Importing mark read data (boards)... ✔
Importing mark read data (topics)... ✔
Importing mark read data... ✔
Importing notifications... ✔
Importing membergroups... ✔
Importing group moderators... ✔
Importing permission profiles... ✔
Importing permissions... ✔
Importing board permissions... ✔
Importing smileys... ✔
Copying smileys... ✔
Importing statistics (activity log)... ✔
Importing logged actions... ✔
Importing reported posts... ✔
Importing reported comments... ✔
Importing spider hits... ✔
Importing spider stats... ✔
Importing subscriptions... ✔
Recalculating statistics... ✔
Title: Re: [SMF 2.1] Importer
Post by: Eliana Tamerin on January 29, 2016, 02:11:01 am
Why not just add this as a Source for the Open Importer?
Title: Re: [SMF 2.1] Importer
Post by: TE on January 29, 2016, 04:43:11 am
Quote from: Eliana Tamerin – Why not just add this as a Source for the Open Importer?
It is open importer (just an older version because the current one is badly broken..) ;)
Title: Re: [SMF 2.1] Importer
Post by: meetdilip on January 29, 2016, 05:00:30 am
Thanks @TE
Title: Re: [SMF 2.1] Importer
Post by: emanuele on January 30, 2016, 05:11:33 pm
Quote from: TE – It is open importer (just an older version because the current one is badly broken..) ;)
The only problem I have with the master branch is with a regex, applying this patch works fine for me:
Code: [Select]
diff --git a/importer/OpenImporter/XmlProcessor.php b/importer/OpenImporter/XmlProcessor.php
index 9f08465..f9bb8ed 100644
--- a/importer/OpenImporter/XmlProcessor.php
+++ b/importer/OpenImporter/XmlProcessor.php
@@ -346,7 +346,7 @@ class XmlProcessor
        protected function detect($table)
        {
                $table = $this->fix_params($table);
-               $table = preg_replace('/^[icode][\w\d]*[/icode]\./i', '', $this->fix_params($table));
+               $table = preg_replace('/^[icode][\w\d\-_]*[/icode]\./i', '', $this->fix_params($table));
 
                $db_name_str = $this->config->source->getDbName();
Title: Re: [SMF 2.1] Importer
Post by: TE on February 01, 2016, 12:32:59 am
Quote from: emanuele – The only problem I have with the master branch is with a regex, applying this patch works fine for me:
yep, you're probably right... sorry.. should have been more specific on the issues I found.

Master branch: At least the MyBB importers are  badly broken due to the var / object $config (it is an object in OI and also a global var in the  MyBB config files)
DEV branch Maybe it's beyond my skills due to composer and all that stuff but I wasn't able to get OI working at all..
Title: Re: [SMF 2.1] Importer
Post by: Spuds on February 01, 2016, 12:18:26 pm
I had some problems as well, but a few were self induced as well  :-[

For the dev revision you need to have composer installed on your system.
Code: [Select]
curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer
Then pull the development branch from GIT, say to /var/www ... then change to the importer directory.  On my vagrant setup that is /var/www/importer (where /var/www is a shared folder to where I have my Open Importer repo).  Then from that directory simply do
Code: [Select]
composer install
It will download all of the dependencies and place them in a /vendor directory, this includes symfony and doctrine.

In my IDE (PhpStorm) I also set the project resource root to the /importer directory so it picked up everything.  I don't really know much (if anything) about symfony or doctrine, but those are more core side of things vs the importer side.  

I did not realize that the master branch was not the current working version, I should have started looking at that branch (just did) so now the dev branch direction makes a lot more sense to me.  I guess master is more like a 1.1 and dev is 2.0 and the current working code is really a "point" in the master branch (maybe we should do a point release of 1.0 from master?)

Quoteobject $config
I'll refactor that to $oi_config (and anything else I see that should be $oi_* to avoid conflicts.  I think the namespace in 1.1 avoids that though)
Title: Re: [SMF 2.1] Importer
Post by: emanuele on February 01, 2016, 01:27:07 pm
Quote from: TE – yep, you're probably right... sorry.. should have been more specific on the issues I found.
Nothing to be sorry about, come on!
I just wanted to be sure I wasn't miss anything big! :D

Quote from: TE – Master branch: At least the MyBB importers are  badly broken due to the var / object $config (it is an object in OI and also a global var in the  MyBB config files)
And in fact I was. *facepalm*!
That's one of the reasons in the dev branch I started "parsing" the configuration files as strings rather than loading them, so there will not be conflicts like this one in the future. The idea is to read the content of the file and use a regex or some similar solution to get the values of the variables.

Quote from: TE – DEV branch Maybe it's beyond my skills due to composer and all that stuff but I wasn't able to get OI working at all..
As Spuds said it needs composer (that once used is not really difficult, but if nobody tells others how to use it it's a bit of a headache... And I didn't write any documentation about it... sorry. :-[ ), and that's why I'd like to have at some point a "build bot" able to prepare packages ready-to-use for easier testing.
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on February 16, 2016, 08:38:40 pm
It's great news to see this.  Thank you!  A couple of quick questions:  1. how do you use this?  2.  Is there a possibility Elk will go to the bcrypt credentials?
Title: Re: [SMF 2.1] Importer
Post by: TE on February 17, 2016, 02:26:38 am
1) extract the attached archive, upload the files to your webserver and call import.php from a webbrowser. then follow the instructions.
2) Maybe ... Haven't checked their new passwords encryption yet, they are using bcrypt too?
Title: Re: [SMF 2.1] Importer
Post by: emanuele on February 17, 2016, 04:56:36 am
If I'm not wrong, Elk is already using CRYPT_BLOWFISH as first choice when available, then CRYPT_EXT_DES and if none of these is available several rounds of md5.
So, if the server has what is required it should already use "bcrypt" (of course if I got right how phpass works).
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on February 17, 2016, 07:58:25 pm
Then I have made an erroneous assumption concerning the passwords not porting.  Thanks for the info! 
Title: Re: [SMF 2.1] Importer
Post by: emanuele on February 18, 2016, 02:44:47 am
The password not being portable may be for many reasons.
Elk uses the http://www.openwall.com/phpass/ library for password hashing (just because we realized that if nearly all the most famous software use it (WordPress, phpBB, Drupal, Joomla) there may be a reason, that it could be, for how much we know about security, a company focused on security is likely to know more than we do), so it adds a $P$ prefix to the string. Then Elk pass to phpass and sha-256 hash of user . password.
These are two differences, there may be more...
Title: Re: [SMF 2.1] Importer
Post by: Runic on March 14, 2016, 08:10:46 pm
im getting error on members import on this, no mods installed says "Unknown column 'hide_email' in 'field list'" caused the error :/

Fixed it by removing that field from the importer file and also the latestMember caused error so removed that seems everything converted fine afterwards.
Title: Re: [SMF 2.1] Importer
Post by: Spuds on March 14, 2016, 08:27:52 pm
Which version of 2.1 were you importing from?  I know there are a couple of beta versions out there.
Title: Re: [SMF 2.1] Importer
Post by: Runic on March 14, 2016, 08:54:08 pm
db was on Beta 2 files were Beta 3, the beta 3 upgrade.php is broken so I never got around to updating db, but its all sorted and working fine so far :)
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on May 29, 2016, 11:31:15 am
Will probably be giving this a whirl soon.  Before this gets underway, here's a question regarding paths and source vs. destination.

If the Source forum exists in /home/XXXX/public_html, I would like the Destination to wind up in the same path.  It is possible to accomplish this with a complex series of transferring files between folders, repair_settings.php, etc.  Is there an easier way to accomplish this?  Thanks!
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on May 29, 2016, 04:05:19 pm
Setup a shadow site for testing purposes.  This is an error given while importing members:

QuoteMessage: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead
Trace: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead
Line: 2967


What is the proper fix for this?   Thanks! 
Title: Re: [SMF 2.1] Importer
Post by: emanuele on May 31, 2016, 03:39:09 am
Sorry, I missed the green icon... :-[

It's a deprecation notice, so for the moment it doesn't affect the outcome (unless in AJAX calls that could break the XML...), so if the conversion is able to go on there is nothing wrong.
For sure there is something to change in the code.
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on May 31, 2016, 07:51:27 am
Quote from: emanuele – Sorry, I missed the green icon... :-[

It's a deprecation notice, so for the moment it doesn't affect the outcome (unless in AJAX calls that could break the XML...), so if the conversion is able to go on there is nothing wrong.
For sure there is something to change in the code.

Hhmmm.  Interesting.  The import process ceased when it gave this error.  Is there a way to make it proceed anyway?
Title: Re: [SMF 2.1] Importer
Post by: emanuele on May 31, 2016, 11:30:32 am
In import.php, find:
Code: [Select]
error_reporting(E_ALL);
and replace it with either:
Code: (conservative version) [Select]
error_reporting(E_ALL & ~E_DEPRECATED);
or:
Code: (aggressive version) [Select]
error_reporting(0);

The first silences only the deprecated warnings, the second silence any error.
Probably the second is fine as well.
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on May 31, 2016, 09:48:30 pm
Thanks emanuele.  Still no luck.  Same error. 

QuoteImporting settings... ✔
Importing members...
Message: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead
Trace: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead
Line: 2967

Even commenting out the error reporting line results in the same.   :(
Title: Re: [SMF 2.1] Importer
Post by: TE on June 01, 2016, 01:13:40 am
IIRC Spuds fixed that one ... Where did you download the importer? I'd recommend the latest master branch:
https://github.com/OpenImporter/openimporter/archive/master.zip
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 01, 2016, 08:06:49 am
I'll try that.  Thanks!   I downloaded the one in the first post of this thread.   ;)
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 01, 2016, 05:08:29 pm
That starts off better.  It gets stuck with a white screen around step 1, substep 17.  Recommendations?
Title: Re: [SMF 2.1] Importer
Post by: TE on June 02, 2016, 01:14:24 am
Quote from: badmonkey – That starts off better.  It gets stuck with a white screen around step 1, substep 17.  Recommendations?
What's the exact error message? (Copy the text or attach a screenshots please)
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 02, 2016, 07:04:57 am
Hi TE.  It's just a white screen with no message.
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 02, 2016, 10:06:05 am
Got it.  Increasing the memory using ini_set did the trick.  Thanks for the help everyone.  Good thing for dry runs!   :D
Title: Re: [SMF 2.1] Importer
Post by: emanuele on June 02, 2016, 10:58:27 am
Out of curiosity, what is the default value?
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 02, 2016, 11:23:21 am
It wasn't set in the script.  The php memory limit on my server is 20M. 
Title: Re: [SMF 2.1] Importer
Post by: emanuele on June 02, 2016, 06:00:44 pm
Yep, tep, I imagined it was set at php.ini level, I was just curious at the actual value.
I asked because at some point I wrote a modified version of convert.php (the original script) that was able to monitor the memory usage and avoid "out of memory" like this. I thought I added it to OI, but apparently I didn't... shame. :-[
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 02, 2016, 10:04:58 pm
Quote from: emanuele – Yep, tep, I imagined it was set at php.ini level, I was just curious at the actual value.
I asked because at some point I wrote a modified version of convert.php (the original script) that was able to monitor the memory usage and avoid "out of memory" like this. I thought I added it to OI, but apparently I didn't... shame. :-[

Indeed it could be useful.  But you can't be responsible for iffy server tuning on my part.  lol
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 03, 2016, 02:39:34 pm
Next hiccup:  attachments.  Directory paths are correct.  But they aren't displaying.  The files still exist in the attachment directories.  Has anyone had this issue?
Title: Re: [SMF 2.1] Importer
Post by: TE on June 03, 2016, 04:00:22 pm
Quote from: badmonkey – Next hiccup:  attachments.  Directory paths are correct.  But they aren't displaying.  The files still exist in the attachment directories.  Has anyone had this issue?
yep, the attachments import is still a bit bggy, especially if your SMF was 1.1 without file encryption before..
This is the attachments and avatar import I used to Import Ruth's Puppenstube.
Code: [Select]
<step>
<title>Importing attachments</title>
<detect>{$from_prefix}attachments</detect>
<destination>{$to_prefix}attachments</destination>
<presql>TRUNCATE {$to_prefix}attachments;</presql>
<preparsecode>
$request = $db->query("
SELECT value
FROM {$from_prefix}settings
WHERE variable='attachmentUploadDir';");
list ($smf_attachments_dir) = $db->fetch_row($request);

$smf_attachments_dir =  rtrim($smf_attachments_dir, '/');

if (empty($row['file_hash']))
{
$row['file_hash'] = createAttachmentFileHash($row['filename']);
$clean_name = $row['filename'];
$se = array("ä", "ö", "ü", "Ä", "Ö", "Ü", "ß");
$repl = array("a", "o", "u", "A", "O", "U", "ss");
$clean_name = str_replace($se, $repl, $clean_name);
$clean_name = strtr($clean_name, 'ŠŽšžŸÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝàáâãäåçèéêëìíîïñòóôõöøùúûüýÿ', 'SZszYAAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy');
$clean_name = strtr($clean_name, array('Þ' => 'TH', 'þ' => 'th', 'Ð' => 'DH', 'ð' => 'dh', 'ß' => 'ss', 'Œ' => 'OE', 'œ' => 'oe', 'Æ' => 'AE', 'æ' => 'ae', 'µ' => 'u'));
$clean_name = preg_replace(array('/\s/', '/[^\w_\.\-]/'), array('_', ''), $clean_name);
$enc_name = $row['id_attach'] . '_' . strtr($clean_name, '.', '_') . md5($clean_name);
$clean_name = preg_replace('~\.[\.]+~', '.', $clean_name);
$clean_name = str_replace('.', '_', $clean_name);

copy_file($smf_attachments_dir . '/' . $enc_name, $attachmentUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
copy_file($smf_attachments_dir . '/' . $clean_name, $attachmentUploadDir . '/' . $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
}
else
copy_file($smf_attachments_dir . '/' . $row['id_attach'] . '_' . $row['file_hash'], $attachmentUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
</preparsecode>
<query>
SELECT
id_attach, id_thumb, id_msg, id_member, attachment_type, filename, file_hash, size, downloads,
width, height, fileext, mime_type, id_folder
FROM {$from_prefix}attachments
WHERE id_msg != 0;
</query>
</step>
<step>
<title>Importing avatars</title>
<detect>{$from_prefix}attachments</detect>
<destination>{$to_prefix}attachments</destination>
<options>
<replace>true</replace>
</options>
<preparsecode>
// Try custom_avatar_dir first
$request = $db->query("
SELECT value
FROM {$from_prefix}settings
WHERE variable = 'custom_avatar_dir';");
list ($smf_attachments_dir) = $db->fetch_row($request);

if (empty($smf_attachments_dir))
{
$request = $db->query("
SELECT value
FROM {$from_prefix}settings
WHERE variable='attachmentUploadDir';");
list ($smf_attachments_dir) = $db->fetch_row($request);
}

if ($avatarUploadDir == $attachmentUploadDir)
$row['attachment_type'] = 0;

if (empty($row['file_hash']))
{
$row['file_hash'] = createAttachmentFileHash($row['filename']);
$clean_name = $row['filename'];
$se = array("ä", "ö", "ü", "Ä", "Ö", "Ü", "ß");
$repl = array("a", "o", "u", "A", "O", "U", "ss");
$clean_name = str_replace($se, $repl, $clean_name);
$clean_name = strtr($clean_name, 'ŠŽšžŸÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝàáâãäåçèéêëìíîïñòóôõöøùúûüýÿ', 'SZszYAAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy');
$clean_name = strtr($clean_name, array('Þ' => 'TH', 'þ' => 'th', 'Ð' => 'DH', 'ð' => 'dh', 'ß' => 'ss', 'Œ' => 'OE', 'œ' => 'oe', 'Æ' => 'AE', 'æ' => 'ae', 'µ' => 'u'));
$clean_name = preg_replace(array('/\s/', '/[^\w_\.\-]/'), array('_', ''), $clean_name);
$enc_name = $row['id_attach'] . '_' . strtr($clean_name, '.', '_') . md5($clean_name);
$clean_name = preg_replace('~\.[\.]+~', '.', $clean_name);

if(file_exists($smf_attachments_dir . '/' . $row['id_attach'] . '_' . str_replace('.', '_', $clean_name). md5($row['filename'] )))
copy_file($smf_attachments_dir . '/' . $row['id_attach'] . '_' . str_replace('.', '_', $clean_name) . md5($row['filename']), $avatarUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');

copy_file($smf_attachments_dir . '/' . $clean_name, $avatarUploadDir . '/' . $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
}
else
copy_file($smf_attachments_dir . '/' . $row['id_attach'] . '_' . $row['file_hash'], $avatarUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
</preparsecode>
<query>
SELECT
id_attach, id_thumb, id_msg, id_member, attachment_type, filename, file_hash, size, downloads,
width, height, fileext, mime_type, id_folder
FROM {$from_prefix}attachments
WHERE id_member != 0;
</query>
</step>
You'd have to replace the attachments and avatars import in smf2-0_importer.xml ..
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 03, 2016, 05:08:13 pm
Just for clarification, smf2-0_importer.xml  or smf2-1_importer.xml?
Title: Re: [SMF 2.1] Importer
Post by: TE on June 04, 2016, 02:41:49 am
Quote from: badmonkey – Just for clarification, smf2-0_importer.xml  or smf2-1_importer.xml?
,2.1
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 04, 2016, 09:04:13 am
Well bummer.  Still a no go.  Looking at the file naming scheme with hashes, afaik everything appears correctly in the database.  The files on disk are not renamed with .elk.  Changing the extension from .dat doesn't make the attachment work.  I can't pinpoint the reason for the inoperable attachments.  Clearly there's a reason but so far no joy.  Uploading a new file works perfectly.  Any thoughts?  What am I missing?
Title: YRe: [SMF 2.1] Importer
Post by: TE on June 05, 2016, 01:37:43 am
Ups.. totally forgot they are using an extension as well.. I have to check the importer again, probably tomorrow..
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 18, 2016, 04:07:21 am
I just converted a 1.1 million post forum to Elkarte using the importer.  The real run was easier than the dry run.  The attachments imported just fine.  The only thing was there were legacy .dat copies in the directories.  Did a manual delete since there are over 100,000 files taking up unnecessary space.  Lol.  Anyway the site is up and running.  Thanks to everyone who invested hours of work on the importer.
Title: Re: [SMF 2.1] Importer
Post by: emanuele on June 18, 2016, 05:55:51 am
You mean the importer copied the .dat files twice one renaming them and one without?
Title: Re: [SMF 2.1] Importer
Post by: badmonkey on June 18, 2016, 09:28:17 am
Quote from: emanuele – You mean the importer copied the .dat files twice one renaming them and one without?


Correct.  We're on the same page, I think.  Didn't sleep much last night.  lol.   There would be one copy of attach_hash.dat and attach_hash.elk.  Not a big deal at all.  Everything functioned perfectly.  :)
Title: Re: [SMF 2.1] Importer
Post by: emanuele on June 18, 2016, 09:36:20 am
Cool!
Title: Re: [SMF 2.1] Importer
Post by: bigguy on March 20, 2018, 09:01:05 pm
What version of 2.1 does this work with. Is it up to date. I just ask because of the date on the last post. (Hello all, first post)
Title: Re: [SMF 2.1] Importer
Post by: ahrasis on March 20, 2018, 09:04:07 pm
Welcome @bigguy. Nice to see you around.
Title: Re: [SMF 2.1] Importer
Post by: bigguy on March 21, 2018, 05:17:39 am
Thank you. :)
Title: Re: [SMF 2.1] Importer
Post by: emanuele on March 22, 2018, 08:26:34 am
I don't think it's up to date, but I'm not sure what changes happened in the last year or two that could affect it.
Title: Re: [SMF 2.1] Importer
Post by: bigguy on March 22, 2018, 06:57:08 pm
I'm not sure either off the top of my head but they have made progress toward an RC1 release.
Title: Re: [SMF 2.1] Importer
Post by: radu81 on March 22, 2018, 07:34:02 pm
Welcome to elkarte bigguy!
Probably the best way to find out if the impoerter is working fine with SMF 2.1 is to try to do an import on a test forum, and if you have problems please report.
Title: Re: [SMF 2.1] Importer
Post by: Allan on March 23, 2018, 09:07:54 pm
@bigguy‍ I think @radu81‍  has the best option, try a test site first before doing it on a live site. 
Title: Re: [SMF 2.1] Importer
Post by: bigguy on March 23, 2018, 09:11:25 pm
I'll do that for sure. Might take me a bit but it will be done. :)
Title: Re: [SMF 2.1] Importer
Post by: Allan on March 23, 2018, 09:13:07 pm
The importer actually goes pretty well.