Skip to main content
[SMF 2.1] Importer Started by TE · · Read 51878 times 0 Members and 2 Guests are viewing this topic. previous topic - next topic

Re: [SMF 2.1] Importer

Reply #30

Yep, tep, I imagined it was set at php.ini level, I was just curious at the actual value.
I asked because at some point I wrote a modified version of convert.php (the original script) that was able to monitor the memory usage and avoid "out of memory" like this. I thought I added it to OI, but apparently I didn't... shame. :-[
Bugs creator.
Features destroyer.
Template killer.

Re: [SMF 2.1] Importer

Reply #31

Quote from: emanuele – Yep, tep, I imagined it was set at php.ini level, I was just curious at the actual value.
I asked because at some point I wrote a modified version of convert.php (the original script) that was able to monitor the memory usage and avoid "out of memory" like this. I thought I added it to OI, but apparently I didn't... shame. :-[

Indeed it could be useful.  But you can't be responsible for iffy server tuning on my part.  lol

Re: [SMF 2.1] Importer

Reply #32

Next hiccup:  attachments.  Directory paths are correct.  But they aren't displaying.  The files still exist in the attachment directories.  Has anyone had this issue?

Re: [SMF 2.1] Importer

Reply #33

Quote from: badmonkey – Next hiccup:  attachments.  Directory paths are correct.  But they aren't displaying.  The files still exist in the attachment directories.  Has anyone had this issue?
yep, the attachments import is still a bit bggy, especially if your SMF was 1.1 without file encryption before..
This is the attachments and avatar import I used to Import Ruth's Puppenstube.
Code: [Select]
<step>
<title>Importing attachments</title>
<detect>{$from_prefix}attachments</detect>
<destination>{$to_prefix}attachments</destination>
<presql>TRUNCATE {$to_prefix}attachments;</presql>
<preparsecode>
$request = $db->query("
SELECT value
FROM {$from_prefix}settings
WHERE variable='attachmentUploadDir';");
list ($smf_attachments_dir) = $db->fetch_row($request);

$smf_attachments_dir =  rtrim($smf_attachments_dir, '/');

if (empty($row['file_hash']))
{
$row['file_hash'] = createAttachmentFileHash($row['filename']);
$clean_name = $row['filename'];
$se = array("ä", "ö", "ü", "Ä", "Ö", "Ü", "ß");
$repl = array("a", "o", "u", "A", "O", "U", "ss");
$clean_name = str_replace($se, $repl, $clean_name);
$clean_name = strtr($clean_name, 'ŠŽšžŸÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝàáâãäåçèéêëìíîïñòóôõöøùúûüýÿ', 'SZszYAAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy');
$clean_name = strtr($clean_name, array('Þ' => 'TH', 'þ' => 'th', 'Ð' => 'DH', 'ð' => 'dh', 'ß' => 'ss', 'Œ' => 'OE', 'œ' => 'oe', 'Æ' => 'AE', 'æ' => 'ae', 'µ' => 'u'));
$clean_name = preg_replace(array('/\s/', '/[^\w_\.\-]/'), array('_', ''), $clean_name);
$enc_name = $row['id_attach'] . '_' . strtr($clean_name, '.', '_') . md5($clean_name);
$clean_name = preg_replace('~\.[\.]+~', '.', $clean_name);
$clean_name = str_replace('.', '_', $clean_name);

copy_file($smf_attachments_dir . '/' . $enc_name, $attachmentUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
copy_file($smf_attachments_dir . '/' . $clean_name, $attachmentUploadDir . '/' . $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
}
else
copy_file($smf_attachments_dir . '/' . $row['id_attach'] . '_' . $row['file_hash'], $attachmentUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
</preparsecode>
<query>
SELECT
id_attach, id_thumb, id_msg, id_member, attachment_type, filename, file_hash, size, downloads,
width, height, fileext, mime_type, id_folder
FROM {$from_prefix}attachments
WHERE id_msg != 0;
</query>
</step>
<step>
<title>Importing avatars</title>
<detect>{$from_prefix}attachments</detect>
<destination>{$to_prefix}attachments</destination>
<options>
<replace>true</replace>
</options>
<preparsecode>
// Try custom_avatar_dir first
$request = $db->query("
SELECT value
FROM {$from_prefix}settings
WHERE variable = 'custom_avatar_dir';");
list ($smf_attachments_dir) = $db->fetch_row($request);

if (empty($smf_attachments_dir))
{
$request = $db->query("
SELECT value
FROM {$from_prefix}settings
WHERE variable='attachmentUploadDir';");
list ($smf_attachments_dir) = $db->fetch_row($request);
}

if ($avatarUploadDir == $attachmentUploadDir)
$row['attachment_type'] = 0;

if (empty($row['file_hash']))
{
$row['file_hash'] = createAttachmentFileHash($row['filename']);
$clean_name = $row['filename'];
$se = array("ä", "ö", "ü", "Ä", "Ö", "Ü", "ß");
$repl = array("a", "o", "u", "A", "O", "U", "ss");
$clean_name = str_replace($se, $repl, $clean_name);
$clean_name = strtr($clean_name, 'ŠŽšžŸÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝàáâãäåçèéêëìíîïñòóôõöøùúûüýÿ', 'SZszYAAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy');
$clean_name = strtr($clean_name, array('Þ' => 'TH', 'þ' => 'th', 'Ð' => 'DH', 'ð' => 'dh', 'ß' => 'ss', 'Œ' => 'OE', 'œ' => 'oe', 'Æ' => 'AE', 'æ' => 'ae', 'µ' => 'u'));
$clean_name = preg_replace(array('/\s/', '/[^\w_\.\-]/'), array('_', ''), $clean_name);
$enc_name = $row['id_attach'] . '_' . strtr($clean_name, '.', '_') . md5($clean_name);
$clean_name = preg_replace('~\.[\.]+~', '.', $clean_name);

if(file_exists($smf_attachments_dir . '/' . $row['id_attach'] . '_' . str_replace('.', '_', $clean_name). md5($row['filename'] )))
copy_file($smf_attachments_dir . '/' . $row['id_attach'] . '_' . str_replace('.', '_', $clean_name) . md5($row['filename']), $avatarUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');

copy_file($smf_attachments_dir . '/' . $clean_name, $avatarUploadDir . '/' . $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
}
else
copy_file($smf_attachments_dir . '/' . $row['id_attach'] . '_' . $row['file_hash'], $avatarUploadDir . '/'. $row['id_attach'] . '_' . $row['file_hash'] . '.elk');
</preparsecode>
<query>
SELECT
id_attach, id_thumb, id_msg, id_member, attachment_type, filename, file_hash, size, downloads,
width, height, fileext, mime_type, id_folder
FROM {$from_prefix}attachments
WHERE id_member != 0;
</query>
</step>
You'd have to replace the attachments and avatars import in smf2-0_importer.xml ..
Thorsten "TE" Eurich
------------------------

 

Re: [SMF 2.1] Importer

Reply #34

Just for clarification, smf2-0_importer.xml  or smf2-1_importer.xml?

Re: [SMF 2.1] Importer

Reply #35

Quote from: badmonkey – Just for clarification, smf2-0_importer.xml  or smf2-1_importer.xml?
,2.1
Thorsten "TE" Eurich
------------------------

Re: [SMF 2.1] Importer

Reply #36

Well bummer.  Still a no go.  Looking at the file naming scheme with hashes, afaik everything appears correctly in the database.  The files on disk are not renamed with .elk.  Changing the extension from .dat doesn't make the attachment work.  I can't pinpoint the reason for the inoperable attachments.  Clearly there's a reason but so far no joy.  Uploading a new file works perfectly.  Any thoughts?  What am I missing?

YRe: [SMF 2.1] Importer

Reply #37

Ups.. totally forgot they are using an extension as well.. I have to check the importer again, probably tomorrow..
Thorsten "TE" Eurich
------------------------

Re: [SMF 2.1] Importer

Reply #38

I just converted a 1.1 million post forum to Elkarte using the importer.  The real run was easier than the dry run.  The attachments imported just fine.  The only thing was there were legacy .dat copies in the directories.  Did a manual delete since there are over 100,000 files taking up unnecessary space.  Lol.  Anyway the site is up and running.  Thanks to everyone who invested hours of work on the importer.

Re: [SMF 2.1] Importer

Reply #39

You mean the importer copied the .dat files twice one renaming them and one without?
Bugs creator.
Features destroyer.
Template killer.

Re: [SMF 2.1] Importer

Reply #40

Quote from: emanuele – You mean the importer copied the .dat files twice one renaming them and one without?


Correct.  We're on the same page, I think.  Didn't sleep much last night.  lol.   There would be one copy of attach_hash.dat and attach_hash.elk.  Not a big deal at all.  Everything functioned perfectly.  :)

Re: [SMF 2.1] Importer

Reply #41

Cool!
Bugs creator.
Features destroyer.
Template killer.

Re: [SMF 2.1] Importer

Reply #42

What version of 2.1 does this work with. Is it up to date. I just ask because of the date on the last post. (Hello all, first post)

Re: [SMF 2.1] Importer

Reply #43

Welcome @bigguy. Nice to see you around.

Re: [SMF 2.1] Importer

Reply #44

Thank you. :)