Skip to main content ↑ ↓
Missing attachments after OpenImporter import from SMF 2.0 Started by overscan · · Read 11019 times 0 Members and 1 Guest are viewing this topic. previous topic - next topic

Missing attachments after OpenImporter import from SMF 2.0

I am missing attachments after using Openimporter. I have done some investigation and it seems missing attachments have different pattern of attachment filenames e.g.:

Original filename = L-227-14.GIF
Attachment filename = 8456_L-227-14_GIFb8f84b0c45c3a15688e133d1c42ebaec

I believe these attachments date back to when the forum was SMF 1.0 or SMF 1.1.

More recent attachments from SMF days are e.g.

84566_ac605dc8170a65c44d4ef597fb9ab090c0b49db0

I note that the second type of filename is purely lowercase alphanumeric and underscores whereas the first is upper and lower and includes hyphens. The length of filename is also variable as it includes the original filename.
 Is it possible that OpenImporter is failing due to filename issues?

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #1 –

Let me ask a few questions just to have a better idea of the problem (that relating to attachments pre-hashing may be messy as usual :P).

Let's use L-227-14.GIF as a reference.

First of all: does the file exist in the Elk attachments directory?
Does it exist in the SMF attachments directory?

If the answer is yes to both (otherwise feel free to stop here), what do you have in the ElkArte database regarding to that file? In particular what are the values of the files: id_folder, attachment_type, filename, file_hash and fileext? (You should be able to find it searching by filename or by id_msg knowing the message the file was attached.)
Bugs creator.
Features destroyer.
Template killer.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #2 –

The files  are missing in the Elk attachment folder but are present in the SMF attachment folder.

I'll check what is in the database.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #3 –

Not sure if its relevant but every time I do the import, first try it stops:

--
http://elkarte.s**********.co.uk/importer/import.php?step=1&substep=18&start=355000

Error:
Trying to get property of non-object
/home/sites/elkarte.s**********.co.uk/web/importer/OpenImporter/Importer.php
470

But subsequent runs always work.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #4 –

Difficult to say, it may be relevant, but at the moment the errors thrown out by OI are not very useful...[1]

Let me know what you have in the database and let's see what we can do about the problem at hand. ;) (I'm going bed now, I'll be back in about 12 hours. 8))
1 Tracked the issue so we don't forget. :D↵
Bugs creator.
Features destroyer.
Template killer.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #5 –

So - the database misses a number of fields for the attachment - filehash and MIME type are blank.

The thumbnail was regenerated since SMF 2.0 has has all missing fields populated - and successfully migrated. Therefore it probably is the missing filehash causing the import issue.

The attachment works fine in SMF 2.0 despite the missing fields.
Last Edit: March 07, 2015, 07:00:43 pm by overscan

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #6 –

http://support.simplemachines.org/function_db/index.php?action=view_function;id=238

getAttachmentFilename (string $filename, int $attachment_id[, string $dir[, bool $new]])

It seems likely that this command returns the filename regardless of whether the filename is stored in the database. It looks to me like none of the new attachment table columns added in SMF 2.0 got filled in when the forum was migrated previously (maybe SMF 1.1 - 2.0)

 A quick fix might be a script to generate the missing filehash and put it in the database.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #7 –

Incidentally, the missing MIME types explains issues seen in Internet Explorer with my forum and attachments for some years :)

I need a repair script to find all attachments with missing data, generate mime type from file extension and also find the filename and put both values in the table. Then the upgrade will be doable.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #8 –

Okay, one last check.
The name of the files in the attachments directory follow the scheme {id_attach}_randomstring.
Can you check if there is a file whose name starts with the id of the L-227-14.GIF attachment?
Bugs creator.
Features destroyer.
Template killer.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #9 –

Attachment 8456 in DB looks like this: id_attach|id_thumb|id_msg|id_member|attachment_type|filename|size|downloads|width|height|file_hash|fileext|mime_type|id_folder|approved
8456|446287|7666|0|0|L-227-14.GIF|53618|1995|800|398|[null]|gif|[null]|1|1

There are no files starting with 8456 except  8456_L-227-14_GIFb8f84b0c45c3a15688e133d1c42ebaec in SMF attachment folder.

There are no files starting with 8456 in Elkarte attachment folder.
Last Edit: March 08, 2015, 05:12:25 am by overscan

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #10 –

Looking through OpenImporter code I saw in SMF 1.1 importer code to fix the missing MIME type.

   if (empty($row['mime_type']))
   {
      $fileext = '';
      $mimetype = '';
      $is_thumb = false;
      if (preg_match('/\.(jpg|jpeg|gif|png)(_thumb)?$/i',$row['filename'],$m))
      {
         $fileext = strtolower($m[1]);
         $is_thumb = !empty($m[2]);
         if (empty($row['mime_type']))
         {
            // AFAIK, all thumbnails got created as PNG
            if ($is_thumb) $mimetype = 'image/png';
            elseif ($fileext == 'jpg') $mimetype = 'image/jpeg';
            else $mimetype = 'image/'.$fileext;
         }

This is not included in SMF 2.0 importer. SMF 2.0 importer does have code for missing filehash:

   if (empty($row['file_hash']))
   {
      $row['file_hash'] = createAttachmentFileHash($row['filename']);
      $source_file = $row['filename'];

This presumably does not generate "8456_L-227-14_GIFb8f84b0c45c3a15688e133d1c42ebaec" but "8456_b8f84b0c45c3a15688e133d1c42ebaec" which then cannot be copied as it doesn't exist.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #11 –

BTW this specific attachment is from October 2006 ;)

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #12 –

That may very well be it.
I may not be able to do anything today (relatives and hospital) I'll try to have a look ASAP. ;)
Bugs creator.
Features destroyer.
Template killer.

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #13 –

SMF 2.1 upgrade handled this correctly. I believe this is the relevant code from upgrade_2-1_mysql.sql

// Iterate through the possible attachment names until we find the one that exists

         $oldFile = $currentFolder . '/' . $row['id_attach']. '' . strtr($row['filename'], '.', '') . md5($row['filename']);


This checks for attachments in the old style, the strtr() replaces dots in the filename with underscores. This will locate my old school SMF 1.0 attachments :) I think the problem is not generating the hash, its the assumption the filename will be [attachmentid]_[file_hash] when for these old style attachments its [attachmentid]_[transformedfilename][file_hash]

If only I could code in PHP I could fix it myself ;)
Last Edit: March 08, 2015, 05:49:20 am by overscan

Re: Missing attachments after OpenImporter import from SMF 2.0

Reply #14 –

Quote from: emanuele – That may very well be it.
I may not be able to do anything today (relatives and hospital) I'll try to have a look ASAP. ;)

No specific rush here. Getting closer to a possible migration, but still weeks/months away at earliest.