I am missing attachments after using Openimporter. I have done some investigation and it seems missing attachments have different pattern of attachment filenames e.g.:
Original filename = L-227-14.GIF
Attachment filename = 8456_L-227-14_GIFb8f84b0c45c3a15688e133d1c42ebaec
I believe these attachments date back to when the forum was SMF 1.0 or SMF 1.1.
More recent attachments from SMF days are e.g.
84566_ac605dc8170a65c44d4ef597fb9ab090c0b49db0
I note that the second type of filename is purely lowercase alphanumeric and underscores whereas the first is upper and lower and includes hyphens. The length of filename is also variable as it includes the original filename.
Is it possible that OpenImporter is failing due to filename issues?
Let me ask a few questions just to have a better idea of the problem (that relating to attachments pre-hashing may be messy as usual :P).
Let's use L-227-14.GIF as a reference.
First of all: does the file exist in the Elk attachments directory?
Does it exist in the SMF attachments directory?
If the answer is yes to both (otherwise feel free to stop here), what do you have in the ElkArte database regarding to that file? In particular what are the values of the files: id_folder, attachment_type, filename, file_hash and fileext? (You should be able to find it searching by filename or by id_msg knowing the message the file was attached.)
The files are missing in the Elk attachment folder but are present in the SMF attachment folder.
I'll check what is in the database.
Not sure if its relevant but every time I do the import, first try it stops:
--
http://elkarte.s**********.co.uk/importer/import.php?step=1&substep=18&start=355000
Error:
Trying to get property of non-object
/home/sites/elkarte.s**********.co.uk/web/importer/OpenImporter/Importer.php
470
But subsequent runs always work.
Difficult to say, it may be relevant, but at the moment the errors thrown out by OI are not very useful...
Let me know what you have in the database and let's see what we can do about the problem at hand. ;) (I'm going bed now, I'll be back in about 12 hours. 8))
So - the database misses a number of fields for the attachment - filehash and MIME type are blank.
The thumbnail was regenerated since SMF 2.0 has has all missing fields populated - and successfully migrated. Therefore it probably is the missing filehash causing the import issue.
The attachment works fine in SMF 2.0 despite the missing fields.
http://support.simplemachines.org/function_db/index.php?action=view_function;id=238
getAttachmentFilename (string $filename, int $attachment_id[, string $dir[, bool $new]])
It seems likely that this command returns the filename regardless of whether the filename is stored in the database. It looks to me like none of the new attachment table columns added in SMF 2.0 got filled in when the forum was migrated previously (maybe SMF 1.1 - 2.0)
A quick fix might be a script to generate the missing filehash and put it in the database.
Incidentally, the missing MIME types explains issues seen in Internet Explorer with my forum and attachments for some years :)
I need a repair script to find all attachments with missing data, generate mime type from file extension and also find the filename and put both values in the table. Then the upgrade will be doable.
Okay, one last check.
The name of the files in the attachments directory follow the scheme {id_attach}_randomstring.
Can you check if there is a file whose name starts with the id of the L-227-14.GIF attachment?
Attachment 8456 in DB looks like this: id_attach|id_thumb|id_msg|id_member|attachment_type|filename|size|downloads|width|height|file_hash|fileext|mime_type|id_folder|approved
8456|446287|7666|0|0|L-227-14.GIF|53618|1995|800|398|[null]|gif|[null]|1|1
There are no files starting with 8456 except 8456_L-227-14_GIFb8f84b0c45c3a15688e133d1c42ebaec in SMF attachment folder.
There are no files starting with 8456 in Elkarte attachment folder.
Looking through OpenImporter code I saw in SMF 1.1 importer code to fix the missing MIME type.
if (empty($row['mime_type']))
{
$fileext = '';
$mimetype = '';
$is_thumb = false;
if (preg_match('/\.(jpg|jpeg|gif|png)(_thumb)?$/i',$row['filename'],$m))
{
$fileext = strtolower($m[1]);
$is_thumb = !empty($m[2]);
if (empty($row['mime_type']))
{
// AFAIK, all thumbnails got created as PNG
if ($is_thumb) $mimetype = 'image/png';
elseif ($fileext == 'jpg') $mimetype = 'image/jpeg';
else $mimetype = 'image/'.$fileext;
}
This is not included in SMF 2.0 importer. SMF 2.0 importer does have code for missing filehash:
if (empty($row['file_hash']))
{
$row['file_hash'] = createAttachmentFileHash($row['filename']);
$source_file = $row['filename'];
This presumably does not generate "8456_L-227-14_GIFb8f84b0c45c3a15688e133d1c42ebaec" but "8456_b8f84b0c45c3a15688e133d1c42ebaec" which then cannot be copied as it doesn't exist.
BTW this specific attachment is from October 2006 ;)
That may very well be it.
I may not be able to do anything today (relatives and hospital) I'll try to have a look ASAP. ;)
SMF 2.1 upgrade handled this correctly. I believe this is the relevant code from upgrade_2-1_mysql.sql
// Iterate through the possible attachment names until we find the one that exists
$oldFile = $currentFolder . '/' . $row['id_attach']. '' . strtr($row['filename'], '.', '') . md5($row['filename']);
This checks for attachments in the old style, the strtr() replaces dots in the filename with underscores. This will locate my old school SMF 1.0 attachments :) I think the problem is not generating the hash, its the assumption the filename will be [attachmentid]_[file_hash] when for these old style attachments its [attachmentid]_[transformedfilename][file_hash]
If only I could code in PHP I could fix it myself ;)
No specific rush here. Getting closer to a possible migration, but still weeks/months away at earliest.
Oh okay, better! :D
So - finally have time to look at this again. Any developments?
hmm... that was a while ago.
I worked quite a bit on the attachments handling in the development version of OI.
Since it has been a while since last time I tried I don't want to say "it's fixed" because I don't remember (I like to be honest...), but I'm confident it is fixed.
There were lots of mistakes around, and the development version should handle them correctly.
ETA: if you want to give it a try, use the package attached to the first message of the topic http://www.elkarte.net/community/index.php?topic=2118.0
I'll give it a go and let you know how I get on, thanks!