ElkArte Community

General => OpenImporter => Topic started by: emanuele on October 14, 2014, 01:01:07 am

Title: OpenImporter
Post by: emanuele on October 14, 2014, 01:01:07 am
Since I can't seem to find a dedicated topic, so here it is. :P

Yesterday I went a bit mad (...no, wait scratch that I'm already mad, so it was probably it was just I was writing bugs as usual :P) and started breaking that one apart too... lol

https://github.com/emanuele45/openimporter/compare/separation?expand=1

I started splitting the code into multiple files, easier to read and maintain.
Then I removed some static classes in favour of objects.
I also tried to start the splitting of the computing and the response (this is currently badly broken).
And then I re-organized a bit also the files.

Since the project is called Open Importer I think it is meant to be able to convert between any two systems, or at least I would love to see something like that, so I started "moving" the code towards that goal.
First step I moved the xml importers into a subdirectory (Importers) and into that one I created another subdir that represents the "destination" platform (for now there is only ElkArte 1.0...).

For now that's all, the first page (script selection) works, but I'm pretty sure everything else is broken and doesn't work (because of the response/template code). lol

ETA: The attached file is a development version, pay attention and always take a backup before trying anything!!
Title: Re: OpenImporter
Post by: Allan on October 14, 2014, 02:30:44 am
I think this is great, reminds me alot of the merge system MyBB has, which is actually very good.
Title: Re: OpenImporter
Post by: TE on October 14, 2014, 08:02:58 am
Since the project is called Open Importer I think it is meant to be able to convert between any two systems,
Yep, that's why the name was choosen... Basically it's the same for all systems.. move data from one database structure to another..

I'm fine with merging, even if it is totally broken for now.. Maybe we should create a "developement" branch until it's working?
Title: Re: OpenImporter
Post by: Spuds on October 14, 2014, 12:17:35 pm
A development branch makes the most sense to me
Title: Re: OpenImporter
Post by: emanuele on October 14, 2014, 12:35:59 pm
The dev branch could be an idea.
I wouldn't merge that one now, because is quite a mess... lol
It would be better to clean it up a bit and have it at least somehow working. :)
Title: Re: OpenImporter
Post by: emanuele on October 14, 2014, 04:47:56 pm
Today some more stuff related to steps 2 and 3, in particular I moved the implementation out the the main import class, into a platform specific file (part into the xml and part into a php).
Since step 2 is common to all the importers for a certain "destination platform" so I moved the code to another class and the xml now just instantiates the class and runs the proper step (this is a bit redundant and I would prefer something less verbose, but for the moment it could work). The same considerations apply to step3.

ETA: I moved a bit further and removed the all the substeps of the step 2 from the xml and delegated everything to the class. The xml now is used to specify the name of the class.
Title: Re: OpenImporter
Post by: emanuele on October 16, 2014, 02:34:47 pm
Still changing my mind quite frequently.
In the last commits I created pairs of xml and php files, moving part of the <general> tag to php fles.
Steps 2 and 3 are now into a class file, along with some special methods that may be necessary in step 1 (see elkarte1.0_importer.php).
All the needed files are automatically loaded when the importer is chosen and the classes are named following a convention (the ones for the destination platform, ElkArte for now) while the source-specific class names are specified in the <general> section of the xml file.
Title: Re: OpenImporter
Post by: emanuele on October 20, 2014, 09:14:52 pm
Back here!

One thing that would be terribly useful, would be to "collect" at least one dummy (with fake data) database for each system of the supported platforms. This would allow two things: 1) at some point set up proper testing, and 2) give a base to let people help in refining existing databases.

If you have a database of any platform and you want to help, feel free to post here or get in contact by PM.
Of course the database should be clean from any personal data and any "real-life" reference (so for example the messages should be just dummy messages, even better if a set of "standard" messages are agreed, so that testing the conversions will be much easier and straightforward).

At the moment there are converters from:
* UBB 7.5
* mybb 1.6
* phpBoost 3
* phpBB 3
* smf 2.0
* vBulletin4
* WBB 3
* WordPress 3
* XenForo 1.1
to ElkArte.
I plan on porting the ones available for the Wedge version in the next few days, so there will be two destination platforms.

If you want to help, an easy way is to provide any of these dummy (remember do not post databases with real data in public) databases. If you have a real one and you don't know how to clean it up, feel free to ask. ;D
Databases of open source (including ElkArte) or freeware projects (like Wedge) should be easy to create: just install create a user, do some posts and at least the basics are covered. Of course it would be wonderful to be able to convert also likes, custom fields, etc., so have them in the system would help.

Todo:
* define a list of standard messages and users for all the tests.
Title: Re: OpenImporter
Post by: Jorin on October 21, 2014, 11:04:46 am
Wordpress? Is there a way to migrate Wordpress blog entries into ElkArte forum posts?
Title: Re: OpenImporter
Post by: emanuele on October 21, 2014, 11:09:23 am
I didn't test it, so I guess only @TE can tell how "complete" the conversion can be.
Title: Re: OpenImporter
Post by: forumsearch0r2 on October 22, 2014, 02:59:06 am
I'd assume that'd work via RSS export/import easily.
Title: Re: OpenImporter
Post by: TE on October 23, 2014, 10:47:40 am
Wordpress? Is there a way to migrate Wordpress blog entries into ElkArte forum posts?
Not just blog entries to Forum Posts .. users and some other things will also be migrated to ElkArte.
Title: Re: OpenImporter
Post by: radu81 on October 23, 2014, 09:16:26 pm
Sorry if this is not the right topic, but converting from smf with multiple folders for attachments it did not work with this version of open importer
Title: Re: OpenImporter
Post by: emanuele on October 23, 2014, 09:41:55 pm
The topic it's okay. ;)

Yep, attachments seem a bit broken at the moment.
Title: Re: OpenImporter
Post by: emanuele on October 24, 2014, 01:09:26 am
https://travis-ci.org/emanuele45/openimporter/builds/38876157 ;D

Of course it doesn't mean the conversions are working, just that I added the skeleton of integration testing. ;)
Title: Re: OpenImporter
Post by: emanuele on October 27, 2014, 01:52:25 am
I know, I know, just numbers. O:-)
https://scrutinizer-ci.com/g/emanuele45/openimporter/?branch=master
https://scrutinizer-ci.com/g/emanuele45/openimporter/?branch=separation

:D

Well, not really *that* important, it only means I spend a few days splitting up methods and multiplying files.
Funny thing is that the importer is still badly broken (and I'm not yet sure how to fix it ROFL), but the code apparently looks good! LOL
Title: Re: OpenImporter
Post by: Spuds on October 27, 2014, 02:14:29 am
Well it may not work, but it looks damn good doing it :P

All kidding aside,  that's a pretty awesome change !
Title: Re: OpenImporter
Post by: emanuele on October 27, 2014, 11:58:33 pm
Spuds, Mr. Regex, any idea how to convert the following /e to a preg_replace_callback? O:-)
https://github.com/emanuele45/openimporter/blob/separation/importer/OpenImporter/Utils.php#L252
I found something similar in Elk code, but I'm too scared to change a preg_replace involving charsets without tests... O_O
Title: Re: OpenImporter
Post by: emanuele on October 28, 2014, 12:07:15 am
All kidding aside,  that's a pretty awesome change !
Let's add some more kidding: I went from 3.67 to 7.83 in 15 days and 150 commits.
Elk is at 3.23 how many commits to 3.67? :P
Title: Re: OpenImporter
Post by: Spuds on October 28, 2014, 12:36:34 am
That is pretty much what the function replaceEntities__callback will do ... I replaced those hideous nested ternary create functions with permanent functions since the former hurt my brain :P

So just copy that function to the importer and do
Code: [Select]
preg_replace_callback('~(&#(\d{1,7}|x[0-9a-fA-F]{1,6});)~', 'replaceEntities__callback', $buf);

Now that second regex one seems a bit mysterious to me, I'm trying to see a pattern that that the first does not cover ... well it would capture &#xx123; but thats not a valid code ... maybe it was added for some damaged input?  I'm tempted to say just the first one is enough as the function checks for hex numbers.
Title: Re: OpenImporter
Post by: Spuds on October 28, 2014, 12:45:14 am
All kidding aside,  that's a pretty awesome change !
Let's add some more kidding: I went from 3.67 to 7.83 in 15 days and 150 commits.
Elk is at 3.23 how many commits to 3.67? :P
Well it took 10K to move it from 0 to 3.23, so if my math is right, its a crap load more !
Title: Re: OpenImporter
Post by: emanuele on October 29, 2014, 03:21:48 pm
preg_replace_callback replaced: https://github.com/emanuele45/openimporter/commit/e33e8a39004e823265cff8172ef6d7f53b7c2edd
Thanks! ;D

I think I almost finished splitting up and fixing everything:
https://github.com/emanuele45/openimporter/tree/separation
I was able to use it to properly (as far as I can tell) import an SMF database into ElkArte. This is at the moment the only import I'm reasonably sure that works.

There is still some code Wedge-specific around, but I completely isolated it, so that move it out when time comes should be pretty easy.

Now step 2: test all the conversions present so far and write tests for them... actually, before that generic tests for ManageImporter, Importer and XmlProcessor are more important... provided these classes are already testable.
Title: Re: OpenImporter
Post by: Spuds on October 29, 2014, 03:59:23 pm
That is nothing short of awesome sauce !
Title: Re: OpenImporter
Post by: Jorin on October 29, 2014, 06:55:15 pm
https://github.com/emanuele45/openimporter/tree/separation
I was able to use it to properly (as far as I can tell) import an SMF database into ElkArte. This is at the moment the only import I'm reasonably sure that works.

Do you want me to test this importer with my SMF?
Title: Re: OpenImporter
Post by: radu81 on October 29, 2014, 07:54:11 pm
Are there any  improvements with multiple folders for attachments?  The last importer I test does not import attachment from smf with multiple folders
Title: Re: OpenImporter
Post by: emanuele on October 30, 2014, 12:29:15 am
I was re-checking the code and there is still an edge case with avatars, I think I fixed it in MyBB, but if you have the avatars stored as attachments it may not be able to import them...

@Jorin of course any testing is very welcome!

@radu81 in theory (as usual lol) it should work, it should also move the attachments to the corresponding directory they were before (and create such directory if it doesn't exist), though at the moment it's all untested code...
Title: Re: OpenImporter
Post by: emanuele on October 30, 2014, 12:33:35 am
radu81 did you get the mention for the post above that one?
I see a strange behaviour I'm not sure about.
Title: Re: OpenImporter
Post by: radu81 on October 30, 2014, 12:43:21 am
Have a look at scrrenshots. I will test the importer, thank you Emanuele
Title: Re: OpenImporter
Post by: Jorin on October 30, 2014, 07:25:25 am
...but if you have the avatars stored as attachments it may not be able to import them...

I will try and test migrate tonight, at the same time with building nice houses on the lake in Minecraft on my PS4.  O:-)
Title: Re: OpenImporter
Post by: radu81 on October 30, 2014, 08:12:04 pm
I would like to try the new importer, but is completely different than the old one

Can I have some basic indications on how to proceed? thanks in advance
Title: Re: OpenImporter
Post by: emanuele on October 30, 2014, 10:33:10 pm
Very quick and dirty guide:
* download
* unzip
* take the content of the "importer" directory and upload it somewhere web-accessible
* go to: http://www.yoursite.tld/dir/you/uploaded/importer.php
* follow the instructions
Title: Re: OpenImporter
Post by: radu81 on October 30, 2014, 11:24:34 pm
I installed elkarte under the smf folder. Inside elkarte I put the importer. I set the paths to smf and elk but importer remained at 0%. So I tried again but I had an error in import.php. I'll try later, I'm on tablet now.
Title: Re: OpenImporter
Post by: emanuele on October 30, 2014, 11:29:45 pm
Drop here any error you see. ;D

I'll test it again tomorrow.
Title: Re: OpenImporter
Post by: radu81 on October 31, 2014, 12:26:10 am
I tried again on a fresh copy of my smf forum. The importer just remains to step 1. I attach a screenshot

after restarting the importer I get this
You must select at least one step to import.
imp.error_line 297
imp.error_file C:\wamp\www\s***2014\s***elk\OpenImporter\Importer.php

so no way to restart the importer. (only in incognito mode with chrome)
Title: Re: OpenImporter
Post by: emanuele on October 31, 2014, 01:13:43 am
hmm... looks almost like I forgot to add a file... :-\
Title: Re: OpenImporter
Post by: emanuele on October 31, 2014, 10:10:21 pm
I did some more work and tests.
The attached version works for me on my old SMF 1.1 db upgraded to 2.0 and then converted with OI.

At a quick glance boards, topics, posts and PMs look fine.
I don't have attachments, so I don't know about that step...
Title: Re: OpenImporter
Post by: radu81 on November 01, 2014, 01:00:17 pm
I tried the importer in attachment but I get this after setting the paths and click continue

Trying to get property of non-object
C:\wamp\www\skodaelk\OpenImporter\Importer.php
469
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 01:35:26 pm
O_O

Okay, this error is completely unexpected...

Line 469 is a loop through the steps of the importer, though at that point, unless of meteorites the xml should be loaded...
I'll try on the windows machine, give me half an hour.
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 02:14:54 pm
At least now I know what going on, the importer strips the directory separator between the source (directory) and the destination (file) on windows, so elkarte1.0/smf2-0_importer.xml becomes elkarte1.0smf2-0_importer.xml and it fails loading the importer.
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 02:39:23 pm
Could you try what fixed the issue for me?
In importer/OpenImporter/Template.php:
Code: (find) [Select]
var url = "import.php?action=validate&xml=true&" + string + "=" + target.value.replace(/\/+$/g, "") + "&import_script=', $this->response->script , '";
Code: (replace with) [Select]
var url = "import.php?action=validate&xml=true&" + string + "=" + target.value.replace(/\/+$/g, "") + "&import_script=', addslashes($this->response->script) , '";
it's more a workaround than a proper fix, but for now it should work.
Title: Re: OpenImporter
Post by: radu81 on November 01, 2014, 03:19:25 pm
Thank you very much Emanuele, that worked!

I just made my first complete import. The attachments were all imported (I use multiple attachments in SMF). I just had to run the fix from Admin -> Maintenance -> Attachments , and the paths were corrected

p.s. there is only a small error when I restart the importer (if the first importer was not finished) I get this:
You must select at least one step to import.
imp.error_line 297
imp.error_file C:\wamp\www\skodaelk\OpenImporter\Importer.php
To solve that I have to go on another browser or use Incognito mode
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 03:30:08 pm
Thank you very much Emanuele, that worked!
Great!

I just made my first complete import. The attachments were all imported (I use multiple attachments in SMF). I just had to run the fix from Admin -> Maintenance -> Attachments , and the paths were corrected
That shouldn't happen, though it's not "terrible".

p.s. there is only a small error when I restart the importer (if the first importer was not finished) I get this:
You must select at least one step to import.
imp.error_line 297
imp.error_file C:\wamp\www\skodaelk\OpenImporter\Importer.php
To solve that I have to go on another browser or use Incognito mode
Yeah, I have to unset the session when the import is finished.
Title: Re: OpenImporter
Post by: radu81 on November 01, 2014, 04:30:50 pm
I have custom fields in user profile and I noticed that there were not imported. The open importer does not handle custom fields?

I also noticed that avatars were not imported (for this I will do a new import on a fresh install, in smf I have a mod for choosing avatar when registering)
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 04:40:47 pm
I didn't check deeply the smf-to-elk converter, but I guess it doesn't move (yet) the custom fields.

What configuration do you use for the avatars?
As attachment? Or did you specify a custom directory?
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 04:56:00 pm
Yes, custom fields are easy:
Code: [Select]
	<step>
<title>Importing custom fields</title>
<detect>{$from_prefix}custom_fields</detect>
<destination>{$to_prefix}custom_fields</destination>
<presql>TRUNCATE {$to_prefix}custom_fields;</presql>
<query>
SELECT
id_field, col_name, field_name, field_desc, field_type, field_length,
field_options, mask, show_reg, show_display, show_profile, private,
active, bbc, can_search, default_value, enclose, placement
FROM {$from_prefix}custom_fields;
</query>
</step>
<step>
<title>Importing custom fields data</title>
<detect>{$from_prefix}custom_fields</detect>
<destination>{$to_prefix}custom_fields_data</destination>
<presql>TRUNCATE {$to_prefix}custom_fields_data;</presql>
<query>
SELECT
id_member, variable, value
FROM {$from_prefix}themes
WHERE variable LIKE 'cust\_%';
</query>
</step>
added to /Importers/elkarte1.0/smf2-0_importer.xml following the XML structure. ;)

Less easy may be the instant message fields (ICQ, AIM, MSN, ???), but... honestly... is there anyone still using any of them? O:-)
Title: Re: OpenImporter
Post by: Jorin on November 01, 2014, 06:41:15 pm
I will wait for a new package of the importer with the latest fixes, because I use custom profile fields.  :D
Title: Re: OpenImporter
Post by: radu81 on November 01, 2014, 07:37:11 pm

What configuration do you use for the avatars?
As attachment? Or did you specify a custom directory?
I have some custom avatars uploaded in my folder /avatars (not imported).
The uploaded avatars are saved as attachments, I think in /attachments folder because in /avatars there are only my custom avatars.

Yes, custom fields are easy:
added to /Importers/elkarte1.0/smf2-0_importer.xml following the XML structure. ;)

Less easy may be the instant message fields (ICQ, AIM, MSN, ???), but... honestly... is there anyone still using any of them? O:-)
that worked like a charm. Custom fields imported, thanks again emanuele!
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 10:38:54 pm
I have some custom avatars uploaded in my folder /avatars (not imported).
hmm... I didn't thought about that.
Avatars are still a bit messy because of the multiple options (from gallery, uploaded, stored as attachments, stored in a directory, etc.), I have to set up a proper testing environment... :-\

The uploaded avatars are saved as attachments, I think in /attachments folder because in /avatars there are only my custom avatars.
And also these ones are not imported?
Just to know exactly were I should look. ;)

that worked like a charm. Custom fields imported, thanks again emanuele!
Great! :D

I tried to change a bit the first page. O:-)
Title: Re: OpenImporter
Post by: emanuele on November 01, 2014, 11:26:24 pm
Okay, and with one of the last commits I removed entirely any system-specific code I was aware of from the "core". Now it should be ready to go beyond ElkArte, SMF and Wedge as "destination" platforms.

If I have time, tomorrow I'll try to write a converter from Elk to WordPress to see if it is really ready. :)
Title: Re: OpenImporter
Post by: radu81 on November 02, 2014, 12:01:47 am
@emanuele not that I want you to convert my forum for me but if could be useful to you I can send you a copy of my files and db to make some tests
Title: Re: OpenImporter
Post by: emanuele on November 02, 2014, 02:38:01 pm
That's an option, thanks! :)

But, maybe, it's possible to set up a matrix of all the combinations so that we can get all the cases imported without troubles.
Let's see.

We are now talking about SMF 2.0.
In that software avatars can be:
* from a gallery
* from an url (either simply linked or "uploaded", see next point)
* uploaded (either as attachment or in a custom directory).
I hope I didn't miss anything.
In that set of possibilities, we know that:

As far as I remember, the same applies to ElkArte as well. That makes things easier.
Case 1) data in the members table is moved with members, so nothing to do. <= done! Then there is just to move the entire gallery to the new location (that actually we can just guess I think, but it's better than nothing), so a "move this directory to the new place" should be enough. <= todo.
Case 2) data in the members table is moved with members, so nothing to do. <= done! What's left is move the avatars from the custom dir to the new one. <= I think it's already done somehow, but I'm not entirely sure. To check.
Case 3) requires 2 things: a) move the attachment to the new attachment/s dir, b) re-create the attachment in the database. I wonder if we can just take care of the substep b in the importing of the attachments, it is in fact the same thing. Then, if b is taken care of, substep a would just be a matter of copying files where they should be.
Title: Re: OpenImporter
Post by: emanuele on November 02, 2014, 02:57:26 pm
@Spuds I have a question for you, master of phpunit. 8)

I would really like to add tests for the converters, and I'm trying to figure out how to do it.
My idea is:
* have a "forum" (database + files)
* install the forum
* run the converter
* compare with the expected result (database + files)
Let's assume I'm going to use Travis-ci.
Do you think it is possible to do the loop install/convert/compare for each test (so that only one forum is installed at a time) in script, or it is necessary to install all of them in the "before script" and then run the tests and compares in the script section?
Title: Re: OpenImporter
Post by: Spuds on November 02, 2014, 06:19:12 pm
I think you could do something like that in the setup/teardown functions in phpunit.

Maybe create an abstract class that extends to implement the before/after each testClass file you have defined.  Just include your abstract class in bootstrp or in the .yaml file.  There are also methods that run before and after test method of the class but I don't think those will help with this.

Code: [Select]
abstract class OpenImp extends PHPUnit_Framework_TestCase
{
// called before the first test of a testClass
public static function setUpBeforeClass()
{
}

// called after the final test of a testClass
public static function tearDownAfterClass()
{
}

// other functions you may want to use in your testcases
    public function other_stuff_i_need(){};
}
Title: Re: OpenImporter
Post by: emanuele on November 03, 2014, 08:44:56 pm
All that stuff is now in "master". :D

@radu81 thank you very much for the data, I was able to squash some other bugs and make the code simpler! :D
Now it imports should import ( :P ) all the avatar irrespective of where they are, unless of some very odd problems.

@Spuds thanks for the hint! I'll give it a try and see what I can come up with! (A mess. LOL)
Title: Re: OpenImporter
Post by: emanuele on November 04, 2014, 12:11:41 am
And here it is an experiment.

Instructions:
* unzip
* upload
* point to import.php

That's an experiment to understand if it is worth exploring the route of the phar archive.
A phar archive is a "special" php file that includes the entire application into a single file (actually it needs two: the phar and a php to run the phar, meh). I thought this may be easier to handle for the "final users" in that specific case, since it would be a "one shot" and have lots of files and directories to deal with could be a little pain, while two files are much easier to track in the process. Also in theory the released package doesn't require any change to the files, etc.

Anyway, I tried this solution, though reading around there are some configurations that do not like phar (e.g. suhosin) and if it doesn't work on the majority of systems, it may not be worth to maintain a double system of distribution (that would create confusion I think).

Let's see.

ETA: package removed.
Title: Re: OpenImporter
Post by: Spuds on November 04, 2014, 02:39:10 am
phar-out baby, I'm going to give it a try !

I should have mentioned that after you create your abstract class then your tests extend that instead of extends PHPUnit_Framework_TestCase so myTest extends OpenImp  yada yada ... Just a way to call those before and after things at the top level of each test class.
Title: Re: OpenImporter
Post by: radu81 on November 04, 2014, 07:42:09 pm
Hi @emanuele ant thanks for the importer, but I have a problem. Have a look

Code: [Select]
Importing...

Importing settings... ✔
Importing members... ✔
Importing attachments...

( ! ) Fatal error: Maximum function nesting level of '100' reached, aborting! in phar://C:/wamp/www/skodaelk/openimporter.phar/OpenImporter/Utils.php on line 88
Call Stack
# Time Memory Function Location
1 0.0003 690488 {main}( ) ..\import.php:0
2 0.0058 951360 require_once( 'C:\wamp\www\skodaelk\openimporter.phar' ) ..\import.php:4
3 0.0059 945168 include( 'phar://C:/wamp/www/skodaelk/openimporter.phar/index.php' ) ..\openimporter.phar:9
4 0.0063 987952 require_once( 'phar://C:/wamp/www/skodaelk/openimporter.phar/import.php' ) ..(null):1
5 1.3333 2557488 ImportManager->process( ) ..(null):77
6 1.3334 2558624 call_user_func ( ) ..(null):236
7 1.3334 2558624 ImportManager->doStep1( ) ..(null):236
8 1.4119 2725448 Importer->doStep1( ) ..(null):471
9 2.7881 2739568 XmlProcessor->processSteps( ) ..(null):516
10 2.7935 2741112 XmlProcessor->doSql( ) ..(null):99
11 2.7990 2810544 XmlProcessor->prepareRow( ) ..(null):157
12 2.7991 2809736 eval( ' moveAttachment($row, $this->db, $this->config->from_prefix, $this->step1_importer->getAttachDir($row)); ' ) ..(null):212
13 2.8001 2809736 moveAttachment( ) ..(null):2
14 2.8003 2813328 copy_file( ) ..(null):109
15 2.8003 2813536 create_folders_recursive( ) ..(null):58
16 2.8003 2813696 create_folders_recursive( ) ..(null):91
17 2.8003 2813848 create_folders_recursive( ) ..(null):91
18 2.8004 2813992 create_folders_recursive( ) ..(null):91
19 2.8004 2814096 create_folders_recursive( ) ..(null):91
20 2.8004 2814176 create_folders_recursive( ) ..(null):91
21 2.8004 2814256 create_folders_recursive( ) ..(null):91
22 2.8004 2814336 create_folders_recursive( ) ..(null):91
23 2.8004 2814416 create_folders_recursive( ) ..(null):91
24 2.8004 2814496 create_folders_recursive( ) ..(null):91
25 2.8004 2814576 create_folders_recursive( ) ..(null):91
26 2.8004 2814656 create_folders_recursive( ) ..(null):91
27 2.8004 2814736 create_folders_recursive( ) ..(null):91
28 2.8005 2814816 create_folders_recursive( ) ..(null):91
29 2.8005 2814896 create_folders_recursive( ) ..(null):91
30 2.8005 2814976 create_folders_recursive( ) ..(null):91
31 2.8005 2815056 create_folders_recursive( ) ..(null):91
32 2.8005 2815136 create_folders_recursive( ) ..(null):91
33 2.8005 2815216 create_folders_recursive( ) ..(null):91
34 2.8005 2815296 create_folders_recursive( ) ..(null):91
35 2.8005 2815376 create_folders_recursive( ) ..(null):91
36 2.8005 2815456 create_folders_recursive( ) ..(null):91
37 2.8005 2815536 create_folders_recursive( ) ..(null):91
38 2.8005 2815616 create_folders_recursive( ) ..(null):91
39 2.8006 2815704 create_folders_recursive( ) ..(null):91
40 2.8006 2815792 create_folders_recursive( ) ..(null):91
41 2.8006 2815880 create_folders_recursive( ) ..(null):91
42 2.8006 2815968 create_folders_recursive( ) ..(null):91
43 2.8006 2816056 create_folders_recursive( ) ..(null):91
44 2.8006 2816152 create_folders_recursive( ) ..(null):91
45 2.8006 2816248 create_folders_recursive( ) ..(null):91
46 2.8006 2816344 create_folders_recursive( ) ..(null):91
47 2.8006 2816440 create_folders_recursive( ) ..(null):91
48 2.8006 2816536 create_folders_recursive( ) ..(null):91
49 2.8007 2816632 create_folders_recursive( ) ..(null):91
50 2.8007 2816728 create_folders_recursive( ) ..(null):91
51 2.8007 2816824 create_folders_recursive( ) ..(null):91
52 2.8007 2816920 create_folders_recursive( ) ..(null):91
53 2.8007 2817016 create_folders_recursive( ) ..(null):91
54 2.8007 2817112 create_folders_recursive( ) ..(null):91
55 2.8007 2817208 create_folders_recursive( ) ..(null):91
56 2.8007 2817304 create_folders_recursive( ) ..(null):91
57 2.8007 2817400 create_folders_recursive( ) ..(null):91
58 2.8007 2817496 create_folders_recursive( ) ..(null):91
59 2.8007 2817592 create_folders_recursive( ) ..(null):91
60 2.8008 2817688 create_folders_recursive( ) ..(null):91
61 2.8008 2817784 create_folders_recursive( ) ..(null):91
62 2.8008 2817880 create_folders_recursive( ) ..(null):91
63 2.8010 2817976 create_folders_recursive( ) ..(null):91
64 2.8010 2818072 create_folders_recursive( ) ..(null):91
65 2.8010 2818168 create_folders_recursive( ) ..(null):91
66 2.8011 2818264 create_folders_recursive( ) ..(null):91
67 2.8011 2818360 create_folders_recursive( ) ..(null):91
68 2.8011 2818456 create_folders_recursive( ) ..(null):91
69 2.8011 2818552 create_folders_recursive( ) ..(null):91
70 2.8011 2818648 create_folders_recursive( ) ..(null):91
71 2.8011 2818744 create_folders_recursive( ) ..(null):91
72 2.8011 2818840 create_folders_recursive( ) ..(null):91
73 2.8011 2818936 create_folders_recursive( ) ..(null):91
74 2.8011 2819032 create_folders_recursive( ) ..(null):91
75 2.8011 2819128 create_folders_recursive( ) ..(null):91
76 2.8011 2819224 create_folders_recursive( ) ..(null):91
77 2.8012 2819320 create_folders_recursive( ) ..(null):91
78 2.8012 2819416 create_folders_recursive( ) ..(null):91
79 2.8012 2819512 create_folders_recursive( ) ..(null):91
80 2.8012 2819608 create_folders_recursive( ) ..(null):91
81 2.8012 2819704 create_folders_recursive( ) ..(null):91
82 2.8012 2819800 create_folders_recursive( ) ..(null):91
83 2.8012 2819896 create_folders_recursive( ) ..(null):91
84 2.8012 2819992 create_folders_recursive( ) ..(null):91
85 2.8012 2820088 create_folders_recursive( ) ..(null):91
86 2.8013 2820184 create_folders_recursive( ) ..(null):91
87 2.8013 2820280 create_folders_recursive( ) ..(null):91
88 2.8013 2820376 create_folders_recursive( ) ..(null):91
89 2.8013 2820472 create_folders_recursive( ) ..(null):91
90 2.8013 2820568 create_folders_recursive( ) ..(null):91
91 2.8013 2820664 create_folders_recursive( ) ..(null):91
92 2.8013 2820760 create_folders_recursive( ) ..(null):91
93 2.8013 2820856 create_folders_recursive( ) ..(null):91
94 2.8013 2820952 create_folders_recursive( ) ..(null):91
95 2.8013 2821048 create_folders_recursive( ) ..(null):91
96 2.8014 2821144 create_folders_recursive( ) ..(null):91
97 2.8014 2821240 create_folders_recursive( ) ..(null):91
98 2.8014 2821336 create_folders_recursive( ) ..(null):91
99 2.8014 2821432 create_folders_recursive( ) ..(null):91
Title: Re: OpenImporter
Post by: emanuele on November 04, 2014, 09:01:42 pm
Just to be sure (I saw it in the backup you sent me, so it should be fine), did you change the paths to the attachments to the local ones? (C:\wamp\www\...)

Could you also try putting the following code in a php file, execute it and report output?
Thank you very much!

Code: [Select]
<?php
error_reporting(E_ALL);

create_folders_recursive('C:\\wamp\\www\\skodaelk\\attachments');

function create_folders_recursive($path)
{
$parent = dirname($path);

if (!file_exists($parent))
{
echo 'Nesting in: ' . $parent . '<br />';
create_folders_recursive($parent);
}

if (!file_exists($path))
{
echo 'Creating: ' . $path . '<br />';
mkdir($path, 0755);
}
}
Title: Re: OpenImporter
Post by: radu81 on November 04, 2014, 09:20:39 pm
I did not change anything, the files are the same that I sent you.

where should I put the php file? in the root of my smf forum? Itried but I see a blank page
Title: Re: OpenImporter
Post by: emanuele on November 04, 2014, 10:10:55 pm
Never mind, phar archives and directory creation in windows are not friends.
I'll try to investigate the reason, but not now.

I attached here a "normal" package, try using this one.
And sorry. ;)

ETA: package removed.
Title: Re: OpenImporter
Post by: radu81 on November 04, 2014, 10:31:01 pm
thank you for your support but I'm still having this

Code: [Select]
mporting settings... ✔
Importing members... ✔
Importing attachments...

( ! ) Fatal error: Maximum function nesting level of '100' reached, aborting! in C:\wamp\www\importer\OpenImporter\Utils.php on line 88
Call Stack
# Time Memory Function Location
1 0.0007 728920 {main}( ) ..\import.php:0
2 1.1266 2292568 ImportManager->process( ) ..\import.php:69
3 1.1268 2293704 call_user_func ( ) ..\ImportManager.php:236
4 1.1268 2293704 ImportManager->doStep1( ) ..\ImportManager.php:236
5 1.2028 2460976 Importer->doStep1( ) ..\ImportManager.php:465
6 2.2083 2475040 XmlProcessor->processSteps( ) ..\Importer.php:516
7 2.2190 2476616 XmlProcessor->doSql( ) ..\XmlProcessor.php:99
8 2.2233 2546024 XmlProcessor->prepareRow( ) ..\XmlProcessor.php:157
9 2.2234 2545160 eval( ' moveAttachment($row, $this->db, $this->config->from_prefix, $this->step1_importer->getAttachDir($row)); ' ) ..\XmlProcessor.php:212
10 2.2239 2545160 moveAttachment( ) ..\XmlProcessor.php(212) : eval()'d code:2
11 2.2241 2548752 copy_file( ) ..\smf2-0_importer.php:109
12 2.2241 2548960 create_folders_recursive( ) ..\Utils.php:58
Title: Re: OpenImporter
Post by: emanuele on November 04, 2014, 11:29:52 pm
/me needs a new brain to understand php on windows... :-[

Sorry, I'll test it tomorrow if I can manage to wake up my brain... :-[
Title: Re: OpenImporter
Post by: radu81 on November 04, 2014, 11:42:07 pm
take your time, there is no hurry for me, it's just a test in localhost  (under wamp on windows  O:-) )
Title: Re: OpenImporter
Post by: emanuele on November 04, 2014, 11:54:05 pm
Maybe I got it: are you using the same installation of Elk that you used before in your tests?
If so, the attachments directory settings may be messed up and so when the importer tries to import the data fails. I realised just now while trying on an already messed up install.
Title: Re: OpenImporter
Post by: radu81 on November 06, 2014, 11:19:14 am
And here it is an experiment.

Instructions:
* unzip
* upload
* point to import.php

That's an experiment to understand if it is worth exploring the route of the phar archive.
A phar archive is a "special" php file that includes the entire application into a single file (actually it needs two: the phar and a php to run the phar, meh). I thought this may be easier to handle for the "final users" in that specific case, since it would be a "one shot" and have lots of files and directories to deal with could be a little pain, while two files are much easier to track in the process. Also in theory the released package doesn't require any change to the files, etc.

Anyway, I tried this solution, though reading around there are some configurations that do not like phar (e.g. suhosin) and if it doesn't work on the majority of systems, it may not be worth to maintain a double system of distribution (that would create confusion I think).

Let's see.
I tried this in a live site and everything was imported ;)
Title: Re: OpenImporter
Post by: emanuele on November 06, 2014, 11:46:48 am
Sounds like I have to check the data taken from the db as well to be sure are "valid".

It's difficult to draw the line where things can be trusted and where not, I guess the attachments directory is one of those that cannot be trusted 100%.
Title: Re: OpenImporter
Post by: radu81 on November 06, 2014, 12:05:03 pm
attachments, avatars and custom fields were imported. I'll check the rest later ;)
Title: Re: OpenImporter
Post by: Jorin on November 06, 2014, 03:04:04 pm
Sounds great!  :)
Title: Re: OpenImporter
Post by: emanuele on November 08, 2014, 12:40:35 am
Another tool for code quality/security:
https://insight.sensiolabs.com/projects/e35e9761-ef31-4f96-bf2c-687ef6cd01e8/analyses/1
Looks interesting. :D
Title: Re: OpenImporter
Post by: Spuds on November 08, 2014, 02:33:40 am
Ah very cool !

We do have sensiolabs_security_checker: true as part of  our scrutinizer.yaml file but I don't think that check is as complete as the running it there !

OK now get rid of all those globals :D
Title: Re: OpenImporter
Post by: emanuele on November 08, 2014, 10:13:09 am
I tried with Elk, but it was not able to finish... xD
Title: Re: OpenImporter
Post by: Jorin on November 19, 2014, 08:16:02 am
Is the importer nearly perfect now? ;)

I want to try to migrate soon but I am not sure which fixes I have to make to the importer. Or is the package up to date?
Title: Re: OpenImporter
Post by: emanuele on November 19, 2014, 03:22:56 pm
"Perfect" is a difficult word. lol

It should work for most of the things (including attachments).
Improvements are always possible. ;)
Title: Re: OpenImporter
Post by: Jorin on November 19, 2014, 03:27:35 pm
Difficult? No... Try it!

P - e - r - f - e - c - aaaaand finally - t !

So easy!  :D
Title: Re: OpenImporter
Post by: emanuele on November 19, 2014, 06:44:55 pm
ROFLMAO!
Title: Re: OpenImporter
Post by: emanuele on November 25, 2014, 04:15:54 pm
Back a moment on the "how" OpenImporter works.
I may already have proposed that, though better have it written all in one place.

At the moment the biggest limitation I see is that OI requires to write an importer for any platform you want to support the conversion to.
This makes it not *much* better than any importer/converter already existing for one of the many forums around the world.

A very cool thing would be to split the "export" process from the "import", that way, writing an importer would be enough to cover the exports to any platform already supported.

Let me explain a bit better. What I have in mind is nothing too fancy, basically is create a sort of "intermediate array" that will act as proxy between importers and exporters.
So, the process will be:
* importers grabs info from the source platform and converts to the "intermediate array"
* exporters takes in the data from the "intermediate array", converts to its own format and put them into the destination database/file system.
The advantage is that the "intermediate array" is common to any importer and exporter, so let's say we have an importer to Elk and Wedge, and exporters from SMF, vB4, XF and WP. If I tomorrow write an importer to phpBB, that will be automatically able to import data from SMF, vB4, XF and WP, without me having to write anything else.

This is of course the theory.

Honestly I have not yet tried anything like that, I'm still in the brainstorming phase, so it may even be impossible.
One potential problem I can see is if two sets of data are needed in order to build a third one, and do it in two phases is not an option. I have no examples of that at the moment, and I'm not sure how to build a "demo" of such a situation.
And it may become messy if a certain set includes data from two different tables.

Well, either way, the idea is here. Opinions, contributions, ideas, anything is welcome! ;D
Title: Re: OpenImporter
Post by: TE on November 26, 2014, 06:37:33 am
A very cool thing would be to split the "export" process from the "import", that way, writing an importer would be enough to cover the exports to any platform already supported.
yep, that would be very cool..

the ultimate question: what to do with this temporary data (export)? store it in "temp" tables in the database or maybe in the file system (e.g. XML files)? Especially for bigger boards that may be a PITA (much more data, slow import + export)
Title: Re: OpenImporter
Post by: emanuele on November 26, 2014, 06:53:12 pm
I was more thinking about an "on-the-fly" thing.
... little example: at the moment the xml file contains queries. These queries are make in such a way that data would fit into the destination database.
I would make the split here. The queries, instead of being formatted for the specific destination database format are designed for a "generic" data structure (another way to say an array with certain defined indexes). Then, the "destination", will have a function able to process this intermediate data and "convert" it into the form specific for that database.
Title: Re: OpenImporter
Post by: Jorin on December 07, 2014, 05:24:57 pm
What do I have to do to get the importer to work? I remember I had to copy import.php and two other files in the forum root folder. But it seems the importer has changed a lot. Can anyone please give me a hint?
Title: Re: OpenImporter
Post by: TE on December 08, 2014, 08:31:09 am
Same as before, just copy all files to your host  (import.php and the file structure located in the Importer-Folder). Then run import.php from your webbrowser.
Title: Re: OpenImporter
Post by: Jorin on December 08, 2014, 08:51:41 am
Import.php is in forum root of the ElkArte forum, importer directory is there too. Is this right? I get the following error if I open import.php:

Code: [Select]
Warning: require_once(/www/htdocs/ftpuser/elkarte/openimporter.phar) [function.require-once]: failed to open stream: No such file or directory in /www/htdocs/ftpuser/elkarte/import.php on line 4

Fatal error: require_once() [function.require]: Failed opening required '/www/htdocs/ftpuser/elkarte/openimporter.phar' (include_path='.:/usr/share/php:..') in /www/htdocs/ftpuser/elkarte/import.php on line 4

Ah, so the files in the importer folder must be copied into the root too?
Title: Re: OpenImporter
Post by: emanuele on December 08, 2014, 01:37:12 pm
O_o

Where did you download the package?
Title: Re: OpenImporter
Post by: Jorin on December 08, 2014, 01:38:24 pm
Here: http://www.elkarte.net/community/index.php?topic=2118.msg13972#msg13972 (http://www.elkarte.net/community/index.php?topic=2118.msg13972#msg13972)
Title: Re: OpenImporter
Post by: emanuele on December 08, 2014, 01:41:17 pm
Better remove it then... not at all sure what "version" it is.

@TE maybe we should start using some kind of versioning?
Title: Re: OpenImporter
Post by: Jorin on December 08, 2014, 01:43:11 pm
I thought this is the acutal version? If not, where can I download the package? Oh, not only think about this, write a short basic manual how to use it too.  :-X

Edit: Is this the place to download the latest version?

http://openimporter.github.io/openimporter/ (http://openimporter.github.io/openimporter/)
Title: Re: OpenImporter
Post by: emanuele on December 08, 2014, 01:53:10 pm
Yep, that should be a "good" latest version.

As per using it:
* download
* extract
* upload (technically anywhere accessible from the internet I'd suggest to create an "importer" directory in the root)
* point the browser to yourdomain.tld/importer/import.php (of course "importer" if you followed my idea, otherwise it depends where you exactly put the code)
* follow instructions.
Title: Re: OpenImporter
Post by: Jorin on December 08, 2014, 01:56:04 pm
Thanks, I will try this package later.
Title: Re: OpenImporter
Post by: TE on December 08, 2014, 02:32:22 pm
@TE maybe we should start using some kind of versioning?
yep, definitely.. plus releases on Github?
Title: Re: OpenImporter
Post by: Jorin on December 09, 2014, 07:45:37 am
Do the import needs a functional domain to the forum? At the moment I am using the subdomain http://forum.hybrid-piloten.de (http://forum.hybrid-piloten.de) for my SMF board. When I start the migration I want to link that subdomain to a small html site showing some information that the board isn't reachable for a while (http://migration.hybrid-piloten.de). Can I do this or will this lead to errors?
Title: Re: OpenImporter
Post by: emanuele on December 09, 2014, 11:47:29 am
The importer doesn't change any configuration.
You have:
* a working or not forum to import from (it can just be the configuration file (Settings.php for example) and the attachments if any, plus the database of course)
* a working forum to import to.
* a way to reach the file import.php
You set the URLs and paths when you install the forum, from that point on, the only thing that counts is that you car reach import.php from the web.

For example, you could set up:
* forum.yourdomain.tld with the forum you want to import from (e.g. SMF)
* forumnew.yourdomain.tld with the forum you want to import to (e.g. ElkArte)
* migrationtool.yourdomain.tld the domain from which you reach import.php
at that point you can even deny access to forum.yourdomain.tld and newforum.yourdomain.tld and you will be able to do the conversion pointing to migrationtool.yourdomain.tld/import.php

I hope what I wrote does make some sense. lol
Title: Re: OpenImporter
Post by: Jorin on December 09, 2014, 11:48:55 am
It does! Thank you!
Title: Re: OpenImporter
Post by: emanuele on December 16, 2014, 06:26:07 pm
I just posted a "technical" thought at github:
https://github.com/OpenImporter/openimporter/issues/38
Title: Re: OpenImporter
Post by: emanuele on December 28, 2014, 11:22:13 pm
Quick question:
https://scrutinizer-ci.com/g/emanuele45/openimporter/inspections/50596641-ced4-4df1-85ed-ff45d174d2e9/issues/files/importer/Importers/sources/phpbb3_importer.php?status=existing&orderField=path&order=asc
https://github.com/emanuele45/openimporter/blob/abstraction/importer/Importers/sources/phpbb3_importer.php#L48

Am I dumb of the method fixBbc is defined 16 lines above the point is used?
Title: Re: OpenImporter
Post by: Spuds on December 29, 2014, 12:12:24 am
Why is it protected ?
Title: Re: OpenImporter
Post by: emanuele on December 29, 2014, 12:38:38 am
Probably just because it's used only from the inside of the class.
I can try with a public and see if something changes.
Title: Re: OpenImporter
Post by: Spuds on December 29, 2014, 04:55:13 am
May help or you may have to tweak it to the parent  ... I always get confused by where the protect methods are available when you extend like that.
Title: Re: OpenImporter
Post by: emanuele on December 29, 2014, 09:50:23 am
Yeah, but in theory it's in the very same class, not inherited, not extended, etc.
Dunno, from time to time Scrutinizer reports odd things. LOL

Anyway, I'm about 30 blocks of xml away from converting all the "exporters", that means a couple of days away from lots of testing! Darn! xD
But also a couple of days away from the (first step of the) original goal! YAY!
In other words, when I have finished converting these blocks, OpenImporter should be able to provide a really "open" platform for any kind of conversion!
I expect (still) quite a lot of limitations because in fact it has been tested only with the SMF-family of scripts as "importers" and it may have some tricky part to iron out (see comments to issue 38), but still a good base for further development. :D
Title: Re: OpenImporter
Post by: emanuele on December 29, 2014, 11:37:25 pm
https://github.com/emanuele45/openimporter/tree/abstraction
Mostly done!! :D

I found that in this new setup three converters (WBB 3.0, WP and phpBB) are broken due to the strategy of using the destination database to query for certain info [1].
I hope there is a solution...
At the moment I need some sleep, so I can't figure out, but I'm pretty sure that using some additional query should be enough to get rid of the cross-queries.
More info here (https://github.com/emanuele45/openimporter/commit/3ad44641c9e28541d8357d7a9073ad19c9b51315).
Title: Re: OpenImporter
Post by: Spuds on December 30, 2014, 03:02:53 am
Oh my .. you have been busy !!

Its looking really nice, I'm just going through the SMF importer to understand how you have done the separation, good stuff, lots to understand yet :D  Not sure what to say about the issue you found, needs to be a solution but I don't yet understand everything you have done to even make a suggestion, not even one of my dumb ones  :P
Title: Re: OpenImporter
Post by: emanuele on December 30, 2014, 10:31:58 am
That gives me the opportunity to explain a bit. :P

The general idea is that:

How this is achieved technically?
Step 1 consists of 2 files:
  * an XML file that contains all the steps a certain "exporter" is able to understand, each step has a specific name[2], and at least a couple of childs: a title (string text that is displayed during the conversion), a detect (it's a table name, OI checks if the table exists or not, if not it sips the step) and a code or a query (one or the other, they cannot coexist). I'm considering moving away from the xml to a yaml or json file.
  * a php file with a class[3]. The class contains some generic methods to describe the class itself and can contain several other methods, sort of "hook" into the conversion process. These methods' names are in the format prefixStepname, where "prefix" is the "hooking point", for example can be preparse or code, while "Stepname" is the name of the step (as defined in the XML file), these methods are called during the process and allow to change the results of the queries and/or of the code blocks.

Step 2 consists of 1 file:
* a php file containing a class[4]. Similarly to the php file in step 1, it may contain methods with names that follow the prefixStepname schema and that will be called at each step. OI passes to these methods the result of step 1. The return may be an array of rows, or nothing.

Step 3 is really an implementation detail, the only thing to know is that it may expect two types of arguments coming from step 2:

I think this summarize what happens during the conversion... I think.
The main point is that all the system should be based on the assumption that the "exporter" doesn't know anything about the "importer" and the "importer" doesn't know anything about the "exporter"... I hope. :P
I'd be tempted to talk about exporter for the "source to intermediate" and importer for the "intermediate to destination", seems easier, but terminology consistency is not yet the main problem.
The known names are defined in the file Importer_skeleton.xml. This file is not yet used, but in the future it may be used to verify the data types and fill up the gaps, for example entries in each array that doesn't have a value.
The name of the class is specified in the generic section of the XML file and extends an abstract with some methods common to all the exporters
Actually, because I'm lazy it contains three of four classes all necessary for the "importer" during the process, but let's not be too picky. :P
Title: Re: OpenImporter
Post by: emanuele on January 01, 2015, 11:42:51 pm
I'm playing with tests now:
https://github.com/emanuele45/openimporter/tree/testing/importer/Importers/sources/Tests
though at the moment travis doesn't want to collaborate much.
Well, I'll call it a day and go sleep for the moment. :P
Title: Re: OpenImporter
Post by: emanuele on January 02, 2015, 06:04:41 pm
Interesting stuff: https://github.com/internetbrands/vbimpex
Title: Re: OpenImporter
Post by: emanuele on January 02, 2015, 09:22:16 pm
YAY!
https://travis-ci.org/emanuele45/openimporter/jobs/45716134
Title: Re: OpenImporter
Post by: TE on January 02, 2015, 09:35:16 pm
YAY!
https://travis-ci.org/emanuele45/openimporter/jobs/45716134
Great :)
Title: Re: OpenImporter
Post by: emanuele on January 02, 2015, 10:01:44 pm
Actually, now I have to understand if keep the two set of tests separated is acceptable.
Then, if it is, I have to find a way to merge the two coverage reports, while if it not acceptable, I have to find a way to merge them...

Maybe @Spuds has some idea about it. O:-)
Title: Re: OpenImporter
Post by: emanuele on January 03, 2015, 12:58:22 am
Regex question of the day.
I have added a dummy class to mimic a database.
The class takes a query string, parses the SELECT and returns an array with the appropriate indexes (sort of).
To do so I used the following code:
Code: [Select]
		// clean tabs
$string = trim($string);
// remove SELECT
$string = trim(substr($string, 6));
$from = strpos($string, 'FROM {');
$string = substr($string, 0, $from);
$chunks = array_map('trim', preg_split('~,(?! [^(]*\))~i', str_replace("\n", '', $string)));

$array = array();
foreach ($chunks as $chunk)
{
$res = explode(' as ', strtolower($chunk));
if (count($res) == 2)
$array[$res[1]] = '';
else
$array[$res[0]] = '';
}
The idea looked simple: strip anything that is not necessary, split on the commas and do something about the "as".
It works on things like:
Code: [Select]
 SELECT something, something2 FROM whatever
and:
Code: [Select]
 SELECT something, IFNULL(something2, 0) FROM whatever
But of course we have weirder things like:
Code: [Select]
 SELECT something, '1, 0' as something2 FROM whatever
and here it fails.
I could add the single quotes to the regex I already used, something like:
Code: [Select]
preg_split('~,(?! [^(\']*[\)\'])~i', 
though we have even more weird things like:
Code: [Select]
 SELECT usepostcounts != 'yes' AS count_posts, '-1,0' AS member_groups FROM whatever
and that regex would fail.

Is there any way to split a similar string simply based on a regexp?
Or do I have to write a parser?
Title: Re: OpenImporter
Post by: emanuele on January 03, 2015, 04:05:14 pm
Scratch the previous message, since I was bored, I wrote a basic "parser".
Title: Re: OpenImporter
Post by: Spuds on January 08, 2015, 12:11:11 am
Actually, now I have to understand if keep the two set of tests separated is acceptable.
Then, if it is, I have to find a way to merge the two coverage reports, while if it not acceptable, I have to find a way to merge them...

Maybe @Spuds has some idea about it. O:-)
Scruit will combine coverage sets, you just have to tell it to expect X number of coverage result files for a run.  https://scrutinizer-ci.com/blog/support-for-merging-code-coverage-data
Title: Re: OpenImporter
Post by: Allan on January 11, 2015, 05:30:36 pm
Any idea if this works with IPB 3.4.x or plans to add it very soon like today?   ;)
Title: Re: OpenImporter
Post by: emanuele on January 11, 2015, 05:53:41 pm
Nope, IPB is not among the alternatives at the moment and I don't even have a way to write such an importer... (i.e. I don't have an IBP 3.4.x database to test it.)
Of course, if you have one and you are willing to let me play with it I can try, but it's very unlikely it will become ready for today.
Title: Re: OpenImporter
Post by: Allan on January 11, 2015, 05:56:13 pm
Oh I wasn't expecting it today just a little humor. Would do you need I can hook you up.
Title: Re: OpenImporter
Post by: emanuele on January 11, 2015, 06:57:51 pm
lol

Yep, would be useful to have the importer available.
What is needed is of course the database (do not post it here, if you have any space to securely (i.e. no file sharing services, and a password protected zip) upload it, it's better, if not drop me a PM we can find a way), and is possible the configuration file (cleaned up of the database passwords and usenames at least, but do not remove the row, just put some xxxxxx where the password and usename is).
With that I should be able to have a look and write the necessary importer.

Of course I will not distribute the database nor part of it, and I will not read the content except few random bits to verify for example the proper importing of the messages in the different format. What else? Nothing I guess.
Title: Re: OpenImporter
Post by: Allan on January 11, 2015, 09:03:25 pm
I just sent you a pm.
Title: Re: OpenImporter
Post by: emanuele on January 12, 2015, 11:18:15 pm
FWIS, this evening I did the members conversion. There are three todo's pending:
Code: [Select]
			// @todo check avatars conversion
// @todo find a way to port titles (post-based groups)
// @todo warn_level what is the value?
The first will be "fixed" as soon as I try a run, the other two may took a bit more.
Title: Re: OpenImporter
Post by: Allan on January 12, 2015, 11:23:40 pm
@emanuele  thank you and please take your time there is no rush.
Title: Re: OpenImporter
Post by: meetdilip on January 12, 2015, 11:34:38 pm
@Allan , is this for alilnook ?
Title: Re: OpenImporter
Post by: Allan on January 13, 2015, 02:48:25 am
It could be  ;D, really turned off by IPB, I thought it would be better then it is. Thankfully I got it for free.
Title: Re: OpenImporter
Post by: emanuele on January 14, 2015, 11:17:38 pm
I decided to do one or two steps a day, at the moment the following are completed (at least theoretically):
* members
* attachments (no idea if it works unfortunately)
* I skipped avatars, maybe I'll do it later
* categories
* boards
* topics
* messages
pending: polls, PMs, several logs, member groups, permissions, custom fields, and likes (and I see the converter doesn't have (yet) a mentions/alerts/notifications/whatever section.

It's a good exercise this converter! :D
Title: Re: OpenImporter
Post by: emanuele on January 17, 2015, 10:11:51 pm
I decided to do one or two steps a day,
Unless of course I have two/three/five days of mess in RL... LOL
Title: Re: OpenImporter
Post by: Allan on January 17, 2015, 11:46:37 pm
That damn RL  gets in the way of everything  :P
Title: Re: OpenImporter
Post by: meetdilip on January 18, 2015, 06:16:37 am
:P
Title: Re: OpenImporter
Post by: emanuele on January 18, 2015, 10:03:52 am
lol!
Title: Re: OpenImporter
Post by: Bunstonious on January 21, 2015, 01:38:16 pm
Hi guys,

Is there any importer planned to be able to go from wedge to Elk? I did an Elk test and my forum members prefer Elk at the moment.
Title: Re: OpenImporter
Post by: emanuele on January 21, 2015, 04:44:20 pm
Hello @Bunstonious and welcome around. :)

At the moment... no, there is not one.
It shouldn't be too difficult to make one, but I have already another one in the pipeline (IPB 3.4) and I had a spike of RL activity, so I have to limit myself on the number of things I start. lol

Anyway, request tracked:
https://github.com/OpenImporter/openimporter/issues/49
Title: Re: OpenImporter
Post by: Bunstonious on January 21, 2015, 09:52:59 pm
Thanks mate  :D
Title: Re: OpenImporter
Post by: meetdilip on January 22, 2015, 02:04:51 pm
Welcome @Bunstonious
Title: Re: OpenImporter
Post by: Allan on February 15, 2015, 02:23:47 pm
@emanuele  wanted to see how the IPB was going.  ;)
Title: Re: OpenImporter
Post by: emanuele on February 15, 2015, 03:49:59 pm
I'll start working on it again this week. Unfortunately, on top of everything else, also a flu appeared and kept me in the bed for almost a week... :-\
Today is the first "free" day since a week and a half (and actually I'd have to: 1) fix a website I left broken since the day before I got the first symptoms of the flu (i.e. two weeks ago), 2) write a program for a collaboration I'm doing, 3) fix some stuff for an ecommerce website I'm co-managing, 4) work on Elk, 5) work on the importer).

So, for today only Stargate Atlantis, Dr. Who, Agents of S.H.I.E.L.D. and some random coding, nothing serious.
Title: Re: OpenImporter
Post by: Allan on February 15, 2015, 04:40:28 pm
Glad you are feeling better and those are some great shows  :D
Title: Re: OpenImporter
Post by: emanuele on February 22, 2015, 12:18:50 am
"Almost" done, I still have few items and then some testing...

/me is scared by testing! :o
Title: Re: OpenImporter
Post by: Allan on February 22, 2015, 04:11:14 am
Sounds great, thank you so much also  :D
Title: Re: OpenImporter
Post by: emanuele on February 22, 2015, 02:10:09 pm
For the record: I skipper reports and friendly URLs that are doable, but I need some time to work on them[1] and moved directly to custom fields and likes.
These are the only two things left to prepare for the conversion before the testing.
Custom fields will be somehow random because from the data I have I can only guess so much, but at least it should be a starting point.

ETA: custom fields are done, I'm trying to understand how likes work but seems a bit more difficult than expected... I'll have to take a break now, will try again probably tomorrow.
Maybe I'll post some comments as an issue at github to keep it tracked.
Title: Re: OpenImporter
Post by: emanuele on February 23, 2015, 09:52:26 pm
I gave up on the likes for the moment (I don't have enough data to understand how they are stored), I'll come back on them in the future if I'll have the opportunity.
I started some testing and now it mostly works... mostly.
There are few oddities in the conversion of the html of the messages into bbc.
Title: Re: OpenImporter
Post by: Allan on February 23, 2015, 11:20:47 pm
I am no t worried about the likes so that is a good thing, the forum is still young so the conversation aren't that big either.  Thank you again for all the help and working on this. :D

Just wanted to ask where the DL link was?
Title: Re: OpenImporter
Post by: emanuele on February 24, 2015, 10:22:48 am
At the moment there isn't a download link. O:-)
I'll post a package when I have it working. ;)
Title: Re: OpenImporter
Post by: emanuele on March 02, 2015, 04:58:25 pm
I'm still looking a bit into board access permissions.
Not an easy neither mission-critical (because it's something done once just after the conversion), but it would be nice to have it done at least nicely.

ETA: quick update, boards accessibility is *somehow* migrated as well (actually, that helped me again in understanding the limits of the system). Now I have just one little thing I'd like to try to move: avatars... unless I already tried and failed, I have to check back the topic. lol
Title: Re: OpenImporter
Post by: Allan on March 03, 2015, 03:18:37 am
Sounds great I am eagerly awaiting  :P
Title: Re: OpenImporter
Post by: emanuele on March 05, 2015, 06:34:45 pm
I had to make a choice and decided to post here the package even without avatars properly supported.

Title: Re: OpenImporter
Post by: emanuele on March 05, 2015, 09:20:31 pm
For the record @Bunstonious I started looking into Wedge as a source.
It should be "easy" because the base is SMF, but I'll have to check few things.
Title: Re: OpenImporter
Post by: Allan on March 05, 2015, 11:22:13 pm
I had to make a choice and decided to post here the package even without avatars properly supported.

  • Install Elk.
  • Extract, upload and point your browser to the "import.php" file.
  • Select Elk as destination, IPB as source, set the paths and try the conversion.
  • Once finished, log to Elk and go to admin > maintenance > routine and do both "find and repair errors" and "recount statistics". Of course check permissions and stuff like that (and also PMs, I based my conversion on the db structure, without looking at the actual content, it should work, but if you can verify it would be better).
Should get time tonight to get this done if not this weekend.  Thank you so much @emanuele
Title: Re: OpenImporter
Post by: Allan on March 06, 2015, 12:00:03 am
@emanuele  everything worked great except is is asking for a validation code to log in to the Elkarte forum. Is there any way to set a code in phpmyadmin?


Quote
An Error Has Occurred
Your email address needs to be validated before you can login. Need another activation email?
Title: Re: OpenImporter
Post by: emanuele on March 06, 2015, 12:09:10 am
*headscratch*
Okay, I got it, I'm an idiot[1]. xD

Yeah, try this:
Code: [Select]
UPDATE elkarte_members
SET is_activated = 1
Fix the table name if you used a different prefix.
Memo to self: I set a "global" is_disabled instead of checking if each single account is actually disabled or not. lol
Title: Re: OpenImporter
Post by: Allan on March 06, 2015, 12:10:47 am
Don't know much about phpmyadmin where do I run that as a query?
Title: Re: OpenImporter
Post by: Allan on March 06, 2015, 12:28:39 am
Never mind found it and everything is working  ;)
 
Title: Re: OpenImporter
Post by: Allan on March 08, 2015, 07:34:21 pm
Just wanted to stop in and say everything worked great on the conversion, thank you.
Title: Re: OpenImporter
Post by: emanuele on March 08, 2015, 08:20:42 pm
You are welcome! :D
And thanks for providing the means to write the importer. 8)
Title: Re: OpenImporter
Post by: meetdilip on March 09, 2015, 09:08:10 pm
Never mind found it and everything is working  ;)
 

Share it please.. :)
Title: Re: OpenImporter
Post by: emanuele on April 04, 2015, 06:11:44 pm
Another tool for code quality/security:
https://insight.sensiolabs.com/projects/e35e9761-ef31-4f96-bf2c-687ef6cd01e8/analyses/1
Looks interesting. :D
Got the bronze medal! :D
https://insight.sensiolabs.com/projects/e35e9761-ef31-4f96-bf2c-687ef6cd01e8/analyses/11
Title: Re: OpenImporter
Post by: Spuds on April 05, 2015, 04:08:40 am
Are you liking that service more than our other buddy?
Title: Re: OpenImporter
Post by: emanuele on April 05, 2015, 11:43:32 am
Not really.
It gives some good info, but I still prefer scrutinizer. And anyway, the Elk analysis is running since 17 hours ago and I have no idea when it will finish! LOL
Title: Re: OpenImporter
Post by: emanuele on April 06, 2015, 04:50:03 pm
For the record: still running. xD
Title: Re: OpenImporter
Post by: emanuele on April 07, 2015, 03:39:25 pm
A new source is waiting integration: SEO-Boards 1.1.
An old (no more supported) and simple forum platform:
https://github.com/OpenImporter/openimporter/pull/64
Title: Re: OpenImporter
Post by: radu81 on April 07, 2015, 04:30:31 pm
just curiosity, the importer from XF works? if yes from which version of xf?
Title: Re: OpenImporter
Post by: TE on April 07, 2015, 04:40:10 pm
just curiosity, the importer from XF works? if yes from which version of xf?
IIRC 1.0 which is outdated these days..
Title: Re: OpenImporter
Post by: emanuele on April 07, 2015, 04:48:05 pm
According to the name it should be 1.1, but either way it is rather outdated, yes. :(
Title: Re: OpenImporter
Post by: radu81 on April 07, 2015, 09:27:21 pm
Just curious, but who knows? :D
Title: Re: OpenImporter
Post by: emanuele on April 07, 2015, 09:53:02 pm
Well, if we can put our dirty hands on a database the converter will be updated! O:-)
Title: Re: OpenImporter
Post by: radu81 on April 08, 2015, 12:04:48 am
For now I don't have plans to migrate, but I can provide all you need to do the tests ;)
Title: Re: OpenImporter
Post by: emanuele on April 11, 2015, 07:27:09 pm
Thanks for the offer @radu81 ! :D

In the meantime: https://insight.sensiolabs.com/projects/e35e9761-ef31-4f96-bf2c-687ef6cd01e8/analyses/38
silver medal! :D
Spoiler (click to show/hide)
Title: Re: OpenImporter
Post by: emanuele on May 10, 2015, 02:15:58 pm
Hi guys,

Is there any importer planned to be able to go from wedge to Elk? I did an Elk test and my forum members prefer Elk at the moment.
FYI: I attached to the first message of this topic a development version of OpenImporter that should support Wedge to Elk as well.
It has only been quickly tested, so it may not import everything properly. I'm pretty sure for example avatars are still broken.
Title: Re: OpenImporter
Post by: ahrasis on May 10, 2015, 02:24:25 pm
This is good news already though a complete import would be better.
Title: Re: OpenImporter
Post by: emanuele on May 10, 2015, 02:31:44 pm
Well, it's an open source project. :P
Title: Re: OpenImporter
Post by: ahrasis on May 12, 2015, 06:59:53 am
Except I have not tried Wedge... :P
Title: Re: OpenImporter
Post by: emanuele on May 14, 2015, 07:07:26 pm
I'm having a bit of fun. :P
https://github.com/emanuele45/openimporter/compare/drop_code...twig
Title: Re: OpenImporter
Post by: Spuds on May 15, 2015, 03:10:59 am
TWIG !!!! 

/me runs to hide as I know nothing about it
Title: Re: OpenImporter
Post by: emanuele on May 15, 2015, 08:46:46 am
LOL

Neither did I, that's why I decided to play with it. :P
Title: Re: OpenImporter
Post by: emanuele on May 16, 2015, 07:19:39 pm
https://github.com/emanuele45/openimporter/compare/moar...emanuele45:phar2

Instead of tons of files, a phar archive, but instead of two files (a phar and a php that runs the phar, only a php that writes the phar and runs it. 8)
Rather cool except for the usual problems coming from permissions, but... who cares! :P
Title: Re: OpenImporter
Post by: radu81 on May 16, 2015, 09:38:13 pm
I just remember that the phar file did not worked in my case so I used the normal importer. I hope this is better for you developers to maintain the importer.

Keep up the good work guys ;)  and greetings from adriatic sea ;)
Title: Re: OpenImporter
Post by: Spuds on May 16, 2015, 10:36:11 pm
phar-out baby :P
Title: Re: OpenImporter
Post by: emanuele on May 17, 2015, 12:34:31 am
Yeah, I remember your problem with phar, but I've never been sure it was a problem with the "phar" itself or a but somewhere. I just blamed phar because it seemed the thing.

Phar are (in theory) more for "end-user" convenience, with a phar archive you have to upload one or two files, instead of... something around 500 of the "normal" application.
If the "single file" version I built demonstrates to be a viable solition, it is 1 file of 520 KiB, instead the original "expanded" directory are 496 files and 2.8 MiB (both after removing all the tests and part of the readme/licences files).
From a "dev" point of view, anyway, the phar looks **much** cleaner than a bunch of files in a directory. 8)
Title: Re: OpenImporter
Post by: radu81 on May 19, 2015, 11:57:09 pm
I downloaded the importer from http://openimporter.github.io/openimporter/
I used the same user for both database, but when I start the importer I get this:
I controlled in cpanel and that user has all the privileges for both databases

Code: [Select]
Sorry, the database connection information used in the specified installation of your forum cannot access the installation of SMF 2.0. This may either mean that the installation doesn't exist, or that the MySQL account used does not have permissions to access it.

The error MySQL gave was: SELECT command denied to user 'XX_xx'@'localhost' for table 'smf_members''
error_line 353
error_file /home/....domain.it/importer/OpenImporter/Importer.php

p.s. I also tried installing elkarte in the same database, but same result
Am I missing something? Thank you in advance
Title: Re: OpenImporter
Post by: emanuele on May 20, 2015, 09:35:50 am
Could you try the one attached to the first message of this topic?
Title: Re: OpenImporter
Post by: radu81 on May 20, 2015, 10:51:19 am
I tried, the importer starts but I am stuck on this:

Code: [Select]
Unsuccessful!
This query:
An exception occurred while executing '
SHOW COLUMNS
FROM `xxx_sharkracingclub`.smf_likes;':

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'xxx_sharkracingclub.smf_likes' doesn't exist;
Caused the error:
Table 'xxx_sharkracingclub.smf_likes' doesn't exist

Line: 151
File: /home/.../importer/OpenImporter/Database.php

I set the importer to import only from thank-o-matic mod and not the other Likes mods

***edit***
I tried without importing from thank-omatic and I was able to complete the migration
Title: Re: OpenImporter
Post by: emanuele on May 20, 2015, 11:19:35 am
Ouch!
There is something broken in the logic of the likes conversion.
I'll have a look this afternoon. ;)

Ohhh yes, I see. I'm an idiot.
I'll fix it this afternoon. :(
Title: Re: OpenImporter
Post by: emanuele on May 20, 2015, 01:20:01 pm
I have some problems testing it right now (i.e. I'm in the middle of some work and can't stop to setup everything needed... :-[), would you mind replacing the file Importers/sources/SMF2_0_Importer.php with the one attached?
It should fix the importing from the Thank you mod.

The problem was that, while redesign the importer, I overlooked the fact some code was executed even if it was not supposed to be. :-[
Title: Re: OpenImporter
Post by: radu81 on May 20, 2015, 02:16:54 pm
Thank you Emanuele. That worked, the "thank-you" data was corrected imported even if I got a notice
Notice: Undefined variable: table in /home/../Importers/sources/SMF2_0_Importer.php on line 130
Importing likes from Thank you mod... ✔

Notice: Undefined variable: table in /home/.../Importers/sources/SMF2_0_Importer.php on line 130
Recalculating statistics...

There is another issue: the category boards were not imported, I have all my boards into one category named "General Category"

p.s. there is no hurry for me, I am only doing some tests ;)
Title: Re: OpenImporter
Post by: Ohhnow on June 18, 2015, 03:56:24 pm
Hi there,

I`m trying to migrate my forum from mybb 1.8.5 to ElkArte using the open importer

when I try to do so I get the following fatal error:

Fatal error: Call to a member function query() on null in /xxx/importer/OpenImporter/Importer.php on line 475

please help ;)
Title: Re: OpenImporter
Post by: emanuele on June 18, 2015, 04:24:49 pm
Hi and welcome to Elk. :)

Ouch, that looks like a bug... will check.
In the meantime, could you try the zip file attached to the first message of this topic?
Title: Re: OpenImporter
Post by: Ohhnow on June 19, 2015, 08:18:53 am
thanks,

sure I will try.
Title: Re: OpenImporter
Post by: Ohhnow on June 19, 2015, 08:39:04 am
ok, the zip file works for me (some DB issues).

Thanks for your help.
Title: Re: OpenImporter
Post by: emanuele on June 19, 2015, 08:49:43 pm
I guess it's mostly due to the fact the converter is still for MyBB 1.6.
Unfortunately, at the moment I have a limited amount of time to take up the preparation of the 1.8 converter... :-[
Title: Re: OpenImporter
Post by: akmfcf on June 20, 2015, 02:40:57 pm
Hi and thank you for your great work! Is there any working version for phpBB 3.0.x -> ElkArte?
The version from post 1 doesn't offer phpBB3 as source, even if the PhpBB3_Importer.php|xml files exists in the sources directory. Is this importer still broken? All the other versions I could find didn't work for me.

Btw, is the importer able to handle big databases (Timeouts?)? Mine has ~800 MB.
Title: Re: OpenImporter
Post by: emanuele on June 20, 2015, 07:25:02 pm
Hi akmfcf and welcome to ElkArte. :)

Yeah, it's "broken" for a technical cavil on the member groups conversion. It shouldn't be too difficult to fix it, I'm currently fighting against the time to transfer my website from a host to another, as soon as I have finished I'll try to have a look.

About the timeouts: they should be handled properly. Unfortunately I worked mostly on the design up to now and I didn't pay too much attention to the problems associated to big databases.
Title: Re: OpenImporter
Post by: akmfcf on June 20, 2015, 07:45:52 pm
We need more caffeine for Emanuele :D
Good luck with the transfer.
Title: Re: OpenImporter
Post by: emanuele on June 20, 2015, 09:17:34 pm
I'd just need a faster connection! xD
Uploading 2 GB with this (http://www.speedtest.net/my-result/4408474876) is **no** fun at all.
Title: Re: OpenImporter
Post by: Burke Knight on June 20, 2015, 10:11:11 pm
How's this?

(https://www.elkarte.net/community/imagecache.php?image=http%3A%2F%2Fwww.speedtest.net%2Fresult%2F4447464597.png&hash=acf52cf093de5cb7ddfd30a7453e9f0b" rel="cached" data-warn="External image, click here to view original" data-url="http://www.speedtest.net/result/4447464597.png) (http://www.speedtest.net/my-result/4447464597)
Title: Re: OpenImporter
Post by: radu81 on June 20, 2015, 10:12:44 pm
I know something emanuele :P welcome to Italy
Title: Re: OpenImporter
Post by: emanuele on June 20, 2015, 10:48:44 pm
/me is tempted to ban @Burke Knight xD
Title: Re: OpenImporter
Post by: akmfcf on June 20, 2015, 11:14:05 pm
Nice  :P  Is there no way to do a direct transfer from server to server? With this speed you will see the sunrise tomorrow...
Title: Re: OpenImporter
Post by: Burke Knight on June 20, 2015, 11:43:04 pm
/me is tempted to ban @Burke Knight xD


ROFL!!!!!!
Title: Re: OpenImporter
Post by: emanuele on June 21, 2015, 12:57:57 am
Is there no way to do a direct transfer from server to server?
I tried, but the VPS timed out connecting to the host via ssh, so direct transfer is a no-go.

With this speed you will see the sunrise tomorrow...
I just accepted to have an incomplete transfer: I moved the most important stuff and kept some images and galleries locally waiting for tomorrow or the day after when I'll have time. :-\
Title: Re: OpenImporter
Post by: akmfcf on June 21, 2015, 09:10:45 am
Hm, a bit off topic, but are you able to connect via SSH to the new system and from there via FTP to the old one? If it's not necessary to encrypt the transfer then you can try this one-liner:
Code: [Select]
lftp -c mirror --parallel=5 ftp://user:password@oldsystem.tld/ targetpath/

If it's a no-go to transfer the files unencrypted and you have SSH-access from your home machine to the old server, you could create an Archive, encrypt it and download it by normal ftp to the new server. I hope one of the methods help you somehow :)
Title: Re: OpenImporter
Post by: emanuele on June 25, 2015, 01:24:20 pm
Sorry, but I got some more stuff to do in real-life and didn't have time to look into this until after Saturday... :(
Title: Re: OpenImporter
Post by: Jorin on June 30, 2015, 08:14:22 am
Nooo! Really? You have a real life?  :o  ;)
Title: Re: OpenImporter
Post by: emanuele on June 30, 2015, 08:16:39 am
And it's biting hard in this period! xD
Title: Re: OpenImporter
Post by: emanuele on July 06, 2015, 12:04:02 am
Memo for the dev version: there is a bug that makes it impossible to convert "big" forums (i.e. those that require breaks). It doesn't start the query at the correct "substep", in other words the query doesn't respect the starting point of the previous one causing duplicate keys errors.
Title: Re: OpenImporter
Post by: emanuele on July 10, 2015, 12:33:55 am
I've almost fixed the above issue, there is a small technical problem, that is actually the result of some other inconsistencies that should be properly fixed, so... some more work to do. lol
Title: Re: OpenImporter
Post by: radu81 on August 23, 2015, 05:18:44 pm
There is another issue: the category boards were not imported, I have all my boards into one category named "General Category"
is there any new version of open importer? I tried the version from page 12 of this topic but category boards were not imported

edit
I just tried this https://github.com/OpenImporter/openimporter
and seems that the went fine ;)
Title: Re: OpenImporter
Post by: emanuele on August 23, 2015, 06:27:37 pm
Did you try the one in the first page?
Title: Re: OpenImporter
Post by: radu81 on August 23, 2015, 06:49:58 pm
Yes but I've got an error during conversion. Seems that the one I linked worked fine. I need to do more tests to confirm that
Title: Re: OpenImporter
Post by: akmfcf on September 19, 2015, 11:12:10 pm
Hm, are there any news regarding phpBB 3.0.x?
(Post 1: no phpBB, GitHub development-version: blank page)
Title: Re: OpenImporter
Post by: emanuele on September 20, 2015, 01:13:06 am
I have been fixing the code design, and lost sight of any other issue... O:-)

The "master" version should be working reasonably fine with phpBB 3 as far as I remember. ;D
Title: Re: OpenImporter
Post by: akmfcf on September 20, 2015, 02:01:18 am
Unfortunately not, otherwise I would already use ElkArte ;)

master (from both GitHub-Repos):
Code: [Select]
error_message Array to string conversion
error_trace Array to string conversion
error_line 305
error_file /var/www/virtual/xxxxx/openimporter-master/importer/OpenImporter/ImportManager.php

development:
empty page

File from first post:
No PhpBB available to choose

Server system: PHP 5.6, Apache 2.2.15, CentOS 6.7

Where do you hide the working version? ;D
Title: Re: OpenImporter
Post by: wintstar on September 20, 2015, 07:01:15 am
Already reported as Issues => error_message Array to string conversion #89   (https://github.com/OpenImporter/openimporter/issues/89)

and still more:
Error import SMF 2.0*  (https://github.com/OpenImporter/openimporter/issues/91)
Platform phpBB missing (https://github.com/OpenImporter/openimporter/issues/90)

Title: Re: OpenImporter
Post by: allxsan on September 20, 2015, 03:41:02 pm
Hello, I'm new here, 
after trying in vain hundreds of times to import/convert to ElkArte from SMF 2.0.10 ( 11 now ) , I have to surrender . The conversion process is not 'never went more than a minute with this error ( only change the kb )


(https://www.elkarte.net/community/imagecache.php?image=http%3A%2F%2Fs19.postimg.org%2Fpib3d5l9f%2FOpenimporter.png&hash=3e761c7aa400aa99a3c8e243654454d6" rel="cached" data-warn="External image, click here to view original" data-url="http://s19.postimg.org/pib3d5l9f/Openimporter.png) (http://postimg.org/image/fl02k3dnj/full/)
upload (http://postimage.org/index.php?lang=italian)

I've tried with this version :    http://openimporter.github.io/openimporter/
and with the one  (slightly newer) on GitHub


The one just downloaded 10 minutes ago from the 1st page worked about 2 minute, when the bar was almost at 70/80% :
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 65488 bytes) in D:\inetpub\webs\forumriascit\elkarte\OpenImporter\OpenImporter\Files.php on line 79

(the ftp client can "see"  the SMF root as is in \forum.riasc.it\public"  and ElkArte root in "\forum.riasc.it\elkarte\" - the 1st level domain is under Linux hosting, the 3rd level is "forum.riasc.it" and hosted under windows )
Thanks

News:
today I've removed 3 times the entire path "elkarte" , removed 3 times the database elkarte_ and reinstalled ..
For some unknown reason the lastest time the conversion/import worked :-D
but...seem that something gone wrong with banned/locked out users. In the original database I have a list of more than 400 users, in the imported Elkarte database the list is empty
Title: Re: OpenImporter
Post by: emanuele on September 20, 2015, 08:37:59 pm
The first problem is apparently related to directory creation: at that point in the code it tries to move attachments from the old path to the new one, it looks like it's going too deep in the recursion. <= Means bug. And is a bug due to the windows server.
Tracked: https://github.com/OpenImporter/openimporter/issues/92

The second, you mean you have 400 bans in the list that disappeared after the import? Or that you have 400 members and after the import you don't have any?
Title: Re: OpenImporter
Post by: allxsan on September 22, 2015, 04:43:56 am
The first problem is apparently related to directory creation: at that point in the code it tries to move attachments from the old path to the new one, it looks like it's going too deep in the recursion. <= Means bug. And is a bug due to the windows server.
Tracked: https://github.com/OpenImporter/openimporter/issues/92

The second, you mean you have 400 bans in the list that disappeared after the import? Or that you have 400 members and after the import you don't have any?

Hello, I'm sorry for the delay, right now a new attempt is running. This time I've tried to simply overwrite the previous imported database from an updated (today posts) SMF database. Before this I've used "repair_settings.php" fom SMF and checked\fixed\optimized the smf_ database. Still running and I can see from the SQL webpanel that the elkarte_ new database is under construction
Title: Re: OpenImporter
Post by: allxsan on September 23, 2015, 10:39:24 am
I'm very sorry to say that for me doesn't work at all.
E.g.  the Categories "container" for the  boards are not imported. So, after the conversion I have all mixed in the main index. As example, I use some main categories  "YELLOW"  contains two boards "GOOD"  "BAD", "GREEN" contains "GOOD "BAD"...but after the conversion I only see the boards: "GOOD" "BAD" "GOOD" "BAD" "GOOD" "BAD" :-D 

About the users, the conversion script seem unable to detect banned users, custom groups (the usual, based on number of posts..)

Thanks
Title: Re: OpenImporter
Post by: emanuele on October 03, 2015, 08:51:10 pm
I'm really sorry a week has passed, unfortunately I had a quite busy round of days, and I wasn't able to touch my development environment for days... :(

I'll not be able to do much this weekend and the next.
Fortunately, starting from the 18th I'll be again ale to write bugs! :D
Title: Re: OpenImporter
Post by: badmonkey on October 04, 2015, 06:35:59 am
Would there be any chance of get an SMF 2.1 convertor? 
Title: Re: OpenImporter
Post by: emanuele on October 18, 2015, 05:13:07 pm
First I'd like to properly fix the converter, after that it would be relatively easier to write new ones... but find time is a little bit tricky at the moment.
Starting this week, though, I should have again more time. ;)
Title: Re: OpenImporter
Post by: bsi on March 02, 2017, 10:42:20 am
is there any chance or option to choose import from database backup instead of installed path?
Title: Re: OpenImporter
Post by: emanuele on March 02, 2017, 09:39:32 pm
The installed path is just to get the config file of the source destination.
SO you can just setup a fake "source path" with the configuration file of the platform and be done with it. :)