Converting from IPB 3.4

Converting from other board software? Good decision! Need help? Have a question about a convertor? Wish to offer a convertor package? Post here.
Post Reply
User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Converting from IPB 3.4

Post by Ger » Mon Jan 09, 2017 12:47 pm

I'm currently preparing to convert a large (± 1 million posts, ±55.000 users) from IPB 3.4 to phpBB 3.2. I've done some research, and I use the convertor from Prototech to convert from IPB 3.4 to phpBB 3.0.12. For the main part it works pretty fine after I have done some tweaking here and there. Afterwards I update to 3.1.10 and than to 3.2.latest (whichever version that will be when we're ready)

However, there is an issue with text parsing I'm trying to sort out. For some reason, many BBcodes don't seem to get parsed properly when going from IPB to phpBB 3.0.12.

It seems to me the problem resides in messages that are being changed with the function phpbb_prepare_message($message), residing in functions_ipb34.php. BBcodes created that way display unparsed. Now I thought to be the clever kid and use the reparse BBcode function from the Support Toolkit, but than I run into the Emoji SQL error.

Perhaps there is a way to either correct the BBcode parsing during the conversion or solve the emoji issue for 3.0.12? Or simply a good regex to remove the unsupported characters?

PS: I know 3.0.12 isn't supported anymore, but for me it's a just a step between IPB and phpBB since there's only a convertor available for 3.0.12.
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

User avatar
A_Jelly_Doughnut
Former Team Member
Posts: 34448
Joined: Sat Jan 18, 2003 1:26 am
Location: Where the Rivers Run
Contact:

Re: Converting from IPB 3.4

Post by A_Jelly_Doughnut » Mon Jan 09, 2017 10:03 pm

BBCodes are automatically reparsed when upgrading to 3.2, so maybe it won't be a problem in the end.

I'm not sure if the BBCode re-parsing for the 3.2 updater requires that the BBCodes have been properly parsed in the first place, or if it treats the post as raw text.

Perhaps you could try a dummy IPB install with a handful of posts to test this.
A Donut's Blog
"Bach's Prelude (Cello Suite No. 1) is driving Indiana country roads in Autumn" - Ann Kish

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Tue Jan 10, 2017 9:01 am

A_Jelly_Doughnut wrote:
Mon Jan 09, 2017 10:03 pm
BBCodes are automatically reparsed when upgrading to 3.2, so maybe it won't be a problem in the end.

I'm not sure if the BBCode re-parsing for the 3.2 updater requires that the BBCodes have been properly parsed in the first place, or if it treats the post as raw text.
As I understand it, reparsing is done in small batches through a cron and not during the upgrade. With literary a million posts, that would take quite long. I took the reparser_resume info from the config_text table (unserialized):

Code: Select all

array (
  'text_reparser.contact_admin_info' => 
  array (
    'range-min' => 1,
    'range-max' => 0,
    'range-size' => 100,
  ),
  'text_reparser.forum_description' => 
  array (
    'range-min' => 1,
    'range-max' => 0,
    'range-size' => 100,
  ),
  'text_reparser.forum_rules' => 
  array (
    'range-min' => 1,
    'range-max' => 0,
    'range-size' => 100,
  ),
  'text_reparser.group_description' => 
  array (
    'range-min' => 1,
    'range-max' => 0,
    'range-size' => 100,
  ),
  'text_reparser.poll_title' => 
  array (
    'range-min' => 1,
    'range-max' => 200361,
    'range-size' => 100,
  ),
  'text_reparser.user_signature' => 
  array (
    'range-min' => 1,
    'range-max' => 76154,
    'range-size' => 100,
  ),
  'text_reparser.post_text' => 
  array (
    'range-min' => 1,
    'range-max' => 1073736,
    'range-size' => 100,
  ),
  'text_reparser.pm_text' => 
  array (
    'range-min' => 1,
    'range-max' => 57194,
    'range-size' => 100,
  ),
)
As I understand, each call to cron.php does 100 reparsing actions, and the cron_interval is set to 10. That would mean about 10000 steps. with 10 seconds between each step at minimum (since the cron depends on visitors). That would take weeks. Isn't there a way to speed this up, of maybe even force the reparsing without use of the cron, so I can run it after the conversion process? I would rather take the forum a day completely offline to get things straight than having an ugly forum for weeks (and no doubt dozens of users pointing out unreparsed posts).
Perhaps you could try a dummy IPB install with a handful of posts to test this.
Well, I've done the process several times now (using different approaches) on my local server. Trust me, posts are ugly. :)
I had the hope that the reparser would take care of this, but unfortunately not right now.
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

User avatar
A_Jelly_Doughnut
Former Team Member
Posts: 34448
Joined: Sat Jan 18, 2003 1:26 am
Location: Where the Rivers Run
Contact:

Re: Converting from IPB 3.4

Post by A_Jelly_Doughnut » Tue Jan 10, 2017 12:58 pm

The reparser can be run in a large batch through the CLI interface. But I see you've discovered that and run into an error: viewtopic.php?f=556&t=2401026

I'll hop over to that thread...
A Donut's Blog
"Bach's Prelude (Cello Suite No. 1) is driving Indiana country roads in Autumn" - Ann Kish

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Tue Jan 10, 2017 2:55 pm

Yup, thought it didn't was really an convertor-issue, so I'd keep thing separated.
Reparsing through CLI only took 45 minutes for a million posts. Seemed to have the expected result, so I'm happy.
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Fri Feb 17, 2017 3:46 pm

Hello,

Still busy prepping this conversion. :)

I ran into some hardware limitations with the process to update from 3.0.12 to 3.1. Main issue is the huge posts table; several migrations (like softdelete_p1 as well as some others) alters this table. The sheer size of this table after conversion from IPB to phpBB 3.0.12 messes up the server. I tried it on 2 local instances (MAMP as well as WAMP) but it doesn't work. I did one testrun on a hosted server and it works, but the serverload skyrockets.

So, I took another approach. I changed the schema for the first 3.0.12 install and altered the convertor by Prototech (see first post) as well as the functions_convert and functions_admin that relate to this table. Basically I prepared the posts table to match the state in 3.2.0. All migration files are altered to match. Everything checks out fine, except 1 thing: the topic/post/reply count for each forum gets messed up.
Now I can perfectly go to the ACP and hit the sync button for each forum, but with about 100 forums this is quite an annoying task. Especially with the larger forums where I have to wait 10 - 15 minutes before I can continue. I'd rather hit 1 button for all forums to synchronise.

TL;DR: how can I get phpBB to synch all forums in 1 go? Can this be achieved by the CLI, can I write a simple script that loops through each forum, some other solution?
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Mon Feb 20, 2017 8:34 am

Bump :)
Ger wrote:
Fri Feb 17, 2017 3:46 pm
TL;DR: how can I get phpBB to synch all forums in 1 go? Can this be achieved by the CLI, can I write a simple script that loops through each forum, some other solution?
== EDIT ==
Well, I've managed to create a rudimentary extension for this. It basically mimics what the sync action does in the ACP, but it loops through all forums and automatically redirects to the next forum_id when done.
It's busy right now, but it seems to work. Since it's a very ugly extension in both presentation and code (and security) I'm not planning on releasing that extension right now. When someone in the future reads this topic and wants the same, I'll think about sharing the code. As it is now, it's simply unsafe to use it anywhere other than on a local server unreachable for the world.
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Thu Aug 10, 2017 7:38 am

I've got a request for sharing my code. Since I'm not allowed to attach .zip files here, I've uploaded them to a filesharing service: https://files.fm/u/d67tsrem

Note that this is a temporary link; the download expires on Oct 09, 2017.

Converting from IPB 3.4 to phpBB 3.2 proved quite a challenge, mainly because of the sheer size of the board, along with some dedicated stuff that was implemented. I took these steps:
  1. Backup current state IPB (files + db)
  2. Install phpBB 3.0.12.custom
  3. Run convertor for IPB to phpBB 3.0.12
    • In ./install/install_convert.php you find the vars $batch_size & $num_rows. Play with these a bit. I noticed that I can ramp them up to about 20000 resp. 600 for the most part, but I need to lower them to the default 6000 resp. 60 when the convertor approaches post_skip_rows ± 700000.
      This depends on your board and server, so make sure you run some tests to prevent timeouts on your board.
  4. Passwords will be reset since I don't want the unsafe triple MD5() that IPB uses to keep working. Therefore, when the conversion to phpBB 3.0.12 has completed, you should immediately request a new password for your own admin account. In the 3.0.12.custom files I've changed the protocol to show your new password on screen and be activated immediately.
    I shouldn't have to write this, but anyway: make sure you are the only one who can do this. Make your installation unreachable for the general public.
  5. Backup current state
  6. Fast update to phpBB 3.1.10.custom:
    1. Remove all files from the phpBB dir except config.php and the folders files, images, store.
    2. Copy the 3.1.10 files to the phpBB dir except above items
    3. Goto ./install/database_update.php and let it complete
    4. Backup current state
  7. Fast update to phpBB 3.2.0.custom:
    1. Remove all files from the phpBB dir except config.php and the folders files, images, store.
    2. Copy the 3.2.x files to the phpBB dir except above items
    3. I've found that the CLI update is easiest: php ./bin/phpbbcli.php db:migrate --safe-mode --> Seems to hang some time half way, but let it go; this is the alteration to the topics table, which is huge.
    4. Remove the install dir
  8. Run the text reparser (optional): php bin/phpbbcli.php reparser:reparse --ansi (this can take quite some time on large boards, but is also the reason to do it manually instead of the usual cron method)
  9. Install, run and remove the extension ger/syncforums (incorporated in the download). This is needed I kinda hacked the convertor to speed up stuff. It recalculates post/topic counters etc. It's not the most proper extension, so make sure to remove it after use. I only intended this for personal use in a protected environment.
  10. Set permissions, groups, etc. to your liking
  11. Configure spambot prevention
  12. Check all other settings
The whole process takes me about 4 hours from start to finish for a board with ± 1 million posts and 50.000 users. I have only done test runs so far but it seems things are working like expected.

https://files.fm/u/d67tsrem
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

jack208
Registered User
Posts: 7
Joined: Sat Sep 02, 2017 4:32 pm

Re: Converting from IPB 3.4

Post by jack208 » Sat Sep 02, 2017 5:04 pm

Am trying to do the same (ie convert from IPB 3.4 using Prototech converter) into phpBB ver 3.0.12 (before upgrading to latest 3.2) and I'm stuck at the Board Path input.

I do not have the Invision Board files - do we need them to convert? I thought we only need the IPB3 database (which I've). IF so, what input do we key into the Board Path?

Thanks

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Sat Sep 02, 2017 6:10 pm

You need a path to the attachments avatars, etc.
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

jack208
Registered User
Posts: 7
Joined: Sat Sep 02, 2017 4:32 pm

Re: Converting from IPB 3.4

Post by jack208 » Sat Sep 02, 2017 6:40 pm

I did create a folder and tried pointing it there. It still tells me "Could not find path". Is it looking for a specific subfolder structure in that path? If yes, do you know what that is (so that I can recreate on my end) as my Invision plan is a hosted plan so I wouldn't know what that structure is.
Ger wrote:
Sat Sep 02, 2017 6:10 pm
You need a path to the attachments avatars, etc.

jack208
Registered User
Posts: 7
Joined: Sat Sep 02, 2017 4:32 pm

Re: Converting from IPB 3.4

Post by jack208 » Sat Sep 02, 2017 6:44 pm

This is what I've....

The URL to run the converter
http://www.mysite.com/phpBB3012/install/index.php

The path where my phpBB is installed into
http://www.mysite.com/phpBB3012/

The path where I created an empty folder "forums"
http://www.mysite.com/forums

I tried various permutations for the Board Path, I suppose this is the correct one but it's still not working
Board Path = "../forums"

jack208
Registered User
Posts: 7
Joined: Sat Sep 02, 2017 4:32 pm

Re: Converting from IPB 3.4

Post by jack208 » Sun Sep 03, 2017 7:49 am

Found the answer. Posting this for other's reference.

I dived into the converter.php codes and in line 240, the converter is looking for a specific file "dav.php". Copying this file into the folder above resolved this. Thanks everyone and apology for hijacking this thread to post this semi-unrelated issue. :P

User avatar
Ger
Registered User
Posts: 1151
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: Converting from IPB 3.4

Post by Ger » Mon Sep 04, 2017 9:12 am

jack208 wrote:
Sun Sep 03, 2017 7:49 am
Thanks everyone and apology for hijacking this thread to post this semi-unrelated issue. :P
No apology needed IMO. It's good to share this info for future reference.
Checkout my extensions: Simple CMS, Feed post bot, Modbreak, Magic OGP links and Live topic update

Like my work? Buy me a coffee to keep it coming. :ugeek:

Post Reply

Return to “[3.2.x] Convertors”

Who is online

Users browsing this forum: No registered users and 7 guests

cron