How to use the textreparser after manual conversion to 3.2?

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
IronCrown
Registered User
Posts: 16
Joined: Thu Oct 18, 2007 6:34 am

Re: How to use the textreparser after manual conversion to 3.2?

Post by IronCrown » Tue Feb 21, 2017 7:41 pm

Theriddler1 wrote:
Tue Feb 21, 2017 1:32 pm
You can reparse the bbcodes with the cli command

Code: Select all

php bin/phpbbcli.php --safe-mode reparser:reparse post_text --ansi
I think you saved me there! With the converter failing on my database, I tried running this command from the shell and it actually started fixing my manually imported old posts. Smilies started showing up properly and tags were beautified.

However, I noticed that some tags were still broken, because the reparser apparently needs clean

Code: Select all

[quote]
tags while my old posts had weird tags like

Code: Select all

[quote:34df345]
So I exited the reparser with CTRL+C (or so I thought), pulled a dump from the posts table and edited the post_text in an external editor that can do RegEx. I cleaned up the tags and then re-imported the table.

Now I wanted to restart the reparser... but it turns out that it is still running! Actually it was reparsing the posts while I was deleting and re-importing them, so now there is a mess of partially reparsed posts that also contain broken tags.

Any idea how I can stop the reparsing process from the shell?

As soon as I can do that, and once more re-import my cleaned-up old posts and restart the reparsing, I think I'm finally there...

edit: Ok, I guess closing the shell kills the process...

But the reparser only fixed the smilies and links, quotes and images are still not shown correctly :(

Code: Select all

[quote=User]This is a quote.[/quote]
Any idea why the reparser does not create proper quotes out of this or what it expects in the source to do it?

IronCrown
Registered User
Posts: 16
Joined: Thu Oct 18, 2007 6:34 am

Re: How to use the textreparser after manual conversion to 3.2?

Post by IronCrown » Tue Feb 21, 2017 8:47 pm

When I quote one of those posts and preview it unaltered, it looks like it should. Isn't the software at this point doing the same thing that the reparser should be doing? :?:

User avatar
Theriddler1
Registered User
Posts: 445
Joined: Sat Aug 27, 2011 11:00 pm
Location: NL
Name: Theriddler❶
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by Theriddler1 » Sun Feb 26, 2017 9:14 am

IronCrown wrote:
Tue Feb 21, 2017 7:41 pm
tags while my old posts had weird tags like

Code: Select all

[quote:34df345]
That's normal (if you're talking about the database post_text content) because every bbcode has an unique id (uid)
If a post contains BBCode and are good to be proccessed , then an unique uid code will be assigned to that bbcode. when that post is called with that bbcode then that bbcode will be proccessed and the bbcode will then be transfered to html, by patterns the BBcode will be recognized and with the BBcode (bbcode.html) template you see the bbcode correctly in html format.

i'm thinking why not first (if possible) update the phpbb 2.0.x board to phpbb 3.0.14

then when on 3.0.x use the phpbb 3 support toolkit and use the database cleaner tool (Allows you to reset certain parts of your phpBB board's database to its original setup (removes all database modifications).

then update to 3.1 and finally update to 3.2
Theriddler - Moderator @ phpBB.nl | phpBBservice.nl Team-member
ImageSupport Tools | Image My Extensions | buy me a beer Image

IronCrown
Registered User
Posts: 16
Joined: Thu Oct 18, 2007 6:34 am

Re: How to use the textreparser after manual conversion to 3.2?

Post by IronCrown » Sun Feb 26, 2017 4:55 pm

Thanks, but the converters for 3.2 as well as 3.0 fail on my old database. Probably because of the modifications I made to my phpbb 2, but the exact reason can never be known because the converter does not seem to log anything or provide any error messages (why...?).

I installed a fresh 3.0.14 just to look at how it stores the post text in the database. Hoping that if I can manually modify my old posts to look just like that, then the 3.2 reparser will fix my problems. But there is nothing in 3.0.14 that would be different from how my data already looks. Well except that the " signs in quote tags are html-encoded, but the quote tags without " (unspecified author), and the img tags, are also not correctly reparsed so that cannot be it.

In the end all that does not matter at all anyway due to this simple fact: 3.2 CAN already fix my old posts without any problem: I just have to edit any old post, click send without making any actual edits, and voila, it is fixed and 100% correct! So how my data looks is definitely not the problem.

So the question is: How do I make phpbb 3.2 just do that to all the posts in the database which it does when I manually "non-edit" them? This must be very simple because the board obviously already has everything that it needs for this. Either there is already some way to do this or it should be extremely trivial for any developer familiar with the code to write some function that does this. I'm trying to do it myself but I just can't seem to find the relevant code for the reparser. I can call it from the command line but I can't find anywhere the code that is called...

User avatar
3Di
Registered User
Posts: 12694
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by 3Di » Mon Feb 27, 2017 6:12 am

IronCrown wrote:
Sun Feb 26, 2017 4:55 pm
So the question is: How do I make phpbb 3.2 just do that to all the posts in the database which it does when I manually "non-edit" them? This must be very simple because the board obviously already has everything that it needs for this. Either there is already some way to do this or it should be extremely trivial for any developer familiar with the code to write some function that does this. I'm trying to do it myself but I just can't seem to find the relevant code for the reparser. I can call it from the command line but I can't find anywhere the code that is called...
When a board is updated from 3.1 to 3.2, the reparser is called into service in two ways. First, migrations are used to reparse some of the smaller database items (forum descriptions, for example). Second, cron tasks are used to incrementally reparse the larger database items (posts & PMs).
Ref.: https://area51.phpbb.com/docs/dev/32x/e ... t-reparser

There is a cron job shipped with 3.2.0, that does automatically the trick, depending on how much big is your DB it will take more time to finish, basically you should see newly reparsed posts every day till the end of the job.
Min =1 / Max = 100 are the constants used there.

phpBB_root/phpbb/cron/task/text_reparser/reparser.php ..

Automatically triggered by visitors, I guess.
<!-- IF not S_IS_BOT -->{RUN_CRON_TASK}<!-- ENDIF -->
Want to compensate me for my interest? Donate
Please PM me only to request paid works. Thx.
Extensions, Scripts, MOD porting, Update/Upgrades
My development's activity º PhpStorm's proud user

User avatar
Ger
Recognised Extension Developer
Posts: 1557
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by Ger » Mon Feb 27, 2017 7:46 am

IronCrown wrote:
Tue Feb 21, 2017 7:41 pm
So I exited the reparser with CTRL+C (or so I thought), pulled a dump from the posts table and edited the post_text in an external editor that can do RegEx. I cleaned up the tags and then re-imported the table.

Now I wanted to restart the reparser... but it turns out that it is still running! Actually it was reparsing the posts while I was deleting and re-importing them, so now there is a mess of partially reparsed posts that also contain broken tags.
You can run this query to reset the reparse_lock:

Code: Select all

UPDATE `phpbb_config` SET `config_value`='0' WHERE `config_name`='reparse_lock';
You can then re-run the CLI command.

Actually in your case I'd use this CLI command:

Code: Select all

php bin/phpbbcli.php --safe-mode reparser:reparse  --ansi
In the previous command you only reparse the post text, (as specified) while this takes all the rich text fields, including, PM's, signatures, etc.
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote
Newest: Quoted Where + anonymize

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

IronCrown
Registered User
Posts: 16
Joined: Thu Oct 18, 2007 6:34 am

Re: How to use the textreparser after manual conversion to 3.2?

Post by IronCrown » Tue Feb 28, 2017 8:26 am

That sounds really promising - will try it as soon as I find the time!

MattisBLN
Registered User
Posts: 20
Joined: Thu Jan 09, 2014 12:30 pm
Location: Berlin, Germany
Name: Matthias Henkel
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by MattisBLN » Fri Mar 09, 2018 2:43 pm

IronCrown wrote:
Sun Feb 26, 2017 4:55 pm
There is a cron job shipped with 3.2.0, that does automatically the trick, depending on how much big is your DB it will take more time to finish, basically you should see newly reparsed posts every day till the end of the job.
Min =1 / Max = 100 are the constants used there.

phpBB_root/phpbb/cron/task/text_reparser/reparser.php ..

Automatically triggered by visitors, I guess.
<!-- IF not S_IS_BOT -->{RUN_CRON_TASK}<!-- ENDIF -->
Hi, does anyone know for sure how these cron_tasks are triggered in phpBB? Is there a way to trigger those tasks by cron-tab (serverside)? And how do I know when the reparsing of all post etc. is finished?

I just migrated a phpBB 3.0.14 installation to 3.1.12 and then to 3.2.2. A direct upgrade from 3.0.14 to 3.2.2 failed...

My forum has about 107.000 posts - so 100 posts reparsed per day migt take a while... Also I use the ABBC3-Extension, does the reparser also recognize those BBCodes aswell as manually added BBCodes?

My problem is, that by opening some posts in the forum I sometimes get error messages from bbcode.php. Editing and reposting those posts does the job. I just would like to have the reparsing done before reopening the forum to public to avoid those error-messages...

Thanks for any idea!

Cheers, Matthias

MattisBLN
Registered User
Posts: 20
Joined: Thu Jan 09, 2014 12:30 pm
Location: Berlin, Germany
Name: Matthias Henkel
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by MattisBLN » Sat Mar 10, 2018 6:13 am

Hi again,

here ist what I found out - maybe it helps someone.

First of all, in phpBB 3.2 cronjobs are triggered with each pageview of the forum by default (overall_footer.html of the installed style). This can be changed to serverside-cronjobs in ACP -> Main -> Server and Domain -> second option. You will have to add "bin/phpbbcli.php cron:run" to the crontab of your server and switch that option in ACP.

The textreparser has a function to check if it should run. This function sets the reparser to run once in an hour (waiting time of 3600 seconds). It is also set to parse a maximum of 100 posts per call. So it would parse 100 posts/hour by default, depending on pageviews.

I temporarily changend the waiting time to 10 seconds. Then I used the chromebrowser with "auto refresh" plugin to reload the pageview of a thread within my forum every 30 seconds. This leads to parsing 200 posts/minute -> 12.000 posts/hour. So reparsing my complete posts would need about 9 hours.

Don't forget to undo the changes after the job is done...

Greetings, Matthias

User avatar
Ger
Recognised Extension Developer
Posts: 1557
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by Ger » Sun Mar 11, 2018 12:34 pm

Have you triedthe cli option?
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote
Newest: Quoted Where + anonymize

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

MattisBLN
Registered User
Posts: 20
Joined: Thu Jan 09, 2014 12:30 pm
Location: Berlin, Germany
Name: Matthias Henkel
Contact:

Re: How to use the textreparser after manual conversion to 3.2?

Post by MattisBLN » Sun Mar 11, 2018 1:29 pm

Not so sure how to use this without shell-access...

Post Reply

Return to “[3.2.x] Support Forum”

Who is online

Users browsing this forum: Exclusive, janus_zonstraal and 44 guests