Automatic Reparser Progress Indicator

Get help with installation and running phpBB 3.2.x here. Please do not post bug reports, feature requests, or extension related questions here.
Post Reply
User avatar
Matthew G. Saroff
Registered User
Posts: 42
Joined: Fri May 09, 2014 8:48 pm
Name: Matthew G. Saroff

Automatic Reparser Progress Indicator

Post by Matthew G. Saroff » Thu Jun 14, 2018 4:40 pm

OK, I've finally got my import working, (Running php7 from the command line for phpbbcli.php).

I have issues with how a lot of the posts are rendered, because they need to be reparsed.

Obviously, they need to be reparsed, but given the number of posts on the board, more than 500K, running this from the command line would take something north of 12 hours, and I've spent enough time staring at the screen, thank you very much.

I do know that the board also runs the reparser as a cron job if set properly (remove all the locks, and in the ACP go to General > server settings > Run periodic tasks from system cron set to "No").

Given the number of posts, I am assuming that it will be a few days/weeks before everything is reparsed.

Is there anything I can look at to indicate the progress of the reparsing? I am sure that some impatient users will want updates.

On edit:
I should note that I have shell access on my server provider, so if a progress file is stored somewhere, I could look at that,

On edit 2:
The 500k is number of posts, not the size of the database. The total size of the phpbb_posts table is around 850 meg.
Last edited by Matthew G. Saroff on Thu Jun 14, 2018 7:54 pm, edited 1 time in total.
--
The Dream involves 4 sets of identical twins, 2 gallons of Cool Whip, 5 quarts of chocolate syrup, 2-1/4 pounds of strawberries, satin sheets, a magnum of champagne, a trapeze, and a python.

User avatar
JoshyPHP
Code Contributor
Posts: 963
Joined: Mon Jul 11, 2011 12:28 am

Re: Automatic Reparser Progress Indicator

Post by JoshyPHP » Thu Jun 14, 2018 6:07 pm

Matthew G. Saroff wrote:
Thu Jun 14, 2018 4:40 pm
given the number of posts on the board, more than 500K, running this from the command line would take something north of 12 hours
In my experience, it should take closer to 20 minutes by CLI depending on your database. Letting the CLI run in the background should always be more efficient and faster than using the cron.
I wrote the thing that does BBCodes in 3.2.

User avatar
Matthew G. Saroff
Registered User
Posts: 42
Joined: Fri May 09, 2014 8:48 pm
Name: Matthew G. Saroff

Re: Automatic Reparser Progress Indicator

Post by Matthew G. Saroff » Fri Jun 15, 2018 2:25 am

JoshyPHP wrote:
Thu Jun 14, 2018 6:07 pm
Matthew G. Saroff wrote:
Thu Jun 14, 2018 4:40 pm
given the number of posts on the board, more than 500K, running this from the command line would take something north of 12 hours
In my experience, it should take closer to 20 minutes by CLI depending on your database. Letting the CLI run in the background should always be more efficient and faster than using the cron.
I was talking about running the CLI from the shell, and appears that it will take 12-14 hours running from the shell.

I ran it on a test install that I had sandboxed, and it processed about 8 posts/second, which means that it takes a long time to run.
--
The Dream involves 4 sets of identical twins, 2 gallons of Cool Whip, 5 quarts of chocolate syrup, 2-1/4 pounds of strawberries, satin sheets, a magnum of champagne, a trapeze, and a python.

User avatar
JoshyPHP
Code Contributor
Posts: 963
Joined: Mon Jul 11, 2011 12:28 am

Re: Automatic Reparser Progress Indicator

Post by JoshyPHP » Fri Jun 15, 2018 7:22 am

8 posts per second is very slow. Try running the CLI with --dry-run to see whether writing to the database is what takes up so much time.

I don't know how your shell works but you should be able to use something like GNU Screen to leave the CLI process running in the background even while you're logged out of the shell.
I wrote the thing that does BBCodes in 3.2.

User avatar
Matthew G. Saroff
Registered User
Posts: 42
Joined: Fri May 09, 2014 8:48 pm
Name: Matthew G. Saroff

Re: Automatic Reparser Progress Indicator

Post by Matthew G. Saroff » Fri Jun 15, 2018 1:29 pm

JoshyPHP wrote:
Fri Jun 15, 2018 7:22 am
8 posts per second is very slow. Try running the CLI with --dry-run to see whether writing to the database is what takes up so much time.

I don't know how your shell works but you should be able to use something like GNU Screen to leave the CLI process running in the background even while you're logged out of the shell.
Doing --dry-run it's all done in under 10 minutes.

What might account for the speed difference?
--
The Dream involves 4 sets of identical twins, 2 gallons of Cool Whip, 5 quarts of chocolate syrup, 2-1/4 pounds of strawberries, satin sheets, a magnum of champagne, a trapeze, and a python.

User avatar
JoshyPHP
Code Contributor
Posts: 963
Joined: Mon Jul 11, 2011 12:28 am

Re: Automatic Reparser Progress Indicator

Post by JoshyPHP » Fri Jun 15, 2018 1:40 pm

A particularly slow database I suppose.
I wrote the thing that does BBCodes in 3.2.

User avatar
Matthew G. Saroff
Registered User
Posts: 42
Joined: Fri May 09, 2014 8:48 pm
Name: Matthew G. Saroff

Re: Automatic Reparser Progress Indicator

Post by Matthew G. Saroff » Sun Jun 17, 2018 3:17 am

I just set up AAMPS, and ran the reparsing on my local machine, and the rate of processing is the change is the same, so this is not the server.

I'm wondering if there is something fsking wrong with the database.
--
The Dream involves 4 sets of identical twins, 2 gallons of Cool Whip, 5 quarts of chocolate syrup, 2-1/4 pounds of strawberries, satin sheets, a magnum of champagne, a trapeze, and a python.

Post Reply

Return to “[3.2.x] Support Forum”

Who is online

Users browsing this forum: Exclusive, johnnycashy, sean_lemieux, stevemaury and 36 guests