phpBB • Free and Open Source Forum Software

mikeinthe852 · by **mikeinthe852** » Sat May 25, 2024 12:10 pm

Everything is updated, phpbb 3.3.11, php 8.1, Digest 3.3.18
Once a week or so we get this cron error. I ran -vvvv for details
Any thoughts?

In exception_subscriber.php line 51:

[RuntimeException (1)]
Could not obtain cron lock.

Exception trace:
at /home/nestaadmin/public_html/forum/phpbb/console/exception_subscriber.php:51
phpbb\console\exception_subscriber->on_error() at /home/nestaadmin/public_html/forum/vendor/symfony/event-dispatcher/EventDispatcher.php:214
Symfony\Component\EventDispatcher\EventDispatcher->doDispatch() at /home/nestaadmin/public_html/forum/vendor/symfony/event-dispatcher/EventDispatcher.php:44
Symfony\Component\EventDispatcher\EventDispatcher->dispatch() at /home/nestaadmin/public_html/forum/phpbb/event/dispatcher.php:62
phpbb\event\dispatcher->dispatch() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Application.php:1028
Symfony\Component\Console\Application->doRunCommand() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Application.php:255
Symfony\Component\Console\Application->doRun() at /home/nestaadmin/public_html/forum/phpbb/console/application.php:137
phpbb\console\application->doRun() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Application.php:148
Symfony\Component\Console\Application->run() at /home/nestaadmin/public_html/forum/bin/phpbbcli.php:90

In run.php line 97:

[phpbb\exception\runtime_exception (1)]
CRON_LOCK_ERROR

Exception trace:
at /home/nestaadmin/public_html/forum/phpbb/console/command/cron/run.php:97
phpbb\console\command\cron\run->execute() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Command/Command.php:255
Symfony\Component\Console\Command\Command->run() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Application.php:1010
Symfony\Component\Console\Application->doRunCommand() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Application.php:255
Symfony\Component\Console\Application->doRun() at /home/nestaadmin/public_html/forum/phpbb/console/application.php:137
phpbb\console\application->doRun() at /home/nestaadmin/public_html/forum/vendor/symfony/console/Application.php:148
Symfony\Component\Console\Application->run() at /home/nestaadmin/public_html/forum/bin/phpbbcli.php:90

MarkDHamill · by **MarkDHamill** » Sat May 25, 2024 1:27 pm

For some reason attempting to lock the cron process for digests is failing. If you use a tool like phpMyAdmin and search the phpbb_config table, search for a config_name of "cron_lock" and change its value to zero, then purge the cache, it should reset, but it's a workaround, not a solution. Likely some other cron is the issue, not digests. There is a dated extension that can help troubleshoot issues like these, but getting it to work on phpBB 3.3 will likely prove challenging.

https://www.phpbb.com/customise/db/exte ... ronstatus/

mikeinthe852 · by **mikeinthe852** » Sat May 25, 2024 2:37 pm

Hi Mark
Yes I have that extension installed. Can see a screenshot here:
Re: line 250: Unable to find template "digests_html.txt"

'tidy sessions' is being locked when the digest is run, and doesn't unlock. Hard to imagine this being related though.
This only happens intermittently, perhaps once a week or two. Resetting the digest unlocks everything, but that's just a workaround, not a fix. Last time it was the change in root/path, but that was a constant breakage. These intermittent bugs are frustrating.

mikeinthe852 · by **mikeinthe852** » Sat May 25, 2024 2:46 pm

FWIW I dumped all this into MS Copilot. It responded:

I understand that the issue occurs intermittently, making it challenging to pinpoint the exact cause. In such cases, let’s explore additional strategies to address the intermittent cron lock issue in your phpBB forum:

Check for Specific Error Logs:
Review any error logs related to cron jobs or the phpBB application.
Look for patterns or specific error messages that coincide with the times when the cron lock occurs.
These logs might provide more context about what triggers the issue.

Monitor System Resources:
Keep an eye on system resources (CPU, memory, disk space) during cron job execution.
Intermittent issues could be related to resource exhaustion or contention.
Consider using monitoring tools or scripts to track resource usage.

Debugging Hooks:
Add additional debug statements or hooks within your cron job scripts.
Log relevant information (e.g., timestamps, intermediate steps, variables) to help diagnose the issue.
These logs can provide insights even when the issue is intermittent.

Review Extensions and Custom Code:
If you’re using any phpBB extensions or custom code, review their behavior.
Extensions might interact with cron jobs in unexpected ways.
Disable or temporarily remove extensions to see if the issue persists.

Database Locks or Deadlocks:
Investigate whether database locks or deadlocks are causing the cron lock.
Check for long-running queries or transactions that might interfere with cron execution.

Database Maintenance:
Optimize your database tables regularly.
Repair and optimize tables to prevent issues related to table locks.

Scheduled Tasks Outside phpBB:
Check if there are other scheduled tasks (outside of phpBB) that coincide with cron job execution.
These external tasks might impact the cron lock.

Third-Party Services:
If your cron job interacts with external services (e.g., SMTP servers), monitor their behavior.
Intermittent failures in external services could lead to cron lock issues.

Remember that intermittent issues can be challenging to diagnose, but thorough monitoring, logging, and systematic debugging can help identify the root cause. If possible, involve a developer familiar with phpBB to assist in troubleshooting.

One idea is to disable all other extensions and see how it goes for a week or two? Then over time add them back in one by one? Time consuming, especially just waiting for it to break intermittently again. We have 15 extensions, but all are up to date and so is the forum.

Copilot also mentioned

Troubleshooting:
To address the issue, focus on the specific file and method mentioned in the trace (exception_subscriber.php).
Investigate why the cron lock cannot be obtained and whether there are any issues related to event dispatching or Symfony components.

Even though you wrote it, just to save you time looking, line 51 is
$exception = new \RuntimeException($message, $original_exception->getCode(), $original_exception);

MarkDHamill · by **MarkDHamill** » Sat May 25, 2024 3:35 pm

Again, I doubt this is a digests issue. Rather digests is reporting the problem that it couldn't obtain a cron lock, probably because some previous cron job did not release the lock.

mikeinthe852 · by **mikeinthe852** » Sun May 26, 2024 2:38 am

Why does the cron stay locked? Shouldn't it unlock after 60 minutes?
Should 'is dynamic' be set to 1?
Any other settings I need to look at here?

mikeinthe852 · by **mikeinthe852** » Sun May 26, 2024 2:43 am

MarkDHamill wrote: probably because some previous cron job did not release the lock.

It seems to always be that 'tidy sessions' is locked when the digest tries to run and doesn't unlock. Perhaps lack of resources? Other?

mikeinthe852 · by **mikeinthe852** » Sun May 26, 2024 8:49 am

Copilot
The tidy_sessions task in phpBB’s cron system is responsible for cleaning up expired sessions and maintaining session-related data. When this task encounters issues, it can lead to cron locks. Let’s explore potential causes:

Database Locks or Deadlocks:
If the tidy_sessions task interacts with the database, locks or deadlocks might occur.
Check for long-running queries or transactions that could interfere with session cleanup.
Ensure that the database tables related to sessions are optimized.

Session Table Size:
If the phpbb_sessions table becomes too large, it can impact the performance of the tidy_sessions task.
Consider monitoring the size of this table and optimizing it periodically.
If the table is excessively large, it might cause locks during cleanup.

Concurrency Issues:
Multiple instances of the tidy_sessions task running concurrently can lead to locks.
Ensure that the cron job is not scheduled too frequently or that there are no overlapping executions.

PHP Version Compatibility:
You mentioned switching to PHP 7.3. Some phpBB extensions or custom code might not be fully compatible with newer PHP versions.
Verify that your extensions are up to date and compatible with PHP 7.3.

Database Configuration:
Check your database configuration (e.g., MySQL settings).
Adjust settings related to locks, timeouts, and concurrency if needed.

Server Resources:
Insufficient server resources (CPU, memory) can cause locks.
Monitor resource usage during cron execution.

Debugging:
Temporarily reduce verbosity in your cron job to identify additional error messages.
Run the cron job manually (without using cron) to observe any direct output or errors.

Scheduled Tasks Outside phpBB:
Check if there are other scheduled tasks (outside of phpBB) that coincide with cron job execution.
These external tasks might impact the tidy_sessions task.

Not sure why copilot thinks I upgraded to php 7.3. We run 8.1 and it was upgraded more than one year ago, so I will ignore that suggestion.
Because I know nothing about phpbb, I have to defer to Mark and other experts to ask if any of this provides clues to a 'fix'?

MarkDHamill · by **MarkDHamill** » Sun May 26, 2024 11:47 am

Digests do lots of database queries so can be resource intensive. Sometimes this will result in mailing of digests stopping abruptly if you abruptly hit a query limit. That could result in a cron lock because digests don't get a chance to clear the lock. Your web host might have insight into how often this happens. They may want to sell you a plan with more resources. If you are subscribing all members to get digests, don't do this. Let people elect to subscribe in the UCP.

mikeinthe852 · by **mikeinthe852** » Mon May 27, 2024 11:54 am

But if the lock is 'tidy_sessions' then it really isn't the digest taking resources that is causing the issue, correct?
In addition, most recently, the lock came during the day when no digests are scheduled to go out.

I know less than zero about tidy_sessions

Is there any way to set the digest to run hourly at quarter past the hour? That would give all other processes time to complete and not cause conflicts. Just brainstorming.

And why does the cron stay locked? Shouldn't it unlock after 60 minutes?

MarkDHamill · by **MarkDHamill** » Mon May 27, 2024 12:32 pm

You can make the suggestion on the feature on tracker.phpbb.com to unlock crons.

The cron can be programmed to start at 15 minutes past the hour if you want.

It is likely some other extension that you have installed is locking the cron and not releasing it. There will be a /cron/task folder for the extension if it has a cron feature. Disabling each of these in turn, unlocking the cron and seeing what happens should reveal the extension causing the problem.

mikeinthe852 · by **mikeinthe852** » Mon May 27, 2024 3:23 pm

Here are our extensions

This is the cron

The cron can be programmed to start at 15 minutes past the hour if you want.

What I meant was the digest runs later, say 15 minutes past the hour, to allow the other services that run at the top of the hour to complete and not fight for resources on our shared server.

And why does the cron stay locked? Shouldn't it unlock after 60 minutes?

Isn't the 60 minute unlock already built into phpbb?

There will be a /cron/task folder for the extension if it has a cron feature.

Ok I see 4 with a cron folder now. Does it have to be cron/task for it to be suspect?
2 have a cron folder, but no task subfolder.

Mediaembed and Viglink have cron folders, but no task subfolders.
Digests and advanced polls have a cron folder and a task subfolder

MarkDHamill · by **MarkDHamill** » Mon May 27, 2024 4:49 pm

The recommended location for cron logic is the /cron/task folder.

See this to learn how to program a cron to start at 15 minutes past the hour:

https://linuxize.com/post/cron-jobs-eve ... 5-minutes/

phpBB has no functionality to reset crons.

mikeinthe852 · by **mikeinthe852** » Tue May 28, 2024 2:02 am

Thanks Mark
With your assistance about 2 years ago I set the cron like this
*/5**** export PATH=/usr/local/bin:$PATH && cd /home/nestaadmin/public_html/forum && ./bin/phpbbcli.php cron:run

My question is a bit different, If the digest is set to run every hour at :00 and other processes (such as tidy_sessions) are also set to run at that same time, could that cause shared hosting to run out of resources and throw a fault, which may cause the other process to lock the cron, therefore preventing the Digest from running? Basically, too many processes are fighting for limited resources and one of them locks up.

I am wondering if adjusting the run time of the digest only may resolve this. I have no idea if that is even possible.

First, I will try disabling the other extensions to see if the lock reoccurs.

MarkDHamill · by **MarkDHamill** » Tue May 28, 2024 2:18 am

It sounds logical. In most cases it takes only a few seconds to run the mailing program. Note */5**** will run the process every five minutes, including at the top of the hour. To run it only at the :15 interval I would think you would want something like 15 * * * *. You should run digests at least once an hour to ensure digests are received in the desired hour.

phpBB • Free and Open Source Forum Software

Digests

Could not obtain cron lock - Digests

Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

Re: Could not obtain cron lock

advertisements