[CDB] Feed post bot

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Anti-Spam Guide
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
User avatar
Ger
Recognised Extension Developer
Posts: 1405
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Tue Mar 13, 2018 10:38 am

Looks good from here.

Can you try to add another feed, like https://www.phpbb.com/feeds/rss/

Also, can you try and change this line to:

Code: Select all

trigger_error($user->lang('FPB_FEED_URL_INVALID') . ' CANNOT DETECT OR FETCH' . adm_back_link($this->u_action), E_USER_WARNING);
and try again. It changes the error message to be a bit more specific.
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote

Like my work? Buy me a coffee to keep it coming. :ugeek:

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Tue Mar 13, 2018 1:08 pm

Invalid feed URL. This may be the result of a duplicate in your feed list or simply an URL that does not meet the specifications CANNOT DETECT OR FETCH
But interesting - the same url shows diiferent results
Here from my forumImage
Last is up

And here from RSS CentralImage
Last is down

Completely different items from https://derstandard.at/?page=rss&ressortid=659

The same result i get from RSS Central with https://derstandard.at/?page=rss&ressort=seite1
So it seems, that in my installation it doesnt tak the complete URL
My installaton acceps https://derstandard.at/?page=rss&ressort

User avatar
Ger
Recognised Extension Developer
Posts: 1405
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Tue Mar 13, 2018 2:57 pm

ottischwenk wrote:
Tue Mar 13, 2018 1:08 pm
Invalid feed URL. This may be the result of a duplicate in your feed list or simply an URL that does not meet the specifications CANNOT DETECT OR FETCH
Is this also when adding the phpBB feed?
But interesting - the same url shows diiferent results--//--
Completely different items from https://derstandard.at/?page=rss&ressortid=659
Hmm, guess that's a bug. It seemed that the & was translated to & which cURL doesn't handle properly. I pushed a small fix just now.
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote

Like my work? Buy me a coffee to keep it coming. :ugeek:

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Tue Mar 13, 2018 3:24 pm

Ger wrote:
Tue Mar 13, 2018 2:57 pm
ottischwenk wrote:
Tue Mar 13, 2018 1:08 pm
Invalid feed URL. This may be the result of a duplicate in your feed list or simply an URL that does not meet the specifications CANNOT DETECT OR FETCH
Is this also when adding the phpBB feed?
No, but in this case i didnt get any result when trying to get a feed - this forum was empty

User avatar
Ger
Recognised Extension Developer
Posts: 1405
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Tue Mar 13, 2018 4:04 pm

ottischwenk wrote:
Tue Mar 13, 2018 3:24 pm
No, but in this case i didnt get any result when trying to get a feed - this forum was empty
I'm not sure I understand what you're saying here...

Anyway: can you try and update to the latest version I pushed earlier today? Does that work for you?
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote

Like my work? Buy me a coffee to keep it coming. :ugeek:

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Tue Mar 13, 2018 4:05 pm

Ger wrote:
Tue Mar 13, 2018 2:57 pm
Hmm, guess that's a bug. It seemed that the & was translated to & which cURL doesn't handle properly. I pushed a small fix just now.
Hmmm - in https://www.presseportal.de/rss/medien-kultur.rss2 is not &, and there is also
Invalid feed URL. This may be the result of a duplicate in your feed list or simply an URL that does not meet the specifications CANNOT DETECT OR FETCH

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Tue Mar 13, 2018 4:29 pm

Ger wrote:
Tue Mar 13, 2018 4:04 pm
ottischwenk wrote:
Tue Mar 13, 2018 3:24 pm
No, but in this case i didnt get any result when trying to get a feed - this forum was empty
I'm not sure I understand what you're saying here...

Anyway: can you try and update to the latest version I pushed earlier today? Does that work for you?
https://derstandard.at/?page=rss&ressortid=659 works correct now.
https://www.presseportal.de/rss/medien-kultur.rss2 gets the error message and doesnt work

User avatar
Ger
Recognised Extension Developer
Posts: 1405
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Wed Mar 14, 2018 9:11 am

It seems to boil down to this function: https://github.com/GerB/feedpostbot/blo ... #L143-L177

Somehow, you don't get any response through cURL (you say you have that enabled). You might want to try enforce the file_get_contents method, by temporary replacing this line by this:

Code: Select all

        if (!function_exists('curl_init') || (strpos($url, 'presseportal.de') !== FALSE) )
You're basically make a hard-coded exception here. It's a long shot though, mostly cURL works better.

I've read your PM, but I don't think my php.ini will help you. cURL information isn't in there. This is from my phpinfo:

Code: Select all

cURL support 	enabled
cURL Information 	7.52.1
Age 	3
Features
AsynchDNS 	No
Debug 	No
GSS-Negotiate 	No
IDN 	No
IPv6 	Yes
Largefile 	Yes
NTLM 	Yes
SPNEGO 	No
SSL 	Yes
SSPI 	No
krb4 	No
libz 	Yes
CharConv 	No
Protocols 	dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, ldaps, pop3, pop3s, rtsp, smb, smbs, smtp, smtps, telnet, tftp
Host 	x86_64-apple-darwin13.4.0
SSL Version 	OpenSSL/1.0.2j
ZLib Version 	1
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote

Like my work? Buy me a coffee to keep it coming. :ugeek:

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Wed Mar 14, 2018 10:36 am

Thank you, that helped

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Wed Mar 14, 2018 12:46 pm

From my phpinfo

Code: Select all

curl
cURL support 	enabled
cURL Information 	7.22.0
Age 	3
Features
AsynchDNS 	No
CharConv 	No
Debug 	No
GSS-Negotiate 	Yes
IDN 	Yes
IPv6 	Yes
krb4 	No
Largefile 	Yes
libz 	Yes
NTLM 	Yes
NTLMWB 	Yes
SPNEGO 	No
SSL 	Yes
SSPI 	No
TLS-SRP 	Yes
Protocols 	dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, pop3, pop3s, rtmp, rtsp, smtp, smtps, telnet, tftp
Host 	x86_64-pc-linux-gnu
SSL Version 	GnuTLS/2.12.14
ZLib Version 	1.2.3.4 
rom my phpinfo

User avatar
Ger
Recognised Extension Developer
Posts: 1405
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Wed Mar 14, 2018 1:06 pm

Your cURL seems just fine. I think somehow it's something on presseportals' end, maybe it detects server sided crawling or something like that.

I've pushed a small change that once cURL fails both in normal style and fails posing as Googlebot as well, it falls back to the file_get_contents method. Effectively doing the same as the previous hardcoded option for presseportal in my previous post. If this works for you than I think it's OK.
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote

Like my work? Buy me a coffee to keep it coming. :ugeek:

ottischwenk
Registered User
Posts: 13
Joined: Tue Mar 06, 2018 2:14 pm

Re: [3.2][RC] Feed post bot

Post by ottischwenk » Wed Mar 14, 2018 1:35 pm

Thank you - it worked for me

whocarez
Registered User
Posts: 131
Joined: Sat Apr 18, 2009 9:19 pm

Re: [3.2][RC] Feed post bot

Post by whocarez » Sun Apr 15, 2018 9:42 am

Hello, strange behaviour of feedpostbot with the latest version from github (Latest commit 456f690 on 14 Mar).
phpbb 3.2.1
PHP 5.6.33-0+deb8u1 (cli) / PHP 7.0.29-1~dotdeb+8.1 (cli) ( NTS )

Every night Feedpostbot is running in an endless loop like this:

Code: Select all

Fatal error: Maximum function nesting level of '1000' reached, aborting! in /var/www/forum/ext/ger/feedpostbot/classes/driver.php on line 146

Call Stack:
    0.0004     365512   1. {main}() /var/www/forum/bin/phpbbcli.php:0
    0.5868   14713488   2. phpbb\console\application->run() /var/www/forum/bin/phpbbcli.php:87
    0.5885   14914488   3. phpbb\console\application->doRun() /var/www/forum/vendor/symfony/console/Application.php:117
    0.5885   14914488   4. phpbb\console\application->doRun() /var/www/forum/phpbb/console/application.php:122
    0.5887   14914488   5. phpbb\console\application->doRunCommand() /var/www/forum/vendor/symfony/console/Application.php:193
    0.5894   14935280   6. phpbb\console\command\cron\run->run() /var/www/forum/vendor/symfony/console/Application.php:860
    0.5895   14936296   7. phpbb\console\command\cron\run->execute() /var/www/forum/vendor/symfony/console/Command/Command.php:242
    0.6093   14936144   8. phpbb\console\command\cron\run->run_all() /var/www/forum/phpbb/console/command/cron/run.php:89
    0.7603   15361360   9. phpbb\cron\task\wrapper->run() /var/www/forum/phpbb/console/command/cron/run.php:125
    0.7603   15361416  10. phpbb\cron\task\wrapper->__call() /var/www/forum/phpbb/console/command/cron/run.php:125
    0.7603   15361792  11. call_user_func_array:{/var/www/forum/phpbb/cron/task/wrapper.php:104}() /var/www/forum/phpbb/cron/task/wrapper.php:104
    0.7603   15361856  12. ger\feedpostbot\cron\task\feedpostbot_cron->run() /var/www/forum/phpbb/cron/task/wrapper.php:104
    0.7603   15361856  13. ger\feedpostbot\classes\driver->fetch_all() /var/www/forum/ext/ger/feedpostbot/cron/task/feedpostbot_cron.php:50
    0.7672   15361856  14. ger\feedpostbot\classes\driver->parse_feed() /var/www/forum/ext/ger/feedpostbot/classes/driver.php:126
    0.7672   15361912  15. ger\feedpostbot\classes\driver->get_content() /var/www/forum/ext/ger/feedpostbot/classes/driver.php:192
    0.8031   15362032  16. ger\feedpostbot\classes\driver->get_content() /var/www/forum/ext/ger/feedpostbot/classes/driver.php:173
    0.8081   15362152  17. ger\feedpostbot\classes\driver->get_content() /var/www/forum/ext/ger/feedpostbot/classes/driver.php:175
[...]
   24.6374   17978736 999. ger\feedpostbot\classes\driver->get_content() /var/www/forum/ext/ger/feedpostbot/classes/driver.php:175
   24.6820   17982808 1000. ger\feedpostbot\classes\driver->get_content() /var/www/forum/ext/ger/feedpostbot/classes/driver.php:173
   
After that Cron is for a while locked, but Feedpostbot doesn't start to post automatically after that, so I have to run it once manually and then Feedpostbot is doing his job till next night. I tried it with both php 5 and 7 and already increased xdebug.max_nesting_level, but without change.

Code: Select all

[xdebug]
xdebug.max_nesting_level = 1000
So actually Feedpostbot is working with seven feeds, but only at night - time is depending on posting intervall - it runs into this problem.
Where to look?

User avatar
Ger
Recognised Extension Developer
Posts: 1405
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Mon Apr 16, 2018 7:14 am

Function nesting limits are exclusive to xdebug AFAIK. PHP itself doesn't care how many nesting you have. You might want to consider disabling xdebug on your production environment.

Now if this works from the ACP but not from the CLI, it might just be that CLI uses another php.ini file. Find out by putting this in your console:
php -i |grep php\.ini
My extensions: Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update and Modern Quote

Like my work? Buy me a coffee to keep it coming. :ugeek:

whocarez
Registered User
Posts: 131
Joined: Sat Apr 18, 2009 9:19 pm

Re: [3.2][RC] Feed post bot

Post by whocarez » Mon Apr 16, 2018 11:20 am

Ger wrote:
Mon Apr 16, 2018 7:14 am
Function nesting limits are exclusive to xdebug AFAIK. PHP itself doesn't care how many nesting you have. You might want to consider disabling xdebug on your production environment.

Now if this works from the ACP but not from the CLI, it might just be that CLI uses another php.ini file. Find out by putting this in your console:
php -i |grep php\.ini
ok, thanks. Switched XDEBUG off for the CLI. Interesting if Feedpostbot will run through next night.

Locked

Return to “Extensions in Development”

Who is online

Users browsing this forum: Acorn, Bermudez, bill1980, Flerex, Froddelaar, MarkDHamill, Naguissa, njs, TheButcher2, wrkalot and 55 guests