[CDB] Feed post bot

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Anti-Spam Guide
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
Locked
scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Wed Aug 30, 2017 11:38 pm

Is there a way to set the limit of feeds fetched each time to maybe 1 or 2? i.e The Depth.
It seem like 5 is the default.

User avatar
sysz
Registered User
Posts: 278
Joined: Mon Jan 30, 2012 11:36 pm
Location: Sweden, Helsingborg
Name: Kimmy Lindell Ekström
Contact:

Re: [3.1][3.2][RC] Feed post bot

Post by sysz » Thu Aug 31, 2017 5:28 am

Got this error on a freshly installed phpbb 3.2.1
[phpBB Debug] PHP Warning: in file [ROOT]/ext/ger/feedpostbot/classes/driver.php on line 73: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions_acp.php on line 137: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3262)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions_acp.php on line 137: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3262)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions_acp.php on line 137: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3262)
Currently working on the community

User avatar
Ger
Recognised Extension Developer
Posts: 1887
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.1][3.2][RC] Feed post bot

Post by Ger » Thu Aug 31, 2017 7:31 am

sysz wrote:
Thu Aug 31, 2017 5:28 am
Got this error on a freshly installed phpbb 3.2.1
[phpBB Debug] PHP Warning: in file [ROOT]/ext/ger/feedpostbot/classes/driver.php on line 73: Invalid argument supplied for foreach()
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions_acp.php on line 137: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3262)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions_acp.php on line 137: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3262)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions_acp.php on line 137: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3262)
Thanks for reporting. I have fixed this in the latest download version (check Github).
scriptman wrote:
Wed Aug 30, 2017 11:38 pm
Is there a way to set the limit of feeds fetched each time to maybe 1 or 2? i.e The Depth.
It seem like 5 is the default.
The Feed post bot just fetches everything that's offered in the feeds you added to the list and is newer than the latest entry previously fetched. Most feeds don't offer their entire history though, but just the latest n items.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Thu Aug 31, 2017 10:23 am

Ger wrote:
Thu Aug 31, 2017 7:31 am
The Feed post bot just fetches everything that's offered in the feeds you added to the list and is newer than the latest entry previously fetched. Most feeds don't offer their entire history though, but just the latest n items.
Could you please help add this feature?

User avatar
Ger
Recognised Extension Developer
Posts: 1887
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.1][3.2][RC] Feed post bot

Post by Ger » Thu Aug 31, 2017 11:27 am

scriptman wrote:
Thu Aug 31, 2017 10:23 am
Ger wrote:
Thu Aug 31, 2017 7:31 am
The Feed post bot just fetches everything that's offered in the feeds you added to the list and is newer than the latest entry previously fetched. Most feeds don't offer their entire history though, but just the latest n items.
Could you please help add this feature?
I think I don't really understand. When it comes to fetch more items, it's simply impossible if the feed doesn't offer more; e.g. what doesn't exist cannot be fetched. When it comes to fetching less items, it seems pretty illogical; usually when you track a feed, you'd want to fetch all items. Just fetching a couple of them seems a bit random to me.

If this isn't what you mean, than by all means elaborate. :)
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Thu Aug 31, 2017 12:42 pm

Ger wrote:
Thu Aug 31, 2017 11:27 am
I think I don't really understand. When it comes to fetch more items, it's simply impossible if the feed doesn't offer more; e.g. what doesn't exist cannot be fetched. When it comes to fetching less items, it seems pretty illogical; usually when you track a feed, you'd want to fetch all items. Just fetching a couple of them seems a bit random to me.

If this isn't what you mean, than by all means elaborate. :)
Am not talking about fetching more. Am talking about fetching less.

E.G
Feed item limit:
Set the maximum number of feed items that should be processed. The smaller the number, the faster the new feed is produced.
Minimum is "1" through "10" and maximum could be "Fetch All Items".

AND YES, some item will be left out if limit is applied as it will only fetch from latest items.

User avatar
Ger
Recognised Extension Developer
Posts: 1887
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.1][3.2][RC] Feed post bot

Post by Ger » Thu Aug 31, 2017 2:15 pm

Hmmm... My first reaction would be that such a limit is against the whole idea of feeds, which is to keep track of every item. Most feeds offer only a couple of new items each hour, many just a couple of items each week.

If you are tracking a feed that puts it's entire history in, than it would be very slow on first fetch (possibly leading to a time-out) but that's not really what feeds are intended to do. But even then; after the first fetch the extension only posts new items. And the slowness will stay since the extension needs to check every item in given feed for identifiers or the timestamp to decide whether or not to post it to your board.

If you have too much items per batch, you can decide to lower the cron interval so the extension will process the feeds more frequently. Each time it runs, you'll have less new items probaly. It's the option Interval for processing feeds (seconds) (default 1800 seconds).
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Thu Aug 31, 2017 4:21 pm

Ger wrote:
Thu Aug 31, 2017 2:15 pm
Hmmm... My first reaction would be that such a limit is against the whole idea of feeds, which is to keep track of every item. Most feeds offer only a couple of new items each hour, many just a couple of items each week.

If you are tracking a feed that puts it's entire history in, than it would be very slow on first fetch (possibly leading to a time-out) but that's not really what feeds are intended to do. But even then; after the first fetch the extension only posts new items. And the slowness will stay since the extension needs to check every item in given feed for identifiers or the timestamp to decide whether or not to post it to your board.

If you have too much items per batch, you can decide to lower the cron interval so the extension will process the feeds more frequently. Each time it runs, you'll have less new items probaly. It's the option Interval for processing feeds (seconds) (default 1800 seconds).
Why not look at it from another perspective. In a scenario where you are fetching feeds from about 10 source. How do you cope if those 10 sources release about 2 post each time?

User avatar
Ger
Recognised Extension Developer
Posts: 1887
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.1][3.2][RC] Feed post bot

Post by Ger » Thu Aug 31, 2017 5:50 pm

Why would that be a problem?
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Thu Aug 31, 2017 7:22 pm

Ger wrote:
Thu Aug 31, 2017 5:50 pm
Why would that be a problem?
It's a forum not a blog. Other users post should also be relevant.
That's where the problem comes in.

Although i have a solution i use by limiting the rss feed posts before using the output link on Feed Post Bot. But it would be nice to do it directly on ACP.

scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Sat Sep 02, 2017 4:31 pm

Wanted to ask, is there a way to include some pre-text for the source url?
E.G Source: http://example.com

Thunder_one
Registered User
Posts: 119
Joined: Sat May 09, 2015 11:00 pm

Re: [3.1][3.2][RC] Feed post bot

Post by Thunder_one » Sat Sep 02, 2017 7:13 pm

Hi

Sorry for my english. I do not understand so well. :(

Question

Can this be adjusted at what intervals he should download the feeds and post into the forum?

Many Thanks

scriptman
Registered User
Posts: 40
Joined: Wed Aug 30, 2017 11:16 pm

Re: [3.1][3.2][RC] Feed post bot

Post by scriptman » Sun Sep 03, 2017 10:59 am

scriptman wrote:
Thu Aug 31, 2017 7:22 pm
Ger wrote:
Thu Aug 31, 2017 5:50 pm
Why would that be a problem?
It's a forum not a blog. Other users post should also be relevant.
That's where the problem comes in.

Although i have a solution i use by limiting the rss feed posts before using the output link on Feed Post Bot. But it would be nice to do it directly on ACP.
Despite the feed having over 10 feeds, the bot pulls just 5 like i said earlier.
I still suggest you add that feature.

wolfofone
Registered User
Posts: 10
Joined: Sat Dec 18, 2010 11:11 pm

Re: [3.1][3.2][RC] Feed post bot

Post by wolfofone » Mon Sep 04, 2017 6:35 am

Thank you for this! You are awesome!

I have been looking for a replacement to SMIX.MODS RSS importer for a long, long time now and I recently had given up and updated to phpBB 3.2 and just went without an rss posting bot... then I happened to come across this!

It was super simple to set up and get working. https://verrytechnical.com/forums/viewt ... f=16&t=786

The latest post was by your tool, all the older ones were done by the SMIX.MODs feed that has been abandoned for forever (I did write two articles on supporting getting it working with 3.0.x forums but didn't have the know-how to update the code to work with 3.1+). I will need to up the text limit count though as I chose 50 just to test, I think 150 will be better :-).

I will certainly point people towards your project and include a link to your donation link in an upcoming blog post :-).

wolfofone
Registered User
Posts: 10
Joined: Sat Dec 18, 2010 11:11 pm

Re: [3.1][3.2][RC] Feed post bot

Post by wolfofone » Mon Sep 04, 2017 6:38 am

Thunder_one wrote:
Sat Sep 02, 2017 7:13 pm
Hi

Sorry for my english. I do not understand so well. :(

Question

Can this be adjusted at what intervals he should download the feeds and post into the forum?

Many Thanks
You can set it to post manually when you go into ACP and manually tell it to go grab the feed and post or by adjusting the auto-posting interval in the "Interval for processing feeds (seconds):" box. 1800 seconds (default is every 30 minutes, you can change this so that it checks and post more or less often).

Locked

Return to “Extensions in Development”