[CDB] Feed post bot

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Suggested Hosts
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
Locked
User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Fri Dec 22, 2017 11:21 am

I have just started using this and I am having some mixed results. In principle I have it working apart from one feed. Having tried to do some debugging I think the problem is that the content is within <content:encoded> tags and this is being dropped when this extension is process the data.

Is this something beyond the scope of this extension or do I need to do somethong else?
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Fri Dec 22, 2017 12:15 pm

It could be that the content you're looking for in the feed isn't within designated tags, but it's hard to say without the source. Can you provide the feed URL and tell me what you'd expect to be processed vs. the result?
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Fri Dec 22, 2017 2:13 pm

Got your feed trough PM.
david63 wrote:
Fri Dec 22, 2017 11:21 am
I think the problem is that the content is within <content:encoded> tags and this is being dropped when this extension is process the data.
You are right.
As you may or may not have seen, I use the SimpleXML extension to parse the feed contents. For RSS, I iterate through the <item> list, check for existence of required or possible fallback properties and push it to the list.

In your case, you use namespacing. I haven't implemented that, so when I dump the contents of an item from your feed, I get these properties:

Code: Select all

object(SimpleXMLElement)[10]
  public 'title' => string '...' (length=20)
  public 'link' => string '...' (length=78)
  public 'category' => string '...' (length=4)
  public 'comments' => string '...' (length=87)
  public 'author' => string '...' (length=26)
  public 'pubDate' => string '...' (length=31)
  public 'guid' => string '...' (length=62)
So your content is missing, and the feed post bot falls back to the title.

In this case I would have to check the namespaces and if the content namespace is available, check if it has the encoded attribute and if it exists, use it for the post content.

I might implement it, but I have to think about that (and what's the optimal approach). It's essentially another fallback in an ever-growing list of fallbacks. When I started this extension I had no idea about the dozens of formats of feeds. Can't you just keep things simple on your end? :P

On a side note: I got some recommendations when validating your feed
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Fri Dec 22, 2017 2:32 pm

Thanks Ger
Ger wrote:
Fri Dec 22, 2017 2:13 pm
On a side note: I got some recommendations when validating your feed
I had seen those but as this is not my code and to be honest I do not really understand much about feeds :? - although I now know more than I did 24 hours ago :)
Ger wrote:
Fri Dec 22, 2017 2:13 pm
Can't you just keep things simple on your end?
I think that I may have found a way around the problem by replacing the <content:encoded> tags in the template from my blog with <description> tags.

On another side note - when I was searching for a solution I noticed that <content:encoded> is used in WordPress so it might be worth considering adding this before all the WP users start complaining :shock: :shock: :o
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Fri Dec 22, 2017 2:52 pm

Yeah, most of them use both the description as the content:encoded part.

However, I think you're right, I have to try and tackle this one.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Fri Dec 22, 2017 3:48 pm

david63 wrote:
Fri Dec 22, 2017 2:32 pm
poke
I think I've managed to handle the content:encoded style. It favours the description. Would you mind testing it for me?

I've also added a locking mechanism that sets an extra config value, so make sure you disable and enable the extension to run the migration.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Fri Dec 22, 2017 3:58 pm

Ger wrote:
Fri Dec 22, 2017 3:48 pm
I think I've managed to handle the content:encoded style. It favours the description. Would you mind testing it for me?
No problem - is it on GitHub? Might not be until tomorrow though.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Fri Dec 22, 2017 4:14 pm

Just downloaded 1.0.5 from Git and it works fine :D :D

Thanks.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Sat Dec 23, 2017 7:04 am

Great, thanks for testing!
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Sat Dec 23, 2017 8:07 am

I think I have found a couple of minor buglets.

I have three feeds set up and when I look at the Admin log after the feeds have been fetched the username of the first feed is that of the user who invoked the cron job (which is what I would expect), but the subsequent feed entries use the username of the user who is set in the ACP as the Feed user id.

The table of feeds and their options in the ACP is not responsive and can result in the table extending beyond the width of the page.

I notice that you have added a lock to this version. I tried this once and there is a possibility that it can cause problems if you have a situation where the job does not complete for some reason then you can end up with the system locked - I would suggest that you either have an "unlock" button or at the start of the cron job (or elsewhere) auto unlock before you start.

Now for some queries (or feature requests ;) )

1. Is it possible to shorten the title?
2. Is it possible when setting up a feed to only import them from a "start date"?
3. Would it be possible to post a feed to a topic instead of a forum?
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Sat Dec 23, 2017 2:44 pm

Another little bug.

When you are setting the feedpostbot_locked variable in driver.php you need to make it dynamic each time.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Sun Dec 24, 2017 1:10 pm

Thanks. I'll look into it after the holidays
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Wed Dec 27, 2017 7:56 am

david63 wrote:
Sat Dec 23, 2017 2:44 pm
Another little bug.

When you are setting the feedpostbot_locked variable in driver.php you need to make it dynamic each time.
When I look at https://github.com/phpbb/phpbb/blob/mas ... g.php#L124
it seems the third parameter isn't used at all?
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

User avatar
david63
Jr. Extension Validator
Posts: 15068
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [3.2][RC] Feed post bot

Post by david63 » Wed Dec 27, 2017 8:12 am

Ger wrote:
Wed Dec 27, 2017 7:56 am
david63 wrote:
Sat Dec 23, 2017 2:44 pm
Another little bug.

When you are setting the feedpostbot_locked variable in driver.php you need to make it dynamic each time.
When I look at https://github.com/phpbb/phpbb/blob/mas ... g.php#L124
it seems the third parameter isn't used at all?
But when you are changing it like this you do not want it to use the cached setting you need it to pick up the revised setting, which is why you need to define it as being dynamic
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

User avatar
Ger
Recognised Extension Developer
Posts: 1792
Joined: Wed Jan 02, 2008 7:35 pm
Location: 192.168.1.100
Contact:

Re: [3.2][RC] Feed post bot

Post by Ger » Wed Dec 27, 2017 9:23 am

david63 wrote:
Sat Dec 23, 2017 8:07 am
I have three feeds set up and when I look at the Admin log after the feeds have been fetched the username of the first feed is that of the user who invoked the cron job (which is what I would expect), but the subsequent feed entries use the username of the user who is set in the ACP as the Feed user id.
User is only switched when there are items to post, so the logging is a bit off due to that. However: in this case I don't think it really matters, does it?
The table of feeds and their options in the ACP is not responsive and can result in the table extending beyond the width of the page.
Yeah, that's a thingy indeed. The table was very simple in the beginning of this extension, but got wider along the months. ;)
From a simple list of urls, forumlist and a delete button it got to this. Not ideal, but I'm keeping it this way for the current version which I'm trying to get past validation. Redesigning the ACP part will probably lead to larger issues.
I notice that you have added a lock to this version. I tried this once and there is a possibility that it can cause problems if you have a situation where the job does not complete for some reason then you can end up with the system locked - I would suggest that you either have an "unlock" button or at the start of the cron job (or elsewhere) auto unlock before you start.
Yeah the lock is there to prevent a double process when running from cron and while that process runs, starting it manually from ACP again. It can lead to double postings. I can imagine it can lead to a permalock when the process interrupts. I think I'll add a button for unlocking when needed, thanks for mentioning.
Now for some queries (or feature requests ;) )

1. Is it possible to shorten the title?
2. Is it possible when setting up a feed to only import them from a "start date"?
3. Would it be possible to post a feed to a topic instead of a forum?
All good ideas, but I'll pass them to a second version. If you want to you can create your own addon extension to enforce 1 and 3 when hooking into ger.feedpostbot.submit_post_before.
My extensions:
Simple CMS, Feed post bot, Avatar Resize, Modbreak, Magic OGP, Live topic update, Modern Quote, Quoted Where (GDPR) and Autoresponder.
Newest: FAQ manager for 3.2

Like my work? Buy me a coffee to keep it coming. :ugeek:
-Available for custom work-

Locked

Return to “Extensions in Development”