[CDB] Smartfeed 3.0.4

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Suggested Hosts
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
Locked
User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Sat Dec 12, 2015 12:20 am

Praggle wrote:On my server I have no mcrypt installed and I don't have access to the php-configuration of this server. My board has almost no public accessable forums.
Is there no chance, to use smartfeed with those private areas?
Probably not. Are you sure you can't add a php.ini to your webroot folder and specify it there?

For the mod I used the PHP Secure Communications library. I tried to integrate it with the 3.1 architecture but it was beyond me, at least at that time. It's possible but unlikely that it will reappear in a future version.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
david63
Registered User
Posts: 16321
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Name: David Wood
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by david63 » Sat Dec 12, 2015 7:50 am

MarkDHamill wrote:I wasn't sure if anyone would use it with subsilver2 for 3.1 but I converted it anyhow.
Personally I would not concern myself with subsilver2 at this stage as it is discontinued in 3.2
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored

Praggle
Registered User
Posts: 64
Joined: Thu Feb 26, 2015 6:07 pm

Re: [RC1] Smartfeed 3.0.2

Post by Praggle » Sat Dec 12, 2015 9:26 pm

MarkDHamill wrote:Are you sure you can't add a php.ini to your webroot folder and specify it there?

For the mod I used the PHP Secure Communications library. I tried to integrate it with the 3.1 architecture but it was beyond me, at least at that time. It's possible but unlikely that it will reappear in a future version.
My phpbb-installation for a non-profit organisation is running on a freehoster account. I don't have any access to the php.ini and to the webroot. I think that would be on every freehosting webspace the same problem. And I have been adviced to keep this board free of charge.

Mark, a bigger part of my users isn't getting in touch with the running dicussions on my board since my update to phpbb3.1. They don't have the digest functionality anymore and they are not happy with making individual abonnements of every forum. So they were looking very much foward to get me running 'smartfeed' on this board. If you see a possibility for a workaround you would make 489 members of this board very happy and me too.

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Sat Dec 12, 2015 10:35 pm

david63 wrote:
MarkDHamill wrote:I wasn't sure if anyone would use it with subsilver2 for 3.1 but I converted it anyhow.
Personally I would not concern myself with subsilver2 at this stage as it is discontinued in 3.2
Someone did port it to 3.1 and it was approved. It is still using tables for styling so I can see why it would be disapproved. I knew if I didn't add it someone would complain since it was in the mod.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Sat Dec 12, 2015 10:40 pm

Praggle wrote:My phpbb-installation for a non-profit organisation is running on a freehoster account. I don't have any access to the php.ini and to the webroot. I think that would be on every freehosting webspace the same problem. And I have been adviced to keep this board free of charge.

Mark, a bigger part of my users isn't getting in touch with the running dicussions on my board since my update to phpbb3.1. They don't have the digest functionality anymore and they are not happy with making individual abonnements of every forum. So they were looking very much foward to get me running 'smartfeed' on this board. If you see a possibility for a workaround you would make 489 members of this board very happy and me too.
Sorry to hear about the php.ini issue. If I get enough complaints it will raise in priority. It was a real hassle to integrate the PHP Secure Communications Library for the mod, suggested by the review team, but in general the review team is hostile to non-phpBB libraries. External libraries rarely meet phpBB's standards for code quality. For most people they should be able to enable it in a php.ini file if it was compiled and linked into PHP on their host.

On the digests issue, it's hot in development. It's not germane to this topic but I am 1-2 months from having a development release for review. Smartfeed is the easier mod to port. You can follow progress for now on my blog.

https://phpbbservices.com/blog/

I'll put out a new topic here when the development version is ready.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

Praggle
Registered User
Posts: 64
Joined: Thu Feb 26, 2015 6:07 pm

Re: [RC1] Smartfeed 3.0.2

Post by Praggle » Sun Dec 13, 2015 10:35 am

Okay, I can understand the difficulties and I'm glad to hear, that you are working on the digest-extension.

Thank you

User avatar
Sajaki
Registered User
Posts: 1354
Joined: Mon Mar 02, 2009 1:41 pm
Name: Andreas
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by Sajaki » Sun Jan 03, 2016 12:00 am

hi Mark,
on line 30 & 97 of \phpbbservices\smartfeed\acp\main_module.php, you're still using request_var
$action = request_var('action', '');
$cfg_array = (isset($_REQUEST['config'])) ? utf8_normalize_nfc(request_var('config', array('' => ''), true)) : $this->new_config;

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Sun Jan 03, 2016 2:45 am

This should be okay. $request->variable is preferred and request_var is still scattered all over the base phpBB code. But it's easy enough to change. In fact I just copied some code from a base ACP module and modified that.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

michael.s
Registered User
Posts: 13
Joined: Sun Jan 24, 2016 9:40 am

Re: [RC1] Smartfeed 3.0.2

Post by michael.s » Sun Jan 24, 2016 10:04 am

Hi Mark,

great work so far. Thank you for your effort. We are running a phpBB 3.0 board for > 5 years using your Smartfeed MOD. Quite some of our users rely on it. Now we want to move to 3.1 and set up a test environment for that.

Testing Smartfeed 3.0.2 RC 1 I am unable to get it running. Our forum has nearly 20.000 posts in > 1000 threads, > 3000 users. We are using a copy of the original database for testing. Mysql 5.5.46, PHP 5.5.9, Apache 2.4.7.

The issue: When generating a feed URL and testing it, I get (after quite some time) an SQL error "SQL ERROR [ mysqli ] - Incorrect key file for table '/var/tmp/#sql_3a4_0.MYI'; try to repair it [126]". After some research I found that mysql is generating huge temporary files - and my harddrive is running out of space. So I increased the space available, but it still does not help. 30 GB of space are eaten up by temp files before SQL throws the error.

PHPmyadmin shows that this SQL query is causing the problem:

Code: Select all

SELECT f . * , t . * , p . * , u . * , tt.mark_time AS topic_mark_time, ft.mark_time AS forum_mark_time
FROM (
phpbb_forums f
CROSS JOIN phpbb_topics t
CROSS JOIN phpbb_posts p
CROSS JOIN phpbb_users u
)
LEFT JOIN phpbb_topics_track tt ON ( t.topic_id = tt.topic_id )
LEFT JOIN phpbb_forums_track ft ON ( f.forum_id = ft.forum_id )
WHERE f.forum_id = t.forum_id
AND t.topic_id = p.topic_id
AND p.poster_id = u.user_id
AND p.post_time >1451035402
AND p.forum_id
IN ( 9, '17' )
AND p.post_visibility =1
ORDER BY t.topic_last_post_time DESC , p.post_time
LIMIT 18446744073709551615
Searching the net I found hints that the ORDER BY parameter can cause mysql to behave like mentioned when trying to order big sized data.

By the way: No matter how low I set parameters like max. number of posts in feed or time limit: As soon as only one single post would be included in the feed, I run into the problem.

Would be a great help if you could check how to solve the issue.

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Mon Jan 25, 2016 1:14 am

One of the problems with using DBAL is it becomes hard to figure out if the join logic is working correctly as it will throw in CROSS JOINs whether you like it or not. I'll look at the SQL more closely to see if a Cartesian product is occurring. That could certainly slow things down. There may be another table join or two that is needed.

Note that there are performance settings for Smartfeed in the ACP to limit the maximum number of items in a feed.

It sounds like a table is corrupt somewhere if a table needs repairing. It's a pretty simple matter to repair a table using phpMyAdmin if you have that. Even better, ditch the MyISAM storage engine. It's no longer the default for phpBB. InnoDB is preferred. To change your storage engine see my blog entry:

https://phpbbservices.com/2015/12/02/ch ... age-engine
Last edited by MarkDHamill on Mon Jan 25, 2016 3:31 am, edited 1 time in total.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Mon Jan 25, 2016 2:23 am

Try changing lines 914-923 of /ext/phpbbservices/smartfeed/controller/main.php to:

Code: Select all

				$sql_array['LEFT_JOIN'] = array(
					array(
						'FROM'	=> array(TOPICS_TRACK_TABLE => 'tt'),
						'ON'	=> 't.topic_id = tt.topic_id AND tt.user_id = u.user_id'
					),
					array(
						'FROM'	=> array(FORUMS_TRACK_TABLE => 'ft'),
						'ON'	=> 'f.forum_id = ft.forum_id AND ft.user_id = u.user_id'
					)
				);
Looks like I needed additional joints for the forums track and topics track tables to connect it with the users table. This should solve the Cartesian product problem.

It's hard for me to test all permutations so I appreciate the "real world" test. Please let me know if it works.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

michael.s
Registered User
Posts: 13
Joined: Sun Jan 24, 2016 9:40 am

Re: [RC1] Smartfeed 3.0.2

Post by michael.s » Mon Jan 25, 2016 12:54 pm

Thank you for the quick reply. Before posting here, I had already tried repairing and analyzing the DB tables. There seemed to be no issues. I also had tested to restrict the maximum number of post in the feed as you suggest. The result was as I described - either an empty feed (no posts fitting my parameters) or mysql showing the mentioned behaviour as soon as at least one post is covered by the feed criteria.

Converting to InnoDB did not change anything.

Replacing the code lines 914-923 to your new code helped. No more SQL errors. Feed is displayed within seconds, short feeds < 1 second. But when choosing ATOM 1.0 the feed is always empty (Firefox, did not test with other feed readers yet). RSS 1.0 and 2.0 are working fine. (btw - before modifying the code, I tested all three feed types with the same sql error results, so this did not seem to be part of my original problem).

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Mon Jan 25, 2016 5:31 pm

The ATOM feed may be there but not displayable. If you look at it in a browser, do a view source. Do you see items in the code or any markup?

I am glad those extra joins solved the problem. I'm surprised I missed it.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

michael.s
Registered User
Posts: 13
Joined: Sun Jan 24, 2016 9:40 am

Re: [RC1] Smartfeed 3.0.2

Post by michael.s » Tue Jan 26, 2016 11:07 am

MarkDHamill wrote:The ATOM feed may be there but not displayable. If you look at it in a browser, do a view source. Do you see items in the code or any markup?
Good point. The feed is there, but not displayed in Firefox. So as of now, everything seems to be running fine. Thanks again for your effort!

User avatar
MarkDHamill
Registered User
Posts: 3855
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC1] Smartfeed 3.0.2

Post by MarkDHamill » Tue Jan 26, 2016 2:40 pm

Thank you for testing and feedback!
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

Locked

Return to “Extensions in Development”