Page 5 of 8

Re: [RC1] Smartfeed 3.0.2

Posted: Sat Dec 12, 2015 12:20 am
by MarkDHamill
Praggle wrote:On my server I have no mcrypt installed and I don't have access to the php-configuration of this server. My board has almost no public accessable forums.
Is there no chance, to use smartfeed with those private areas?
Probably not. Are you sure you can't add a php.ini to your webroot folder and specify it there?

For the mod I used the PHP Secure Communications library. I tried to integrate it with the 3.1 architecture but it was beyond me, at least at that time. It's possible but unlikely that it will reappear in a future version.

Re: [RC1] Smartfeed 3.0.2

Posted: Sat Dec 12, 2015 7:50 am
by david63
MarkDHamill wrote:I wasn't sure if anyone would use it with subsilver2 for 3.1 but I converted it anyhow.
Personally I would not concern myself with subsilver2 at this stage as it is discontinued in 3.2

Re: [RC1] Smartfeed 3.0.2

Posted: Sat Dec 12, 2015 9:26 pm
by Praggle
MarkDHamill wrote:Are you sure you can't add a php.ini to your webroot folder and specify it there?

For the mod I used the PHP Secure Communications library. I tried to integrate it with the 3.1 architecture but it was beyond me, at least at that time. It's possible but unlikely that it will reappear in a future version.
My phpbb-installation for a non-profit organisation is running on a freehoster account. I don't have any access to the php.ini and to the webroot. I think that would be on every freehosting webspace the same problem. And I have been adviced to keep this board free of charge.

Mark, a bigger part of my users isn't getting in touch with the running dicussions on my board since my update to phpbb3.1. They don't have the digest functionality anymore and they are not happy with making individual abonnements of every forum. So they were looking very much foward to get me running 'smartfeed' on this board. If you see a possibility for a workaround you would make 489 members of this board very happy and me too.

Re: [RC1] Smartfeed 3.0.2

Posted: Sat Dec 12, 2015 10:35 pm
by MarkDHamill
david63 wrote:
MarkDHamill wrote:I wasn't sure if anyone would use it with subsilver2 for 3.1 but I converted it anyhow.
Personally I would not concern myself with subsilver2 at this stage as it is discontinued in 3.2
Someone did port it to 3.1 and it was approved. It is still using tables for styling so I can see why it would be disapproved. I knew if I didn't add it someone would complain since it was in the mod.

Re: [RC1] Smartfeed 3.0.2

Posted: Sat Dec 12, 2015 10:40 pm
by MarkDHamill
Praggle wrote:My phpbb-installation for a non-profit organisation is running on a freehoster account. I don't have any access to the php.ini and to the webroot. I think that would be on every freehosting webspace the same problem. And I have been adviced to keep this board free of charge.

Mark, a bigger part of my users isn't getting in touch with the running dicussions on my board since my update to phpbb3.1. They don't have the digest functionality anymore and they are not happy with making individual abonnements of every forum. So they were looking very much foward to get me running 'smartfeed' on this board. If you see a possibility for a workaround you would make 489 members of this board very happy and me too.
Sorry to hear about the php.ini issue. If I get enough complaints it will raise in priority. It was a real hassle to integrate the PHP Secure Communications Library for the mod, suggested by the review team, but in general the review team is hostile to non-phpBB libraries. External libraries rarely meet phpBB's standards for code quality. For most people they should be able to enable it in a php.ini file if it was compiled and linked into PHP on their host.

On the digests issue, it's hot in development. It's not germane to this topic but I am 1-2 months from having a development release for review. Smartfeed is the easier mod to port. You can follow progress for now on my blog.

https://phpbbservices.com/blog/

I'll put out a new topic here when the development version is ready.

Re: [RC1] Smartfeed 3.0.2

Posted: Sun Dec 13, 2015 10:35 am
by Praggle
Okay, I can understand the difficulties and I'm glad to hear, that you are working on the digest-extension.

Thank you

Re: [RC1] Smartfeed 3.0.2

Posted: Sun Jan 03, 2016 12:00 am
by Sajaki
hi Mark,
on line 30 & 97 of \phpbbservices\smartfeed\acp\main_module.php, you're still using request_var
$action = request_var('action', '');
$cfg_array = (isset($_REQUEST['config'])) ? utf8_normalize_nfc(request_var('config', array('' => ''), true)) : $this->new_config;

Re: [RC1] Smartfeed 3.0.2

Posted: Sun Jan 03, 2016 2:45 am
by MarkDHamill
This should be okay. $request->variable is preferred and request_var is still scattered all over the base phpBB code. But it's easy enough to change. In fact I just copied some code from a base ACP module and modified that.

Re: [RC1] Smartfeed 3.0.2

Posted: Sun Jan 24, 2016 10:04 am
by michael.s
Hi Mark,

great work so far. Thank you for your effort. We are running a phpBB 3.0 board for > 5 years using your Smartfeed MOD. Quite some of our users rely on it. Now we want to move to 3.1 and set up a test environment for that.

Testing Smartfeed 3.0.2 RC 1 I am unable to get it running. Our forum has nearly 20.000 posts in > 1000 threads, > 3000 users. We are using a copy of the original database for testing. Mysql 5.5.46, PHP 5.5.9, Apache 2.4.7.

The issue: When generating a feed URL and testing it, I get (after quite some time) an SQL error "SQL ERROR [ mysqli ] - Incorrect key file for table '/var/tmp/#sql_3a4_0.MYI'; try to repair it [126]". After some research I found that mysql is generating huge temporary files - and my harddrive is running out of space. So I increased the space available, but it still does not help. 30 GB of space are eaten up by temp files before SQL throws the error.

PHPmyadmin shows that this SQL query is causing the problem:

Code: Select all

SELECT f . * , t . * , p . * , u . * , tt.mark_time AS topic_mark_time, ft.mark_time AS forum_mark_time
FROM (
phpbb_forums f
CROSS JOIN phpbb_topics t
CROSS JOIN phpbb_posts p
CROSS JOIN phpbb_users u
)
LEFT JOIN phpbb_topics_track tt ON ( t.topic_id = tt.topic_id )
LEFT JOIN phpbb_forums_track ft ON ( f.forum_id = ft.forum_id )
WHERE f.forum_id = t.forum_id
AND t.topic_id = p.topic_id
AND p.poster_id = u.user_id
AND p.post_time >1451035402
AND p.forum_id
IN ( 9, '17' )
AND p.post_visibility =1
ORDER BY t.topic_last_post_time DESC , p.post_time
LIMIT 18446744073709551615
Searching the net I found hints that the ORDER BY parameter can cause mysql to behave like mentioned when trying to order big sized data.

By the way: No matter how low I set parameters like max. number of posts in feed or time limit: As soon as only one single post would be included in the feed, I run into the problem.

Would be a great help if you could check how to solve the issue.

Re: [RC1] Smartfeed 3.0.2

Posted: Mon Jan 25, 2016 1:14 am
by MarkDHamill
One of the problems with using DBAL is it becomes hard to figure out if the join logic is working correctly as it will throw in CROSS JOINs whether you like it or not. I'll look at the SQL more closely to see if a Cartesian product is occurring. That could certainly slow things down. There may be another table join or two that is needed.

Note that there are performance settings for Smartfeed in the ACP to limit the maximum number of items in a feed.

It sounds like a table is corrupt somewhere if a table needs repairing. It's a pretty simple matter to repair a table using phpMyAdmin if you have that. Even better, ditch the MyISAM storage engine. It's no longer the default for phpBB. InnoDB is preferred. To change your storage engine see my blog entry:

https://phpbbservices.com/2015/12/02/ch ... age-engine

Re: [RC1] Smartfeed 3.0.2

Posted: Mon Jan 25, 2016 2:23 am
by MarkDHamill
Try changing lines 914-923 of /ext/phpbbservices/smartfeed/controller/main.php to:

Code: Select all

				$sql_array['LEFT_JOIN'] = array(
					array(
						'FROM'	=> array(TOPICS_TRACK_TABLE => 'tt'),
						'ON'	=> 't.topic_id = tt.topic_id AND tt.user_id = u.user_id'
					),
					array(
						'FROM'	=> array(FORUMS_TRACK_TABLE => 'ft'),
						'ON'	=> 'f.forum_id = ft.forum_id AND ft.user_id = u.user_id'
					)
				);
Looks like I needed additional joints for the forums track and topics track tables to connect it with the users table. This should solve the Cartesian product problem.

It's hard for me to test all permutations so I appreciate the "real world" test. Please let me know if it works.

Re: [RC1] Smartfeed 3.0.2

Posted: Mon Jan 25, 2016 12:54 pm
by michael.s
Thank you for the quick reply. Before posting here, I had already tried repairing and analyzing the DB tables. There seemed to be no issues. I also had tested to restrict the maximum number of post in the feed as you suggest. The result was as I described - either an empty feed (no posts fitting my parameters) or mysql showing the mentioned behaviour as soon as at least one post is covered by the feed criteria.

Converting to InnoDB did not change anything.

Replacing the code lines 914-923 to your new code helped. No more SQL errors. Feed is displayed within seconds, short feeds < 1 second. But when choosing ATOM 1.0 the feed is always empty (Firefox, did not test with other feed readers yet). RSS 1.0 and 2.0 are working fine. (btw - before modifying the code, I tested all three feed types with the same sql error results, so this did not seem to be part of my original problem).

Re: [RC1] Smartfeed 3.0.2

Posted: Mon Jan 25, 2016 5:31 pm
by MarkDHamill
The ATOM feed may be there but not displayable. If you look at it in a browser, do a view source. Do you see items in the code or any markup?

I am glad those extra joins solved the problem. I'm surprised I missed it.

Re: [RC1] Smartfeed 3.0.2

Posted: Tue Jan 26, 2016 11:07 am
by michael.s
MarkDHamill wrote:The ATOM feed may be there but not displayable. If you look at it in a browser, do a view source. Do you see items in the code or any markup?
Good point. The feed is there, but not displayed in Firefox. So as of now, everything seems to be running fine. Thanks again for your effort!

Re: [RC1] Smartfeed 3.0.2

Posted: Tue Jan 26, 2016 2:40 pm
by MarkDHamill
Thank you for testing and feedback!