[2.0.x] Tweaks for large forums

The 2.0.x discussion forum has been locked; this will remain read-only. The 3.0.x discussion forum has been renamed phpBB Discussion.
Acecool
Registered User
Posts: 1013
Joined: Sat Jul 13, 2002 4:51 am
Location: Behind my computer
Contact:

Post by Acecool » Sat Sep 27, 2003 5:06 pm

One word: wow...


Your board is amazing :-) and please share the code for search table thing :-)
Visit Acecoolco.com :: Image

If you plan on contacting me, please read this: Legal Terms & Conditions

wep333
Registered User
Posts: 7
Joined: Tue May 27, 2003 3:29 pm

Post by wep333 » Sat Sep 27, 2003 5:24 pm

WHERE main_type = 'c'


should be removed from the query in your first tweak for index.php.


probably a non standard attribue you added.


Apart from that, this tweak indeed results in a speed increase on my board ( >> 1M posts )

thanks !

wep333
Registered User
Posts: 7
Joined: Tue May 27, 2003 3:29 pm

Post by wep333 » Sat Sep 27, 2003 5:28 pm

oh well apart from that,

I completely disabled seach on my forums. Without doing that, the forums lags like s*it.
As I am too lazy to rewrite the code of "add_searchword" function in PERL. It would really be nice to have a perl script taking your entire board and populating searchwords table from scratch.

Most other tasks are scripted in perl ( inactive user flush, forum pruning, etc.) scheduled at offpeak hours.

ra[g]e
Registered User
Posts: 21
Joined: Wed May 14, 2003 9:10 am

Post by ra[g]e » Sun Sep 28, 2003 10:49 am

lanzer wrote: I've taken a look at the queries in 2.2 and they're much like 2.0.6, where the user and post text info are joined when searching for posts to display.


I think that is sad... it'll be great if phpBB forums could be engineered for a large and active forum... applying Lanzer's tweaks to phpBB 2.2 would be a start, one ignorant of php and mySQL tweaking wouldn't be forced to turn to vBulletin when their forums get too big.

lanzer
Registered User
Posts: 152
Joined: Wed Oct 10, 2001 10:00 am
Contact:

Post by lanzer » Sun Sep 28, 2003 1:24 pm

wep333 wrote:
WHERE main_type = 'c'


should be removed from the query in your first tweak for index.php.


Image

Sorry, that was there for a sub category hack that I slipped in.

One note about the sub-category hack. The one that's available right now is a terrific hack. It's extremely featureful with tons of customization options. Though it's not made for medium or large forums due to a few queries in there that are really big. It needs about 5 major changes until it can be placed in production environment.

lanzer
Registered User
Posts: 152
Joined: Wed Oct 10, 2001 10:00 am
Contact:

Post by lanzer » Sun Sep 28, 2003 2:02 pm

I'm very glad that there are people who benefitted from the tweaks. :D I was wondering for a moment if anyone would read this, so I didn't add any more codes/tweaks.

Anyways, here's the class that I snipped off the phpbb 2.2.0 CVS file. I'm glad that it's nice and portable now so we can try out different people approach in solving search table problems. Anyone can slip this code at the end of the functions_search.php file (before the "?>" line) and use it on version 2.0.x. But you must alter the database to have it working properly. The necessary changes are mentioned below.

I proofed read this once. Hope there isn't any of my own variables in there.

Code: Select all

// Parses a given message and updates/maintains the fulltext tables
class fulltext_search
{
	function split_search_words(&$text)
	{
		global $user, $phpbb_root_path;

        // This is where I specified the min and max for words to be added to the wordsearch table.
        // You can actually make these entries in the board_config table and make $board_config global
        $config = array('min_search_chars' => 3, 'max_search_chars' => 15);

		static $drop_char_match, $drop_char_replace, $stopwords, $synonyms;

		if (empty($drop_char_match))
		{
			$drop_char_match =   array('^', '$', '&', '(', ')', '<', '>', '`', '\'', '"', '|', ',', '@', '_', '?', '%', '-', '~', '+', '.', '[', ']', '{', '}', ':', '\\', '/', '=', '#', '\'', ';', '!', '*');
			$drop_char_replace = array(' ', ' ', ' ', ' ', ' ', ' ', ' ', '',  '',   ' ', ' ', ' ', ' ', '',  ' ', ' ', '',  ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', '' ,  ' ', ' ', ' ', ' ',  ' ', ' ', ' ');
            // If you're not running an english board, change the following lines to match the language!
			$stopwords = @file($phpbb_root_path.'language/lang_english/search_stopwords.txt');
			$synonyms = @file($phpbb_root_path.'language/lang_english/search_synonyms.txt');
		}

		$match = array();
		// New lines, carriage returns
		$match[] = "#[\n\r]+#";
		// NCRs like & etc.
		$match[] = '#&[\#a-z0-9]+?;#i';
		// URL's
		$match[] = '#\b[\w]+:\/\/[a-z0-9\.\-]+(\/[a-z0-9\?\.%_\-\+=&\/]+)?#';
		// quoted text
		$match[] = '#\[quote:[a-z0-9]{10,}(=.*?)?\].*?\[\/quote:[a-z0-9]{10,}\]#';
        // BBcode
		$match[] = '#\[img:[a-z0-9]{10,}\].*?\[\/img:[a-z0-9]{10,}\]#';
		$match[] = '#\[\/?url(=.*?)?\]#';
		$match[] = '#\[\/?[a-z\*=\+\-]+(\:?[0-9a-z]+)?:[a-z0-9]{10,}(\:[a-z0-9]+)?=?.*?\]#';
		$match[] = '#\b([a-z0-9]{1,' . $config['min_search_chars'] . '}|[a-z0-9]{' . $config['max_search_chars'] . ',})\b#is';

		$text = preg_replace($match, ' ', ' ' . strtolower($text) . ' ');

		// Filter out non-alphabetical chars
		$text = str_replace($drop_char_match, $drop_char_replace, $text);

		if (!empty($stopwords))
		{
            $stopword_list = array();
            foreach ($stopwords as $st) $stopword_list[] = trim($st);
		}

		if (!empty($synonyms))
		{
			foreach ($synonyms as $temp_line)
			{
				list($replace_synonym, $match_synonym) = preg_split('/[\t\s]+/', trim(strtolower($temp_line)));
					$text =  preg_replace('#\b' . $match_synonym . '\b#', ' ' . $replace_synonym . ' ', $text);
			}
		}

		preg_match_all('#\b([\w]+)\b#', $text, $split_entries);

        return array_diff(array_unique($split_entries[1]), $stopword_list);
	}

	function add(&$mode, &$post_id, &$message, &$subject)
	{
		global $board_config, $db;

		$split_text = $this->split_search_words($message);
		$split_title = ($subject) ? $this->split_search_words($subject) : array();
        // Catalog first 80 words only.  Feel free to change 80 to a larger value if you want to index more words
        if (sizeof($split_text) > 80) {
            $split_text = array_splice($split_text,0,80);
        }
		$words = array();
		if ($mode == 'edit' || $mode == 'editpost')
		{
			$sql = 'SELECT w.word_id, w.word_text, m.title_match
				FROM ' . SEARCH_WORD_TABLE . ' w, ' . SEARCH_MATCH_TABLE . " m
				WHERE m.post_id = $post_id 
					AND w.word_id = m.word_id";
			$result = $db->sql_query($sql);

			$cur_words = array();
			while ($row = $db->sql_fetchrow($result))
			{
				$which = ($row['title_match']) ? 'title' : 'post';
				$cur_words[$which][$row['word_text']] = $row['word_id'];
			}
			$db->sql_freeresult($result);

            if ( count($cur_words) ) {
    			$words['add']['post'] = array_diff($split_text, array_keys($cur_words['post']));
    			$words['add']['title'] = array_diff($split_title, array_keys($cur_words['title']));
    			$words['del']['post'] = array_diff(array_keys($cur_words['post']), $split_text);
    			$words['del']['title'] = array_diff(array_keys($cur_words['title']), $split_title);
            } else {
                $words['add']['post'] = $split_text;
                $words['add']['title'] = $split_title;
                $words['del']['post'] = array();
                $words['del']['title'] = array();
            }
		}
		else
		{
			$words['add']['post'] = $split_text;
			$words['add']['title'] = $split_title;
			$words['del']['post'] = array();
			$words['del']['title'] = array();
		}
		unset($split_text);
		unset($split_title);
		// Get unique words from the above arrays
		$unique_add_words = array_unique(array_merge($words['add']['post'], $words['add']['title']));

		// We now have unique arrays of all words to be added and removed and
		// individual arrays of added and removed words for text and title. What
		// we need to do now is add the new words (if they don't already exist)
		// and then add (or remove) matches between the words and this post
		if (sizeof($unique_add_words) >= 20)
		{
            foreach ($unique_add_words as $pickword) {
                if (! preg_match('/[0-9]/',$pickword) ) {
                    if (! (substr_count($pickword,substr($pickword,0,1)) > 4 || substr_count($pickword,substr($pickword,1,1)) > 4)) $new_unique_add_words[] = $pickword;
                }
            }
            if (sizeof($new_unique_add_words)) {
                $unique_add_words = $new_unique_add_words;
            }
			$sql = 'SELECT word_id, word_text FROM ' . SEARCH_WORD_TABLE . ' WHERE word_text IN (' . implode(', ', preg_replace('#^(.*)$#', '\'\1\'', $unique_add_words)) . ")";
			$result = $db->sql_query($sql);

			$word_ids = array();
			while ($row = $db->sql_fetchrow($result))
			{
				$word_ids[$row['word_text']] = $row['word_id'];
			}
			$db->sql_freeresult($result);

			$new_words = array_diff($unique_add_words, array_keys($word_ids));
			unset($unique_add_words);

			if (sizeof($new_words))
			{
				switch (SQL_LAYER)
				{
					case 'mysql':
					case 'mysql4':
						$sql = 'INSERT INTO ' . SEARCH_WORD_TABLE . " (word_text, post_id) VALUES ('" . implode("',$post_id),('", $new_words ) ."',$post_id)";
						$db->sql_query($sql);
						break;

					case 'mssql':
					case 'sqlite':
						$sql = 'INSERT INTO ' . SEARCH_WORD_TABLE . ' (word_text) ' . implode(' UNION ALL ', preg_replace('#^(.*)$#', "SELECT '\\1'",  $new_words));
						$db->sql_query($sql);
						break;

					default:
						foreach ($new_words as $word)
						{
							$sql = 'INSERT INTO ' . SEARCH_WORD_TABLE . " (word_text) VALUES ('$word')";
							$db->sql_query($sql);
						}
						break;
				}
			}
			unset($new_words);

            if (sizeof($word_ids) && $post_id) {
                $sql_text = implode(",",$word_ids);
                $sql = " UPDATE " . SEARCH_WORD_TABLE . " SET word_common = word_common + 1 , post_id = " . $post_id . " WHERE word_id IN (" . $sql_text . ")";
                $db->sql_query($sql);
            }
		}


		foreach ($words['del'] as $word_in => $word_ary)
		{
			$title_match = ($word_in == 'title') ? 1 : 0;

			if (sizeof($word_ary))
			{
				$sql_in = array();
				foreach ($word_ary as $word)
				{
					$sql_in[] = $cur_words[$word_in][$word];
				}

				$sql = 'DELETE FROM ' . SEARCH_MATCH_TABLE . ' 
					WHERE word_id IN (' . implode(', ', $sql_in) . ') 
						AND post_id = ' . intval($post_id) . " 
						AND title_match = $title_match";
				$db->sql_query($sql);
				unset($sql_in);
			}
		}

		foreach ($words['add'] as $word_in => $word_ary)
		{
			$title_match = ($word_in == 'title') ? 1 : 0;

			if (sizeof($word_ary))
			{
				$sql = 'INSERT INTO ' . SEARCH_MATCH_TABLE . " (post_id, word_id, title_match) 
					SELECT $post_id, word_id, $title_match 
					FROM " . SEARCH_WORD_TABLE . ' 
					WHERE word_text IN (' . implode(', ', preg_replace('#^(.*)$#', '\'\1\'', $word_ary)) . ')';
				$db->sql_query($sql);
			}
		}

		unset($words);

        return true;
	}

}
The second change is on the functions_post.php file.

First comment out the following line:

Code: Select all

        add_search_words('single', $post_id, stripslashes($post_message), stripslashes($post_subject));
And replace with:

Code: Select all

        $search = new fulltext_search();
        $result = $search->add($mode, $post_id, stripslashes($post_message), stripslashes($post_subject));
Though instead of just two lines above, what I have instead is something to filter out even more posts to be added to the word search table:

Code: Select all

    $start_search = TRUE;
    if ($mode == 'newtopic') {
        // Note that I'm skipping wordmatch on the chatter and the test forums
        // if ($forum_id == 23 || $forum_id == 8 || $forum_id == 39) $start_search = FALSE;

        // I personally don't like topics with only one word as the subject since they're usually spam
        $post_subject = trim($post_subject);
        if (! preg_match("/ /",$post_subject)) $start_search = FALSE;
    } else {
        $sql2 = "SELECT topic_replies, topic_status, topic_title, forum_id  FROM " . TOPICS_TABLE . " WHERE topic_id = " . $topic_id;
        $result2 = $db->sql_query($sql2);
        list($topic_replies, $topic_status, $topic_title, $fid) = $db->sql_fetchrow($result2);
        // $fid = forum_id for this reply
        // Note that I'm skipping wordmatch on the chatter and the test forums 
        // if ($fid == 23 || $fid == 8 || $fid == 39) $start_search = FALSE;

        // Here I stop indexing posts when the reply starts to span into the second page.  It's here because we have a lot of huge role playing threads.
        // if ($topic_replies > 20) $start_search = FALSE;
        
        // I personally don't like topics with only one word as the subject since they're usually spam
        if (! preg_match("/ /",$topic_title)) $start_search = FALSE;

    }
    
    if ($start_search == TRUE) {
        $search = new fulltext_search();
        $result = $search->add($mode, $post_id, stripslashes($post_message), stripslashes($post_subject));
    } 
Lastly, you need to pop open MySQL and type in the following:

Code: Select all

ALTER TABLE phpbb_search_wordlist ADD post_id MEDIUMINT UNSIGNED not null;
ALTER TABLE phpbb_search_wordlist CHANGE word_common word_common MEDIUMINT (8) UNSIGNED DEFAULT '0' not null;
If you're starting new, you can just use this instead:

Code: Select all

CREATE TABLE phpbb_search_wordlist (
   word_text varchar(15) binary NOT NULL,
   word_id mediumint(8) unsigned NOT NULL auto_increment,
   word_common mediumint(8) unsigned DEFAULT '0' NOT NULL,
   post_id mediumint(8) unsigned DEFAULT '0' NOT NULL,
   PRIMARY KEY (word_text),
   KEY word_id (word_id)
);
Since I would only be pruning weekly, I'd rather save the RAM by not specifying post_id as an index.

I actually haven't written anything to prune the searchword tables yet. I'll post my purning scripts once I have them installed and tested.

PS - It's been about two weeks and I currently have a wordmatch table of about 1M entries, and a wordlist table with 40,000 words. Considering that we have about 120,000 messages a day things are doing okay.
Last edited by lanzer on Wed Oct 15, 2003 10:56 pm, edited 1 time in total.

testit
Registered User
Posts: 34
Joined: Sun Jan 06, 2002 6:58 pm

Why is this search not successfull in your board?

Post by testit » Sun Sep 28, 2003 5:15 pm

Hi Lanzer,

first of all thx a lot for your very interesting statements related to performance issues.

I visited your site and performed a search for "phew banners" with checked option "search for all terms".

Although thoses words exits here:
http://ian.go-gaia.com/forum/viewtopic.php?t=57873

The search is not successfull!

Any idea?

Regards
Volker

lanzer
Registered User
Posts: 152
Joined: Wed Oct 10, 2001 10:00 am
Contact:

Re: Why is this search not successfull in your board?

Post by lanzer » Tue Sep 30, 2003 12:22 am

Oh, the reason the search didn't work for you was due to two things:

- Note that I skipped forum # 8, 23, and 39. Thoser are the test forum, friends chatter (hidden, to be released feature), and chatterbox forums. The post you wanted to find was in the chatterbox. Sorry. ^^;
- Secondly, the search feature was only implemented a few days ago. The thread you were trying to search was made 2 months ago, so it wouldn't be in the search table. I haven't ran any scripts to add old posts into the search table yet. Gonna do that later.

Otherwise, please feel free to try the search on more recent posts. ^^
testit wrote: Hi Lanzer,

first of all thx a lot for your very interesting statements related to performance issues.

I visited your site and performed a search for "phew banners" with checked option "search for all terms".

Although thoses words exits here:
http://ian.go-gaia.com/forum/viewtopic.php?t=57873

The search is not successfull!

Any idea?

Regards
Volker

wep333
Registered User
Posts: 7
Joined: Tue May 27, 2003 3:29 pm

Post by wep333 » Sun Oct 05, 2003 3:08 pm

Hello Again Lanzer

Again a comment on your tweak,

It seems the code you pasted uses the column "word_common" completely differently than in 2.0.X versions.

Code: Select all

$sql = " UPDATE " . SEARCH_WORD_TABLE . " SET word_common = word_common + 1 , post_id = " . $post_id . " WHERE word_id IN (" . $sql_text . ")"; 
now a counter of occurences, instead of 1/0 (wether or not the words is common.)

My comment ; you have to slightly change search.php ( word_common <> 1 ) in order to avoid some weirdness.

PS
The 2.2 code seems indeed well written, and skips much more crap than 2.0.X, starting with quoted text for example.

PPS
can I have your email addy please ?

User avatar
Remix_88
Registered User
Posts: 46
Joined: Wed Apr 23, 2003 12:52 pm
Location: Hampshire, UK

Thanks for the info

Post by Remix_88 » Thu Oct 09, 2003 4:16 pm

lanzer wrote: Sorry, that was there for a sub category hack that I slipped in.

One note about the sub-category hack. The one that's available right now is a terrific hack. It's extremely featureful with tons of customization options. Though it's not made for medium or large forums due to a few queries in there that are really big. It needs about 5 major changes until it can be placed in production environment.


Thanks for all the info regarding the performance tweaks you have added to you site, it makes interesting reading and I am planning to add some of your tweaks to my site next week (not that it is anywhere near the size of yours :-))

You mentioned the sub-category mod in an earlier post and indicated that you have done some work to improve it's performance too. Is the sub-category mod you are referring too Categories hierarchy v1.1.0...

http://www.phpbb.com/phpBB/viewtopic.php?t=87278

If so, any chance you could share you tweaks for it?
Regards, Remix_88.

lanzer
Registered User
Posts: 152
Joined: Wed Oct 10, 2001 10:00 am
Contact:

Post by lanzer » Fri Oct 10, 2003 6:03 am

Hello Wep, thank you for pointing out the deal with the word_common and search.php. With the hack I have it'll end up omitting a lot of normal words. By default the board tries to figure out what words ends up showing on 40% of the postings and mark them as common. Since my algorithm omits quite a lot of postings that method cannot be used. I took a peak at my recent search word table and I have this:

Code: Select all

word_text	word_common
 stuff 	 1635 
 always 	 1644 
 while 	 1655 
 didn 	 1676 
 help 	 1686 
 again 	 1714 
 give 	 1730 
 made 	 1764 
 game 	 1775 
 actually 	 1781 
 name 	 1782 
 sure 	 1807 
 same 	 1828 
 work 	 1828 
 long 	 1832 
 anyone 	 1879 
 since 	 1907 
 gold 	 1949 
 anything 	 1987 
 someone 	 2106 
 love 	 2316 
 things 	 2447 
 still 	 2604 
 thing 	 2637 
 right 	 2641 
 back 	 2800 
 first 	 2867 
 make 	 3542 
 think 	 5574 
 people 	 5590 
I plan to cut off the top 5% of the common words from being searched, which means, from a word search table of 60,000 words, I would run a query like:

Code: Select all

SELECT word_common FROM bb_search_wordlist ORDER BY word_common DESC LIMIT 3000, 1
Result will be saved to a $board_config variable in the database, and under search.php, the query involving word_common would be "word_common < $board_config['word_common']" instead of "word_common <> 1".

I actually have yet to do this, right now I just have "word_common < 100" since I'm still experimenting with the system.

lanzer
Registered User
Posts: 152
Joined: Wed Oct 10, 2001 10:00 am
Contact:

Re: Thanks for the info

Post by lanzer » Fri Oct 10, 2003 7:11 am

Remix_88 wrote: You mentioned the sub-category mod in an earlier post and indicated that you have done some work to improve it's performance too. Is the sub-category mod you are referring too Categories hierarchy v1.1.0...

http://www.phpbb.com/phpBB/viewtopic.php?t=87278

If so, any chance you could share you tweaks for it?


Hi Remix, the sub-category hack I mentioned is infact the one from Ptirhiik. Though I'm afraid there isn't a quick hack for that mod because I didn't implement his whole mod then tweak it. Instead, I selected a few of his database changes, a few of the mods on the viewforum.php page, and most of the admin page modifications. While the meat of the mod was a series of functions to catalogue all the categories, and forums, it's also a series of extremely database intensive queries, so I ended up only selecting a portion of the code to give me sub category info with their authentication levels, excluding the post related info.

The mod fetches all categories, forums, their auth levels, plus new posts made within every forum. (similar to what the index page does in gathering all the unread posts made since last visit) This query happens on every page after the mod and that's something I couldn't afford. At the end, only about 30% of the mod was implemented in various spots, and that's why it's too hard for me to point out each individual change I did.

User avatar
Remix_88
Registered User
Posts: 46
Joined: Wed Apr 23, 2003 12:52 pm
Location: Hampshire, UK

Post by Remix_88 » Fri Oct 10, 2003 10:59 am

Thank you for taking the time to reply to my question. My site is not (and will never be) on the same scale as yours so my question was really a point of interest.

I do look forward to seeing any other tweak you maybe cooking up, especially the additional search optimizations :-)
Regards, Remix_88.

lanzer
Registered User
Posts: 152
Joined: Wed Oct 10, 2001 10:00 am
Contact:

Post by lanzer » Wed Oct 15, 2003 2:44 am

This tweak if on the pagination code. Since our site has a lot of long threads, the standard code was not sufficient for people to hop back and forth between hundreds of pages. The 2 major changes I made were to add more page numbers, and a "skip by 10 pages" option if there are more than 10 pages in a thread.

The second change will add a navigation bar for advancing by 10 pages, 100 pages, or to jump to the midpoint between the current page and the last page.

example:

Code: Select all

Goto page Previous  1, 2 ... 105, 106, 107, 108, 109, 110, 111, 112 ... 2339, 2340, 2341  Next
First  [|<]  [<<]  [<]  [>]  [>>]  [>|]  Last
Oh yeah, note the lack of the "append_sid" function. Tiny attempt to speed things up.

Please note that there are no use of proper language files. Non-english forum admins beware. :)

Code: Select all

function generate_pagination($base_url, $num_items, $per_page, $start_item, $add_prevnext_text = TRUE)
{
	global $lang;

	$total_pages = ceil($num_items/$per_page);
    $base_url = append_sid($base_url);

	if ( $total_pages == 1 )
	{
		return '';
	}

	$on_page = floor($start_item / $per_page) + 1;

	$page_string = '';
    $page_bar = '';
	if ( $total_pages > 20 )
	{
		if ($total_pages > 100) {
            if ($on_page != 0) {
                $page_bar .= '<a href="' . $base_url . '&start=0' . '">First</a>&&';
            } else {
                $page_bar .= 'First&&';
            }
            if ($on_page > 100) {
                $to_page = floor($on_page / 2);
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Back to page ".$to_page."\">|&</a>]&&";
                $to_page = $on_page - 100;
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Back 100 pages\">&&</a>]&&";
            } else {
                $page_bar .= '[<b>--</b>]&&';
                $page_bar .= '[<b>--</b>]&&';
            }
            if ($on_page > 10) {
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( ($on_page - 10) * $per_page ) . "\" title=\"Back 10 pages\">&</a>]&&";
            } else {
                $page_bar .= '[<b>-</b>]&&';
            }
            
            if ($on_page < $total_pages - 10) {
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( ($on_page + 10) * $per_page ) . "\" title=\"Skip 10 pages\">></a>]&&";
            } else {
                $page_bar .= '[<b>-</b>]&&';
            }
            if ($on_page < $total_pages - 100) {
                $to_page = 100 + $on_page;
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Skip 100 pages\">>></a>]&&";
                $to_page = floor(($total_pages - $on_page) / 2) + $on_page;
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Skip to page ".$to_page."\">>|</a>]&&";
            } else {
                $page_bar .= '[<b>--</b>]&&';
                $page_bar .= '[<b>--</b>]&&';
            }
            if ($on_page != $total_pages) {
                $page_bar .= '<a href="' . $base_url . "&start=" . ( ( $total_pages - 1 ) * $per_page ) . '">Last</a>';
            } else {
                $page_bar .= 'Last';
            }
        } else {
            if ($on_page < $total_pages - 10) {
                $page_string .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( ($on_page + 10) * $per_page ) . "\" title=\"Skip 10 pages\">></a>]&";
            }
        }
        
        $init_page_max = ( $on_page < 3 ) ? 6 : 2;

		for($i = 1; $i < $init_page_max + 1; $i++)
		{
			$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
			if ( $i <  $init_page_max )
			{
				$page_string .= ", ";
			}
		}

        if ( $on_page > 1  && $on_page < $total_pages )
        {
            $page_string .= ( $on_page > 5 ) ? ' ... ' : ', ';

            $init_page_min = ( $on_page > 4 ) ? $on_page : 5;
            $init_page_max = ( $on_page < $total_pages - 4 ) ? $on_page : $total_pages - 4;

            if ($init_page_max + 6 >= $total_pages - 2) $mid_page_index = $total_pages - 2;
              else $mid_page_index = $init_page_max + 6;
            for($i = $init_page_min - 2; $i < $mid_page_index; $i++)
            {
                $page_string .= ($i == $on_page) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
                if ( $i <  $mid_page_index - 1 )
                {
                    $page_string .= ', ';
                }
            }

            $page_string .= ( $on_page < $total_pages - 3 ) ? ' ... ' : ', ';
        }
        else
        {
            $page_string .= ' ... ';
        }

        for($i = $total_pages - 2; $i < $total_pages + 1; $i++)
        {
            $page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>'  : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
            if( $i <  $total_pages )
            {
                $page_string .= ", ";
            }
        }
	} else if ( $total_pages > 10 )
	{
		$init_page_max = ( $total_pages > 3 ) ? 3 : $total_pages;

		for($i = 1; $i < $init_page_max + 1; $i++)
		{
			$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
			if ( $i <  $init_page_max )
			{
				$page_string .= ", ";
			}
		}

		if ( $total_pages > 3 )
		{
			if ( $on_page > 1  && $on_page < $total_pages )
			{
				$page_string .= ( $on_page > 5 ) ? ' ... ' : ', ';

				$init_page_min = ( $on_page > 4 ) ? $on_page : 5;
				$init_page_max = ( $on_page < $total_pages - 4 ) ? $on_page : $total_pages - 4;

				for($i = $init_page_min - 1; $i < $init_page_max + 2; $i++)
				{
					$page_string .= ($i == $on_page) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
					if ( $i <  $init_page_max + 1 )
					{
						$page_string .= ', ';
					}
				}

				$page_string .= ( $on_page < $total_pages - 4 ) ? ' ... ' : ', ';
			}
			else
			{
				$page_string .= ' ... ';
			}

			for($i = $total_pages - 2; $i < $total_pages + 1; $i++)
			{
				$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>'  : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
				if( $i <  $total_pages )
				{
					$page_string .= ", ";
				}
			}
		}
	}
	else
	{
		for($i = 1; $i < $total_pages + 1; $i++)
		{
			$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
			if ( $i <  $total_pages )
			{
				$page_string .= ', ';
			}
		}
	}

	if ( $add_prevnext_text )
	{
		if ( $on_page > 1 )
		{
			$page_string = ' <a href="' . $base_url . "&start=" . ( ( $on_page - 2 ) * $per_page ) . '" title="Page '.($on_page - 1).'">' . $lang['Previous'] . '</a>&&' . $page_string;
		}

		if ( $on_page < $total_pages )
		{
			$page_string .= '&&<a href="' . $base_url . "&start=" . ( $on_page * $per_page ) . '" title="Page '.($on_page + 1).'">' . $lang['Next'] . '</a>';
		}

	}

    if ($page_bar) {
        $page_string .= '<BR>' . $page_bar;
    }

	$page_string = $lang['Goto_page'] . ' ' . $page_string;

	return $page_string;
}
Last edited by lanzer on Tue Nov 25, 2003 3:13 pm, edited 1 time in total.

gravyplaya
Registered User
Posts: 15
Joined: Sun Jun 22, 2003 7:29 pm

where does all this go?

Post by gravyplaya » Wed Oct 15, 2003 6:03 am

What file does all this go into? Im having a hard time tryin to insert your code because its hard to tell what file it goes into. Checkout my forums at www.hieroglyphics.com/hoopla the General Discussion thread takes waaay to long to load. Sometimes errors out with:
Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 1987079 bytes) in /includes/functions.php on line 568
lanzer wrote: This tweak if on the pagination code. Since our site has a lot of long threads, the standard code was not sufficient for people to hop back and forth between hundreds of pages. The 2 major changes I made were to add more page numbers, and a "skip by 10 pages" option if there are more than 10 pages in a thread.

The second change will add a navigation bar for advancing by 10 pages, 100 pages, or to jump to the midpoint between the current page and the last page.

example:

Code: Select all

Goto page Previous  1, 2 ... 105, 106, 107, 108, 109, 110, 111, 112 ... 2339, 2340, 2341  Next
First  [|<]  [<<]  [<]  [>]  [>>]  [>|]  Last
Oh yeah, note the lack of the "append_sid" function. Tiny attempt to speed things up.

Please note that there are no use of proper language files. Non-english forum admins beware. :)

Code: Select all

function generate_pagination($base_url, $num_items, $per_page, $start_item, $add_prevnext_text = TRUE)
{
	global $lang;

	$total_pages = ceil($num_items/$per_page);
    $base_url = append_sid($base_url);

	if ( $total_pages == 1 )
	{
		return '';
	}

	$on_page = floor($start_item / $per_page) + 1;

	$page_string = '';
    $page_bar = '';
	if ( $total_pages > 20 )
	{
		if ($total_pages > 100) {
            if ($on_page != 0) {
                $page_bar .= '<a href="' . $base_url . '&start=0' . '">First</a>&&';
            } else {
                $page_bar .= 'First&&';
            }
            if ($on_page > 100) {
                $to_page = floor($on_page / 2);
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Back to page ".$to_page."\">|&</a>]&&";
                $to_page = $on_page - 100;
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Back 100 pages\">&&</a>]&&";
            } else {
                $page_bar .= '[<b>--</b>]&&';
                $page_bar .= '[<b>--</b>]&&';
            }
            if ($on_page > 10) {
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( ($on_page - 10) * $per_page ) . "\" title=\"Back 10 pages\">&</a>]&&";
            } else {
                $page_bar .= '[<b>-</b>]&&';
            }
            
            if ($on_page < $total_pages - 10) {
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( ($on_page + 10) * $per_page ) . "\" title=\"Skip 10 pages\">></a>]&&";
            } else {
                $page_bar .= '[<b>-</b>]&&';
            }
            if ($on_page < $total_pages - 100) {
                $to_page = 100 + $on_page;
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Skip 100 pages\">>></a>]&&";
                $to_page = floor(($total_pages - $on_page) / 2) + $on_page;
                $page_bar .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( $to_page * $per_page ) . "\" title=\"Skip to page ".$to_page."\">>|</a>]&&";
            } else {
                $page_bar .= '[<b>--</b>]&&';
                $page_bar .= '[<b>--</b>]&&';
            }
            if ($on_page != $total_pages) {
                $page_bar .= '<a href="' . $base_url . "&start=" . ( $total_pages * $per_page ) . '">Last</a>';
            } else {
                $page_bar .= 'Last';
            }
        } else {
            if ($on_page < $total_pages - 10) {
                $page_string .= '[<a style="text-decoration:none;font-weight:bold;" href="' . $base_url . "&start=" . ( ($on_page + 10) * $per_page ) . "\" title=\"Skip 10 pages\">></a>]&";
            }
        }
        
        $init_page_max = ( $on_page < 3 ) ? 6 : 2;

		for($i = 1; $i < $init_page_max + 1; $i++)
		{
			$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
			if ( $i <  $init_page_max )
			{
				$page_string .= ", ";
			}
		}

        if ( $on_page > 1  && $on_page < $total_pages )
        {
            $page_string .= ( $on_page > 5 ) ? ' ... ' : ', ';

            $init_page_min = ( $on_page > 4 ) ? $on_page : 5;
            $init_page_max = ( $on_page < $total_pages - 4 ) ? $on_page : $total_pages - 4;

            if ($init_page_max + 6 >= $total_pages - 2) $mid_page_index = $total_pages - 2;
              else $mid_page_index = $init_page_max + 6;
            for($i = $init_page_min - 2; $i < $mid_page_index; $i++)
            {
                $page_string .= ($i == $on_page) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
                if ( $i <  $mid_page_index - 1 )
                {
                    $page_string .= ', ';
                }
            }

            $page_string .= ( $on_page < $total_pages - 3 ) ? ' ... ' : ', ';
        }
        else
        {
            $page_string .= ' ... ';
        }

        for($i = $total_pages - 2; $i < $total_pages + 1; $i++)
        {
            $page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>'  : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
            if( $i <  $total_pages )
            {
                $page_string .= ", ";
            }
        }
	} else if ( $total_pages > 10 )
	{
		$init_page_max = ( $total_pages > 3 ) ? 3 : $total_pages;

		for($i = 1; $i < $init_page_max + 1; $i++)
		{
			$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
			if ( $i <  $init_page_max )
			{
				$page_string .= ", ";
			}
		}

		if ( $total_pages > 3 )
		{
			if ( $on_page > 1  && $on_page < $total_pages )
			{
				$page_string .= ( $on_page > 5 ) ? ' ... ' : ', ';

				$init_page_min = ( $on_page > 4 ) ? $on_page : 5;
				$init_page_max = ( $on_page < $total_pages - 4 ) ? $on_page : $total_pages - 4;

				for($i = $init_page_min - 1; $i < $init_page_max + 2; $i++)
				{
					$page_string .= ($i == $on_page) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
					if ( $i <  $init_page_max + 1 )
					{
						$page_string .= ', ';
					}
				}

				$page_string .= ( $on_page < $total_pages - 4 ) ? ' ... ' : ', ';
			}
			else
			{
				$page_string .= ' ... ';
			}

			for($i = $total_pages - 2; $i < $total_pages + 1; $i++)
			{
				$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>'  : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
				if( $i <  $total_pages )
				{
					$page_string .= ", ";
				}
			}
		}
	}
	else
	{
		for($i = 1; $i < $total_pages + 1; $i++)
		{
			$page_string .= ( $i == $on_page ) ? '<b>' . $i . '</b>' : '<a href="' . $base_url . "&start=" . ( ( $i - 1 ) * $per_page ) . '">' . $i . '</a>';
			if ( $i <  $total_pages )
			{
				$page_string .= ', ';
			}
		}
	}

	if ( $add_prevnext_text )
	{
		if ( $on_page > 1 )
		{
			$page_string = ' <a href="' . $base_url . "&start=" . ( ( $on_page - 2 ) * $per_page ) . '" title="Page '.($on_page - 1).'">' . $lang['Previous'] . '</a>&&' . $page_string;
		}

		if ( $on_page < $total_pages )
		{
			$page_string .= '&&<a href="' . $base_url . "&start=" . ( $on_page * $per_page ) . '" title="Page '.($on_page + 1).'">' . $lang['Next'] . '</a>';
		}

	}

    if ($page_bar) {
        $page_string .= '<BR>' . $page_bar;
    }

	$page_string = $lang['Goto_page'] . ' ' . $page_string;

	return $page_string;
}

Locked

Return to “2.0.x Discussion”