Page 2 of 2

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Jul 16, 2008 6:24 am
by Eelke
Something in the confusable is wrong..? Confusing indeed...

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Jul 16, 2008 4:45 pm
by mkruer
Eelke wrote:Something in the confusable is wrong..? Confusing indeed...
Strike that. The files are identical. There was little DB changes, and we can rule that out anyway because the error appears in preview view mode (nothing is being written to the db in that case) So it comes down to the phpbb3 parser or php. AFAIK the php has not been updated. Could it be there is a bug in the parser logic? I would be happy to test, but I don't control the server. Just let me know what you need and i will try to accommodate you as much as possible.

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Jul 16, 2008 6:46 pm
by Eelke
I'm sorry, I should have been more clear. I do not understand what you mean by "confusable". What files are identical?

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Jul 16, 2008 8:42 pm
by mkruer
During the install there is a file called confusables.php located in phpbb3\install\data in it it replaced multibyte character with single byte charters I thought it was updated and might now have a bug in it, but they are identical. The problem i an having is identical to the one Nicholas the Italian reported.

http://www.amdzone.com/phpbb3/viewtopic ... 2&t=135316

You can register and try any two multibyte characters to kick off the bug

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Jul 16, 2008 11:24 pm
by Nicholas the Italian
Dundurs wrote:īdzīgi glāžšķūņu rūķīši
That's waaay worse than Italian, which probably explains why in your case the problem happens more frequently. :)
I've not experienced the problem for a couple of weeks now.
Someone should check the contents of their database. Preferably, also the HTTP traffic with an HTTP tracer such as YATT. It needs to be determined where the error is being introduced.
The problem with that is that if it needs to be installed on the server, it's not easily done on shared hosting.
Also mkruer is right, the problem happens before data is actually committed to the database.

Can you guys please check what exact version of PHP does your server run?

Funny thing, it would be much easier to trace if it affected english-speaking boards. Lucky thou who rely on ASCII... :twisted:

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Jul 17, 2008 12:49 am
by SamG
Would it be convenient to experiment with JavaScript disabled?

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Jul 17, 2008 5:15 am
by mkruer
My System info:
- PHP Version 5.2.0-8+etch5~pu1
- MySQL 5.0.45-log
- tables are utf8_bin, MyISAM engine;
- board is a phpBB2 upgraded;

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Jul 17, 2008 5:21 am
by Phil
SamG wrote:Would it be convenient to experiment with JavaScript disabled?

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Jul 17, 2008 7:08 am
by Eelke
Nicholas the Italian wrote:
Someone should check the contents of their database. Preferably, also the HTTP traffic with an HTTP tracer such as YATT. It needs to be determined where the error is being introduced.
The problem with that is that if it needs to be installed on the server, it's not easily done on shared hosting.
Also mkruer is right, the problem happens before data is actually committed to the database.
YATT is a client-side tool for Windows. http://www.pocketsoap.com/YATT/
Don't bother with the WinPCAP download, YATT has the option to use a system built into Windows.

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Jul 17, 2008 8:48 am
by Dundurs
My system info:
PHP Version 5.2.6-pl2-gentoo
MySQL 5.0.60-log
Tables utf8_bin MyISAM
Board PhpBB3 clean install.

PS.
For now I did solution from bug report by setting mbstring overload to 4 in .htaccess and bypassing utf8_normalization function. Now I have third day without problems. Ref.: http://www.phpbb.com/bugs/phpbb3/ticket ... t_id=19175

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Aug 27, 2008 1:13 pm
by sotis
Somebody found where the problem is?

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Aug 27, 2008 1:59 pm
by Techie-Micheal
To add to the confusion, I've seen this in my Firefox 3 on both phpBB.com in the past and other plain-jane sites. In fact, on one, I had to change from UTF8 to ISO-8859-1 in order to view the page correctly, because I got the same "unknown Unicode character box" thingy.

Re: Unicode "unknown character" squares randomly appear

Posted: Wed Aug 27, 2008 8:00 pm
by Dundurs
As I wrote 3 messages above. From that date until now I have no problems with UTF-8. Of course this is not the right solution but it takes me free from headaches...

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Aug 28, 2008 1:11 pm
by sotis
Dundurs wrote:As I wrote 3 messages above. From that date until now I have no problems with UTF-8. Of course this is not the right solution but it takes me free from headaches...
What did you exactly do?
Don't you have another side effects or problems, I am worried about this if I change the code.

Re: Unicode "unknown character" squares randomly appear

Posted: Thu Aug 28, 2008 8:16 pm
by Dundurs
http://www.phpbb.com/bugs/phpbb3/ticket ... t_id=19175

I did solution given by pongraczi on link above. It's simple to bypass utf8_normalize_nfc function in includes/utf/utf_tools.php. Next change .htaccess as writen in the same post. Of course second will work only if you are on Apache webserver.

By now I didn't find any sideffect on this. At least not in Latvian, Russian or English posts and PM's. I'm still pretty sure the problem is in PHPBB3 code witch conflicts with some PHP 5 newest releases. I found the same problem with some CMS also but I'm out of time to investigate for better solution.