Unicode "unknown character" squares randomly appear

Get help with installation and running phpBB 3.0.x here. Please do not post bug reports, feature requests, or MOD-related questions here.
Suggested Hosts
Forum rules
END OF SUPPORT: 1 January 2017 (announcement)
User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Unicode "unknown character" squares randomly appear

Post by Eelke » Wed Jul 16, 2008 6:24 am

Something in the confusable is wrong..? Confusing indeed...

mkruer
Registered User
Posts: 74
Joined: Mon Apr 28, 2003 7:49 pm

Re: Unicode "unknown character" squares randomly appear

Post by mkruer » Wed Jul 16, 2008 4:45 pm

Eelke wrote:Something in the confusable is wrong..? Confusing indeed...
Strike that. The files are identical. There was little DB changes, and we can rule that out anyway because the error appears in preview view mode (nothing is being written to the db in that case) So it comes down to the phpbb3 parser or php. AFAIK the php has not been updated. Could it be there is a bug in the parser logic? I would be happy to test, but I don't control the server. Just let me know what you need and i will try to accommodate you as much as possible.

User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Unicode "unknown character" squares randomly appear

Post by Eelke » Wed Jul 16, 2008 6:46 pm

I'm sorry, I should have been more clear. I do not understand what you mean by "confusable". What files are identical?

mkruer
Registered User
Posts: 74
Joined: Mon Apr 28, 2003 7:49 pm

Re: Unicode "unknown character" squares randomly appear

Post by mkruer » Wed Jul 16, 2008 8:42 pm

During the install there is a file called confusables.php located in phpbb3\install\data in it it replaced multibyte character with single byte charters I thought it was updated and might now have a bug in it, but they are identical. The problem i an having is identical to the one Nicholas the Italian reported.

http://www.amdzone.com/phpbb3/viewtopic ... 2&t=135316

You can register and try any two multibyte characters to kick off the bug

User avatar
Nicholas the Italian
Registered User
Posts: 170
Joined: Tue Nov 21, 2006 5:18 pm

Re: Unicode "unknown character" squares randomly appear

Post by Nicholas the Italian » Wed Jul 16, 2008 11:24 pm

Dundurs wrote:īdzīgi glāžšķūņu rūķīši
That's waaay worse than Italian, which probably explains why in your case the problem happens more frequently. :)
I've not experienced the problem for a couple of weeks now.
Someone should check the contents of their database. Preferably, also the HTTP traffic with an HTTP tracer such as YATT. It needs to be determined where the error is being introduced.
The problem with that is that if it needs to be installed on the server, it's not easily done on shared hosting.
Also mkruer is right, the problem happens before data is actually committed to the database.

Can you guys please check what exact version of PHP does your server run?

Funny thing, it would be much easier to trace if it affected english-speaking boards. Lucky thou who rely on ASCII... :twisted:
Whatever I say, it's not my fault.

SamG
Former Team Member
Posts: 3221
Joined: Fri Aug 31, 2001 6:35 pm
Location: Beautiful Northwest Lower Michigan
Name: Sam Graf

Re: Unicode "unknown character" squares randomly appear

Post by SamG » Thu Jul 17, 2008 12:49 am

Would it be convenient to experiment with JavaScript disabled?
We should talk less, and say more.

mkruer
Registered User
Posts: 74
Joined: Mon Apr 28, 2003 7:49 pm

Re: Unicode "unknown character" squares randomly appear

Post by mkruer » Thu Jul 17, 2008 5:15 am

My System info:
- PHP Version 5.2.0-8+etch5~pu1
- MySQL 5.0.45-log
- tables are utf8_bin, MyISAM engine;
- board is a phpBB2 upgraded;

User avatar
Phil
Former Team Member
Posts: 10403
Joined: Sat Nov 25, 2006 4:11 am
Name: Phil Crumm
Contact:

Re: Unicode "unknown character" squares randomly appear

Post by Phil » Thu Jul 17, 2008 5:21 am

SamG wrote:Would it be convenient to experiment with JavaScript disabled?
Moving on, with the wind. | My Corner of the Web

User avatar
Eelke
QA Team
Posts: 2903
Joined: Thu Dec 20, 2001 8:00 am
Location: NL, Bussum
Name: Eelke Blok
Contact:

Re: Unicode "unknown character" squares randomly appear

Post by Eelke » Thu Jul 17, 2008 7:08 am

Nicholas the Italian wrote:
Someone should check the contents of their database. Preferably, also the HTTP traffic with an HTTP tracer such as YATT. It needs to be determined where the error is being introduced.
The problem with that is that if it needs to be installed on the server, it's not easily done on shared hosting.
Also mkruer is right, the problem happens before data is actually committed to the database.
YATT is a client-side tool for Windows. http://www.pocketsoap.com/YATT/
Don't bother with the WinPCAP download, YATT has the option to use a system built into Windows.

Dundurs
Registered User
Posts: 14
Joined: Wed Jan 05, 2005 9:40 am

Re: Unicode "unknown character" squares randomly appear

Post by Dundurs » Thu Jul 17, 2008 8:48 am

My system info:
PHP Version 5.2.6-pl2-gentoo
MySQL 5.0.60-log
Tables utf8_bin MyISAM
Board PhpBB3 clean install.

PS.
For now I did solution from bug report by setting mbstring overload to 4 in .htaccess and bypassing utf8_normalization function. Now I have third day without problems. Ref.: http://www.phpbb.com/bugs/phpbb3/ticket ... t_id=19175

sotis
Registered User
Posts: 294
Joined: Tue Oct 17, 2006 4:56 pm
Location: Bulgaria
Contact:

Re: Unicode "unknown character" squares randomly appear

Post by sotis » Wed Aug 27, 2008 1:13 pm

Somebody found where the problem is?

User avatar
Techie-Micheal
Security Consultant
Posts: 19511
Joined: Sun Oct 14, 2001 12:11 am
Location: In your servers

Re: Unicode "unknown character" squares randomly appear

Post by Techie-Micheal » Wed Aug 27, 2008 1:59 pm

To add to the confusion, I've seen this in my Firefox 3 on both phpBB.com in the past and other plain-jane sites. In fact, on one, I had to change from UTF8 to ISO-8859-1 in order to view the page correctly, because I got the same "unknown Unicode character box" thingy.
Proven Offensive Security Expertise. OSCP - GXPN

Dundurs
Registered User
Posts: 14
Joined: Wed Jan 05, 2005 9:40 am

Re: Unicode "unknown character" squares randomly appear

Post by Dundurs » Wed Aug 27, 2008 8:00 pm

As I wrote 3 messages above. From that date until now I have no problems with UTF-8. Of course this is not the right solution but it takes me free from headaches...

sotis
Registered User
Posts: 294
Joined: Tue Oct 17, 2006 4:56 pm
Location: Bulgaria
Contact:

Re: Unicode "unknown character" squares randomly appear

Post by sotis » Thu Aug 28, 2008 1:11 pm

Dundurs wrote:As I wrote 3 messages above. From that date until now I have no problems with UTF-8. Of course this is not the right solution but it takes me free from headaches...
What did you exactly do?
Don't you have another side effects or problems, I am worried about this if I change the code.

Dundurs
Registered User
Posts: 14
Joined: Wed Jan 05, 2005 9:40 am

Re: Unicode "unknown character" squares randomly appear

Post by Dundurs » Thu Aug 28, 2008 8:16 pm

http://www.phpbb.com/bugs/phpbb3/ticket ... t_id=19175

I did solution given by pongraczi on link above. It's simple to bypass utf8_normalize_nfc function in includes/utf/utf_tools.php. Next change .htaccess as writen in the same post. Of course second will work only if you are on Apache webserver.

By now I didn't find any sideffect on this. At least not in Latvian, Russian or English posts and PM's. I'm still pretty sure the problem is in PHPBB3 code witch conflicts with some PHP 5 newest releases. I found the same problem with some CMS also but I'm out of time to investigate for better solution.

Locked

Return to “[3.0.x] Support Forum”