Today Gaia has a about 150 web servers behind our load balancer, and about 20 databases handling different features. Many of our old servers had been either de-commissioned or converted to storage servers. The new servers we're installing are all Opteron servers that are 4 times faster than older machines with the same power requirement. Since the ISP charges a lot for power, it's much more economical for us to upgrade.
We don't really believe in clustering. Rather we changed our database structure and the database abstraction layer to divide up data into seperate physical databases. That way we only need to add more of the same servers when upgrading, and we don't have to spend money on expensive clustering solutions and risk supporting a complex system.
We've just gotten another 20 database servers to replicate the existing ones for redundency. No replication is used for scalability or handling bandwidth. We still use MyISAM for small logs and data that doesn't get updated more than once a second, otherwise everything else is InnoDB.
The session handler had been completely re-written to use the new database structure and to accomidate a whole bunch of Gaia related features. A caching system for gold earning for example. The user id determines which of the 4 database server to use. When we need more performance we will expand that up to 8 and change the session handler to accomidate 8 servers.
The database structure had also changed for the ability to handle threads of unlimited posts. Changes like these involved a few guys working together and a span of months to complete. Once I find the time I'll publish them for the phpbb team. Right now I'm still pulling my hair trying to get the latest features to release.
There are many other nifty things that we're doing, thanks to the new IT guy that joined our team, and many software developed by the guys at Livejournal such as memcache, PerlBal and MogleFS. More details will be posted once the dust settles.
What would be interesting to know is what is gaiaonline server setup ?
Single server, cluestered ?
Separated sql, with slaves ?
Which db type ?
What is your session trick (30 000 at a time, must be hard to deal with sometimes) ?
And since obviously you tweaked it a lot, did you implement other things than the one demonstarted here?
This may have olready been said somewhere, but I am wondering.
By the way, bravo for gaia, hudge, and still fast.
Thanks to you, nobody can really say phpbb is not a working script (even though you must have changed a lot of things).