I found it!
i've been "debugging" the p_master class to find out how the permissons and menu items are build up and I finally discovered it:
in table phpbb_acl_options I've added 2 new rows: "c_" and "c_test_option"
in table phpbb_modules I duplicated row "ACP_USER_ROLES" (module_id 80) and changed the following fields
* langname => "C_CUSTOM_ROLES"
* module_mode => "custom_roles"
* left_id => 201
* right_id => 202
(left- and right_id are the next indexes next to "ACP_FORUM_ROLES". That way it's the next menu-element)
now the tricky part is the left_id and right_id, it took me a while to figure that out correctly. The menu-items in the ACP are grouped by pages and categories.
so to add my role type to the menu I had to do the following:
the page is ACP_CAT_PERMISSIONS (which consists of all modules who's left- and right_id is between 167 and 216)
and category is ACP_PERMISSION_ROLES (which consists of all modules who's left- and right_id is between 192 and 202).
so, for my custom role type to actuall show up in the acp "permissions" page I had to update all higher left_ and right_ids:
UPDATE phpbb_modules SET left_id = left_id + 1 WHERE left_id >= 200 AND module_id <> 199;
UPDATE phpbb_modules SET right_id = right_id + 1 WHERE right_id >= 201 AND module_id <> 199;
this updated the ACP_PERMISSIONS_ROLES to include my new module and thus adding it correctly to the modules-list in p_master().
So now It's listed and I can create my custom roles with my custom permissions. Next challange is to be able to add my custom roles to a group (if I figure that out I'll post it back here too. Maybe someone can put it in a real tutorial or copy it to the wiki with more and better explanation?)
A thought for the PHPBB Development team: I assume there are many more sites like me who'd want to use the PHPBB Forum as the framework for their site (since it's simply a great framework: it has very good and secure session management, user registration & control options, captcha's, logging, db-classes, template & caching functions, ... oh and a superb forum too ;o). It would be great if you'd make it easier to add custom role-types. That should be part of the system options in the ACP too. Would have save me a few hours