How it affects people is going to vary. There are three pieces of tracking information for each user: forum mark time, topic mark time, and user last mark track (mark all). If there is no topic mark time, it falls back to the forum mark time, if there's no forum mark time, it falls back to the last time that the user hit "Mark all forums read."
When a topic is read, an entry is placed into the topic track table. When a forum is marked as read, an entry is placed into the forum track table, and all topics in that forum are removed from the topic track table. When "mark all forums read" is clicked, all entries for that user are removed from the forum and topic track tables, and the user_lastmark entry in the user table is updated.
If the topic track table is filling up, then your users aren't making use of the "mark forum/all read" options. If you delete those entries, they'll show up as unread. What you would probably want to do is delete all entries after some time X and then update all users lastmarks to that time if their mark is older than that. You could do the same thing for the forum track table too.