It is kinda impactful on the server because it requires a new table join on the DB query.
For this kind of thing I think that the option that IP.B uses is one of the best options regarding this.
There's an arrow-kind image that you may click. When you click it, it does an AJAX call to the server to get up to 3 posts. The 1st post of the topic, the post after the last one the user read (only if logged in) and the last post of the topic (if there's an overlap, then duplicates do not appear).
I think this way we can balance things as I think that, most of the times (considering all users), the users want to see most posts, not just the 1st one or just the last ones, etc... Anyway, this seems to be a good way of solving (not necessarily the best one)