Many thanks to the author of this tool and everyone in this thread who posted useful information
I was able to create 5000+ users without much issues.
One problem that I found was that the first user in the csv was always reported as "already exists" so I created a dummy user for the first line of the csv.
I also has some users with a blank space as last character of their username, those would clash with the same username without the space at the end so I had to rename them before import.
There was also one user with a username like this "John > The Master" where the ">" seemed to be a problem.