In principle,  duplicates are  removed during startup.  My guess  is that
many of the entries are "old" (no  longer in any of the lists). You could
write  a  program  to  eliminate   entries  which  aren't  in  any  list.
Technically, this  isn't a 100% accurate  thing to do. Some  people might
have registered a  password for other reasons, for instance,  or might be
AFDed, and so on.
 
  Eric