Yeah I've posted over and over before about this.
Outside of the imperial or Clone Navy scum and villainy have a larger presence in the canon, old canon, printed and film media than any of the other factions. Add rhem up and they may have larger armies. Maul's combined crime organization would have rivaled the Imperials in size. I'm not counting the navy because this isn't Armada. But number all the ruffians, hired guns, gangs, informants, drug dealers and associates on all the planets in the systems not to mention tribal groups like Sand people and what not and you're including a large portion of the galaxy which could be in the faction. Tons of options.