As one of the Spider players who actively pushed the Integration agenda at Winter Court, please let me assure everyone that the idea of the Spider Clan as a Great Clan of the Empire, monsters and all, was a VERY challenging proposal, rooted deeply in what I consider central flaws to the narrative of L5R.
In most ways, L5R is a straight up American / Western European fantasy epic narrative. It simply appropriates Bushido, Samurai, and several Japanese and some Chinese elements to flavor its otherwise normal American fantasy story. Legend of the Five Rings treats bushido and samurai with the same hand that Dragonlance treats dragons and knights. They are simply the trappings of an otherwise straight forward Good vs Evil, Light vs Darkness war story that has been present in Western European and American storytelling since Chretien de Troyes. It may claim to be inspired by Musashi's "Way of the Five Rings" and Sun Tzu's "The Art of War", and the CCG actually felt like it was. The setting, however, and its central narratives were not.
The integration of the Spider Clan into Rokugan only works when you put the world through a paradigm shift, and move it out of dualism and towards wholism, which is actually closer to the fantasy epics produced by the cultures L5R was appropriating its trappings from. Seriously, read Tale of Genji or Journey to the West or Romance of the Three Kingdoms sometimes if you want to get a better feel for Chinese and Japanese storytelling. Or read some of the old Japanese ghost story Noh plays. Or watch Onmyouji (which is a great film regardless, and you should watch anyways). In a more wholistic world, it is perfectly acceptable for the Emperor to keep a private army of monsters and demons, lead by Tainted men and women, to be wielded as a weapon against their foreign enemies. If evil does not exist in opposition to good, but co-exists with good in a state of harmonious service to the Son of Heaven (or Fate, or the what have you), then there is no inherent problem in the existence of the Spider Clan (all bad blood aside... the conflict with the Crab Clan would still be acceptable, as how can you expect a clan of demon killing samurai to not want to kill the clan of demonic samurai?).
Unfortunately, this is not the story that L5R has been for the 20 years of its history, nor the story the majority of its players wanted to participate in. They wanted their Good vs Evil narrative, with the big bad evil needing the Clans to unite to overcome. In that narrative, the Spider Clan cannot exist as a part of the Empire as they are by their nature the enemy of the Empire, unless they rejected that nature and all the associated baggage. Which was why Walk in the Light Story Path was not popular, though the Shourido Above All Else Story Path was still a compelling attempt at synthesis (and a movement towards Spider Clan identity that was separate from the Shadowlands Horde).
TL; DR -> L5R is written as a Good vs Evil narrative because that is what people wanted. The Spider Clan cannot function in a Good vs Evil narrative without serious problems.
On another side note, all this arguing is again causing the "Battestar Galactica Reboot" thoughts in my head again. Which is always an interesting thought experiment.