Ive started writing up a part of my game where the party are to guard and convoy a cocky young Duros Podracer to the Boonta eve classic, he is the hot pick for the win which unfortunatley for him has annoyed a certain Hutt, so i start off by statting him up as a driver which seemed to fit alot better for a podracer than the Pilot specialization and i come across all terrain driver, i read it and it says the character suffers no penalties when driving through difficult terrain using planetary vehicles, does this mean he ignores all the setback dice for say weaving inbetween giant rocks? because this seems very powerful for a 5 point skill.

Yep.

I don't have them but Jewel of Yavin is meant to have a race in it and (I imagine) Suns of Fortune has something to say about keeping races challenging.

Note that Difficult Terrain (table 7-15 of the CRB p. 253) and Navigational Hazards (table 7-8, page 240) have some (very minor and little arbitrary) differences. Utilize this to keep things interesting, but not to totally screw over the player with the All-Terrain Driver talent

Given the similarities of the items on both tables, I'm thinking I'd run into problems deciding if a given factor counted as difficult terrain or as a navigation hazard ("dense undergrowth" appears as a single-setback entry on both charts, for example). I'd have an even harder time explaining to the guy who bought the talent why I was still applying the setback even though it appeared on the terrain chart.

So, I'm thinking for the purposes of the talent that I'll count as terrain anything that the vehicle will likely be riding over (road surface, streams, low-lying brush, etc.) and hazards as anything that the vehicle will need to be steered around (crumbling walls, traffic, trees, etc.)

I would say

Difficult terrain

Navigation Hazard, come across this in canyon flying high speed Swoop.

"Does 'threading the needle' ring any bell, Hotshot." Swoop race Champion to first timer.

The thing that makes driving through terrain of any kind (everything except open space and flat, open ground areas) isn't the setback dice - it's the "Speed upgraded a number of times equal to 1/2 Silhouette". A podracer would be moving at Speed 5, like the fastest swoops, and have a Silhouette of 2. That's one red Challenge die and four purple Difficulty dice. And those don't go away.