if the attack and defender both has re roll abilities like an elite storm trooper and diala. who re rolls first?

Attacker abilities generally go before defender abilities.

An even more ambiguous question is: Can attack dice be rerolled after a defense dice is rerolled?

So when an HK attacks, can he reroll the defense first to see what happens, then reroll an attack? Or vice versa? Or does it have to be all decided before any rerolls occur?

During an attack the timing (page 2 of RRG): Mission rules (including conditions), Attacker abilities, Defender abilities. Attacker decides whether to reroll first, then the defender.

HK Assassin Droid: It is less ambiguous. Both abilities are attacker abilities, so the attacker can use them in the order of their choice. (It does not matter whose dice are rerolled nor who performs the reroll, only whose abilities they are.)

Additionally, with the HK in particular, the attacker can CHOOSE to trigger those rerolls. So, during "Step 3 (Rerolls)", the HK could trigger the defender reroll, and wait to see what that result is before triggering the attacker reroll.

Also note, that a die may only be rerolled once, regardless of the source. So if the attacker forces the defender to reroll his only defense die, the defender may not use ability or effect to reroll that defense die again during that attack.