When instructed to roll one less die is that to a minimum of one or can it be taken to zero if I only have agility of one?

Zero. Nowhere on the card does it say to a minimum of 1

I once had a Decimator down to negative-3 defense dice, thanks to tractor tokens + double structural damage!