Failing a task, and failing again

By Musha Shukou, in Elder Sign

I know I'm not the only person to have asked this, but I simply don't have the time and patience to wade through all the search results.

So..

When you fail a task, you must set aside a die. Then, you may re-roll all remaining dice to attempt this task again. If you fail AGAIN, would you set aside ANOTHER die? Or can you only lose 1 die per task?

One each time you fail, until eventually you don't have enough dice to complete all the tasks and fail the Adventure.