I feel like I'm doing something wrong. Even if I don't have enough glyphs to possibly fill out my adventures, I have to continue rolling and discarding every glyph,. Am I doing something wrong or is this just crappy design?

Actually, you do have the option to 'fail by choice'. Click on the banner with the adventure's title at the top after you are actively trying to complete it. This will bring up a screen where, at the bottom, you have the option to fail the adventure, rather than continue rolling. Huge time saver- I use that option all the time.

If the adventure automatically failed for you once you don't have enough glyphs, I think many players (especially novices) would find this too jarring, and wouldn't understand why they failed. Better to let the player reason this for themselves.