So this Crisis says that it remains in play until you jump. But what does that mean?

The crisis card says:

Keep this card in play until a civilian ship or a basestar is destroyed. If it is in play when the fleet jumps, shuffle it into the deck.

After playing the ships shown on the attack crisis set the card to the side. If you haven't either lost a civ ship or destroyed a basestar whenever you next jump Galactica you reshuffle the attack card into the deck. It's just a cylon attack crisis that can re-occur if you don't deal with it properly.