Notice on page 204 every single skill mentioned under the "Perform a skill check" are those that have no negative impact ont he enemy. Also notice that under "Perform a combat check" it says you perform a combat check when you use a combat skill to attack a target." You are reading this as "You perform a combat check when AND ONLY WHEN you use a combat skill to attack a target."
It is quite obvious that the developers could not think of and include every way that each skill could be used to harm enemies and thus specifically call out those uses as combat checks. They assume some common sense on the part of players. For example, in a factory an individual might try to use computer to make a robotic arm hit an opponent or a conveyor belt to suddently begin moving tripping his target. In this example the character is not using any of the combat skills but is still making an attack. The way you are reading the text Adversary would not apply to these obviously combat activities.
Your argument reminds me of those on the Pathfinder boards that argue that since the RAW for the "Dead" condition does not state specifically you are unable to make actions you can continue to do so when dead. A little common sense goes a long way.