Defiantly challenge it. You weren't harming anyone by passing on the right, and you made a logical decision. It's not like you went off-roading to get around the car. Joe's idea is good too, though ;)