A Human in the Loop