If a referee is appointed by the court, each party shall have the same right to challenge the appointment. Challenges shall be made and determined in the same manner and with like effect as in the formation of juries, except that neither party shall be entitled to a peremptory challenge.