A custody evaluation is an evaluation performed by a psychologist or other qualified/licensed professional aimed at helping the court determine what physical custody situation is in the best interest of the child. It usually involves the parties and child/ren meeting the evaluator for several sessions, sometimes alone and sometimes together. Sometimes it will also involve psychological testing. Usually the evaluator also acts as a mediator and tries to get the parties to settle the case. If the case does not settle, then the evaluator will issue a recommendation stating their findings, along with a proposed custody schedule.
You may or may not be required to have a custody evaluation. Some court systems favor them, whereas other courts recognize that they can create additional obstacles to resolving a custody case. It really just depends on where your case will be heard.