Samuel was born in Hamilton, Victoria, the son of Maree and Clifford Samuel. He grew up in Adelaide, South Australia and graduated from Rostrevor College in 2001. He has a younger brother, Benedict, a writer, producer and actor, as well as an older sister, Bridget, a stage manager. Despite completing his senior secondary years at Rostrevor College, Samuel undertook final year drama at Christian Brothers College under the tutelage of Amanda Portus, where he played the part of Tom Snout (the wall) in Rostrevor College's production of Shakespeare's A Midsummer Night's Dream as well as playing "Belvile" in CBC's production of Aphra Behn's The Rover (The Banished Cavaliers). Xavier Samuel later attended Flinders University Drama Centre in 2005, where he studied under renowned acting teacher Professor Julie Holledge. He played Hamlet in the graduate production.