Tom Parker Bowles had a "nightmare holiday" with Queen Camilla and the rest of his family shortly after leaving school.