PEOPLE: This movie star left his wife down under for actress Linda Kozlowski