PEOPLE: He said, "I...really never considered myself a TV star. I always thought I was a neighbor who just came in for a visit"