You can live in many of the nice neighborhoods of Port Jefferson or in some of the active adult communities in the area (see link at top right). Many more people live in surrounding towns like the Setaukets, Belle Terre (more upscale), Poquott, Mt. Sinai, and Miller Place
Median home prices are high in Port Jefferson, as are taxes. City-Data's reported sales that the median price of a home in 2012 was about $570,000.