The 1940s structure is just a private pathway away from the beach