I'd suggest going when they first open or shortly thereafter because you can take the entire day to see everything...the main estate, the gardens, conservatory and eating & shopping there. The parking lot is conveniently located to where the trams can pick you up or you can enjoy the scenery and take the walk to the main house entrance.
The Walled Garden is a 4 acre garden that utilizes the style of gardening near the close of the 19th Century. Each season the colors and details of the Walled garden change.
The Conservatory is located near the Walled Garden. The Conservatory features a glass-roofed building to grow and raise orchids, ferns, exotic palms as well as providing all the plants and flowers for the main Estate.
From the main Estate, you can walk to the South Terrace and continue on to the various gardens. View walking from the main house to the South Terrace area.