The Historic Dockyard is one of the most interesting places to explore. You can buy a yearly ticket for about £16 which allows you access to most of the attractions down there.
On a summer's, or even a sunny, but cold winter's day, walking from Eastney to Old Portsmouth to Gunwharf Quays is very nice. Takes around an hour. Along the way you have the Rose Gardens (spring/summer only), D-Day museum, Southsea Castle, Canoe lake, Blue Reef Aquarium, Gunwharf Quays (Cinema, Outlet shopping, restaurants), Spinnaker Tower and the fairground. There are places to stop for coffee, cake, ice cream, etc.