Yep, remember that it is national park and that bouldering activities can have a heavy impact on vegetation, and the heathland vegetation in that area is some of Sydney's most accessible (and most accessed too due to the Spit to Manly walkway).
Lambertia formosa Mountain Devil - on the track to Grotto Point.
I love that area, and have been a 42 year local (hiya Dave K), so please consider minimal impact activities. There is some OK deep water soloing (when the tide is up) between Castle Rock Beach and Grotto Point - just follow the shoreline from Castle Rock.
The area also used to be heavily used by aborigines, and there is still some evidence of habitation in the area. I was there last October and 'rediscovered' (I remember it from when I was very young) the original aboriginal track used to access Washaway Beach - there was a very distinct rock shelter on the track with midden shells, fireplace and axe sharpening grooves in the rock (not to mention some spectacular rock art of animals closer to the carpark).