The area around Foresthill has some nice spots, especially for mountain and road bikers.
At 39°09’10.5″N 120°46’37.7″W you’ll find fantastic spot on the edge of the canyon overlooking American river. It’s just off the road, which would likely have a few dirt bikers going by on weekends, but should be fine mid week. Roads in the area all great for mountain bikes, and none require 4WD (tho high clearance may be helpful). It’s a short bike ride to a really nice reservoir for swimming. You may want to stay at the Morning Star Lake Campground mid-week to kayak, or just use it for the day. Elevation around 5000ft.
At 39°00’11.4″N 120°43’57.0″W you find a boondocking area right on the Rubicon River. Its a long drive down a windy (but paved) road. Great area for road biking. River too cold for swimming.
I’ve also seen a number of people camping near the China Wall Staging Area at the end of the snow plowed portion of Foresthill Rd. Its not all that nice, but there are restrooms, and a good location to continue exploring the area.
There is also camping with restrooms at the Robinson Flat Campground.
Further out, the Talbot Campground is a great free free, small campground near a trailhead into the Granite Chief area.