I dont have the chops to work out the math, but here's an idea that you may not have tried yet for an analytical formulation.  Since you are finding the bisector here, the problem is analogous to a reflection problem for light which uses Fermat's principle to derive the light reflection path.  See this illustration

http://hyperphysics.phy-astr.gsu.edu/hbase/phyopt/fermat.html.  

Your problem adds a bit of additional complexity in that the point x shown in the fermat principle diagram must lie on a circle of know radius r, but I believe that if you blend the equation of a circle and the equation from fermat's principle you can get it into an analytical form.  

So, I couldn't find an actual analytic solution to this. Tried again today at the cafe over a fresh ginger+darjeeling tea, but I get stuck every time. I put my dad on the case, he's a maths teacher.

In the meantime, I wrote a little numeric solver for this in a VB script. If the accuracy is not to your liking I can improve it. It only works (or rather, is only tested) on the World XY plane.

Hope it helps.

--

David Rutten

david@mcneel.com

Poprad, Slovakia

Ok, I think this might work for a geometrical construciton:

1) Notice that if the give circle were a line, the following construction would give the right point:

But it's not....so we need inversive geometry.

2) Construct a reference circle anywhere on the circumference of the original circle and invert it, along with the given points A and B.

3) Perform step 1 on the line L

4) The resulting point C', when inverted back should be the point needed....because although inversion distorts angles, it does preserve the cross ratio (see this theorem).

I haven't constructed it yet, so I hope it works! I'll post a definition soon....