If the pair of lines joining the origin and the points of intersection of the line $ax+by=1$ and the curve $x^2+y^2-x-y-1=0$ are at right angles, then the locus of the point $(a,b)$ is a circle of radius

The equation of the line is given as $a x+b y=1$ and the
equation of the curve is given as, $x^2+y^2-x-y-1=0$
By homogenising both equations,

$x^2+y^2-x(ax+by)-y(ax+by)-(ax+by{)}^2=0$

$x^2+y^2-a{x}^2-bxy-axy-b{y}^2-\left(a^2{x}^2+b^2{y}^2+2abxy\right)=0$

$x^2\left(1-a-a^2\right)+xy(-a-b-2ab)+y^2\left(1-b-b^2\right)=0$

Since it is given that the pair of lines joining the origin and the points of intersection of the line and curve are at right angles.

Coefficient of $x^2+$ coefficient of $y^2=0$

$1-a-a^2+1-b-b^2=0$

$a^2+b^2+a+b-2=0$

The locus of the circle from the above equation $(a,b)$
Now,

$x^2+y^2+x+y-2=0$

On comparing in equation $\left(\mathrm{i}\right)$ with $x^2+y^2+2gx+2fy+c=0$. The radius is,

$r=\sqrt{g^2+f^2-c}$

$=\sqrt{{\left(\frac{1}{2}\right)}^2+{\left(\frac{1}{2}\right)}^2+2}$

$=\sqrt{\frac{1}{4}+\frac{1}{2}+2}$

$=\sqrt{\frac{5}{2}}$