(A) 5

(B) 10

(C) 8

(D) 13

Correct Answer: 10

Explanation

Angle $ABC$ is formed by the lines $BA$ and $BC$.

Equation of line $BA$ through $B(2,-1)$ and $A(1,0)$:

$$ \frac{y+1}{0+1}=\frac{x-2}{1-2} \Rightarrow y+1=-(x-2) $$

$$ x+y-1=0 \quad (1) $$

Equation of line $BC$ through $B(2,-1)$ and $C\left(\frac73,\frac43\right)$:

$$ \frac{y+1}{\frac43+1}=\frac{x-2}{\frac73-2} \Rightarrow \frac{y+1}{\frac73}=\frac{x-2}{\frac13} $$

$$ y+1=7(x-2) $$

$$ 7x-y-15=0 \quad (2) $$

Using the angle bisector formula for lines

$$ \frac{a_1x+b_1y+c_1}{\sqrt{a_1^2+b_1^2}} =\frac{a_2x+b_2y+c_2}{\sqrt{a_2^2+b_2^2}} $$

Substitute from (1) and (2):

$$ \frac{x+y-1}{\sqrt{2}} =\frac{7x-y-15}{\sqrt{50}} $$

Cross multiplying:

$$ 5(x+y-1)=7x-y-15 $$

$$ 5x+5y-5=7x-y-15 $$

$$ 2x-6y-10=0 $$

$$ x-3y=5 $$

Comparing with $\alpha x+\beta y=5$:

$$ \alpha=1,\quad \beta=-3 $$

Hence,

$$ \alpha^2+\beta^2=1+9=\boxed{10} $$

Related JEE Main Mathematics Questions