The total number of matrices $A=\left(\begin{array}{ccc}0& 2y& 1\\ 2x& y& -1\\ 2x& -y& 1\end{array}\right), \left(x, y\in R, x\ne y\right)$ for which $A^{T}A=3I_3$ is:

Given $A^{T}A=3I_3$

$\left(\begin{array}{ccc}0& 2x& 2x\\ 2y& y& -y\\ 1& -1& 1\end{array}\right)\left(\begin{array}{ccc}0& 2y& 1\\ 2x& y& -1\\ 2x& -y& 1\end{array}\right)=\left(\begin{array}{ccc}3& 0& 0\\ 0& 3& 0\\ 0& 0& 3\end{array}\right)$

$\left(\begin{array}{ccc}8x^2& 0& 0\\ 0& 6y^2& 0\\ 0& 0& 3\end{array}\right)=\left(\begin{array}{ccc}3& 0& 0\\ 0& 3& 0\\ 0& 0& 3\end{array}\right)$

On comparing the two matrices, we get

$8x^2=3$ and $6y^2=3$

$\Rightarrow x^2=\frac{3}{8}$ and $y^2=\frac{1}{2}$

$\Rightarrow x=\pm \sqrt{\frac{3}{8}}$ and $y=\pm \sqrt{\frac{1}{2}}$

For each of $x \& y$ we have two possible values, hence there are total $2\times 2=4$ combinations of values are possible.

Hence, total $4$ matrices are possible.