The angles between the sides will not change as a and b change, since it is always an isosceles triangle with sides in the ratio 2,2,1. I have dropped an altitude to the midpoint of the base, giving us a right-angled triangle, whose smallest angle is x.
sin(x) = (ab/4)/ab = 1/4.
By Pythagoras the length of the altitude is ab*sqrt(15)/4.
cos(x) = ab*(sqrt(15)/4)/ab = sqrt(15)/4.
To find cos(2x) we can use the double angle identity:
cos(2x)=(cos(x))^2-(sin(x))^2 = 14/16 = 7/8.
Now focusing on the upper right triangle that has angle 2x, hypotenuse a(b-1) and adjacent side b(a-1).
cos = adj/hyp
7/8 = b(a-1)/(a(b-1))
7a(b-1) = 8b(a-1)
7ab-7a = 8ab-8b
8b-ab = 7a
b(8-a) = 7a
b = 7a/(8-a)
7a+ab = 8b
a(7+b) = 8b
a = 8b/(7+b)
Let’s see what happens when we let b increase. As b becomes massive the 7 in the denominator will become insignificant, and a will get closer and closer to 8, but never quite reach it, so an open upper bound for a is 8.
ab must be greater than a for the line across the triangle to be within the triangle, therefore an open lower bound for b is 1. This corresponds to an open lower bound for a of 1 also.
So the integer values that a could take are 2,3,4,5,6 or 7. Calculating the corresponding values of b, it is only an integer when a is 4, 6 or 7
The three integer solutions are a=4,b=7; a=6,b=21; a=7,b=49.