The shape of your body is largely determined by your muscle mass, however that is assuming that you don’t have an excess of body fat distorting you true shape. The point of working out with weights is to manipulate your muscle mass to augment your look. Having fat covering all of those muscles makes your effort in the gym seem pointless.
Part of the reason fat distorts your shape is because it is not evenly distributed over you body. For example men typically build up significantly more fat around their abdomen and women typically store more fat around their hips and thighs.
If we gained fat evenly over our whole body then fat gain would look exactly like gaining muscle (the only difference would be less definition, but proportions would always be good)…this however isn’t the case. As we gain more and more fat both men and women start to look more like blobs with most of the fat stored closer to the middle of our bodies (abdomen and hips thighs) and thus proportions look worse and worse the more fat we gain.
The point of both gaining muscle and losing fat (besides any health benefit) is to improve our proportions for the aesthetic appeal.