My son's team (13-14 yr olds) was playing a team that had only 9 players. The pitcher was thrown out of the game and at the time they did not have anyone on the bench so they played the next 2 innings with 8 taking an out in his spot. A player showed up in the 4th inning and they inserted him into the ejected player's spot in the batting order. Is this allowed?

Under OBR this is not allowed at all.
When a team is unable or refuses to field 9 players the game is forfeited.

Under Fed Rules it is permissible to continue the game with 8 players. The team takes an out each time that position in the batting order comes up. The team can not, however, return to 9 players later if more players show up.

I feel that I should add, though, that some summer leagues will allow a player to be added in later if the player who left the game went out due to injury, illness or other just cause.
Most won't allow a later replacement if the player was ejected for an unsportsmanlike act.

it is permissible for the late player to enter the game as a sub, but the team must continue to only use 8 players on offense and defense.