What sci-fi show is more realistic Battlestar Galactica or Babylon 5?
Great question but a difficult call. Obviously neither was "realistic" in any conventional sense - not on shows portraying FTL drives, jumpgates, aliens, human-level+ AI, etc. But it goes deeper. Both shows had elements of mysticism and "Clarke's Law" level technology (that is, events as easily explained by magic as technology - or vice versa). These tend to push the "science fantasy" classification for both shows which, almost by definition, reduces their realism or need for such.
Obviously it's not fair to consider the special effects. A decade of difference in technology plus a larger budget makes comparison at that level almost meaningless. Both had internal consistency issues, both had high and low points relative to the primary story arc. Acting was generally good in both but Battlestar Galactica again had advantages given its sponsoring network and intended audience - allowing more interesting and adult stories for some characters and events.
So... I'm going to mentally flip a coin and say Battlestar Galactica, despite its weak ending, narrowly beats Babylon 5 in the "hard SF" class and, by my way of thinking, is more "real", for good or bad, in its handling of the story and characters. I look forward to reading other opinions and reasoning.