Not necessarily. It was definitely one of those matchups where you needed to know his Oga setups for your character. If he was fishing for those (and there were obvious tells in most cases), it was a matter of blocking normally, not as a crossup. Definitely not a near-unblockable.
And yeah, him getting a ton of damage was easy because he didn't have a reason to burn bar until Ultra, when they made the swap on his HK and EX Gekiro. Even then, he was still good.
EDIT: The only period where he suffered was during Super, when he lost a lot of his BS from vanilla and didn't get his bread and butters until AE.