Did you go and check to and see whether your "factor portfolios" would have had enough capacity to do what Buffett did? One problem with some quantitative analyses is that they assume infinite market depth, zero transaction costs, strategies have no market impact costs, etc.

In short, I think there are a lot of things that you missed.