I think your conclusion is the correct one. Alpha has always been a flawed concept. It is very easy to move the goal posts ex-post on which factors or models are included to "explain away" outperformance(of course active managers do the same thing by choosing inappropriate benchmarks too).

I have always favored absolute performance measures where outperformance or underperformance can be measured with a simple yes or no. While they all individually have flaws, looking at a mosaic of absolute rolling returns vs an appropriate bench, relative sharpe, relative sortino, and relative upside/downside will get you 90% of the way there. Factors/alpha can help you determine an appropriate benchmark, an attribution, and look for style consistency, but they are 100% the wrong approach to measuring manager skill in my opinion.