We prove that independent families of permutation invariant random matrices are asymptotically free with amalgamation over the diagonal, both in expectation and in probability, under a uniform boundedness assumption on the operator norm. We can relax the operator norm assumption to an estimate on sums associated to graphs of matrices, further extending the range of applications (e.g., to Wigner matrices with exploding moments and the sparse regime of the Erdos–Rényi model). The result still holds even if the matrices are multiplied entrywise by random variables satisfying a certain growth condition (e.g., as in the case of matrices with a variance profile and percolation models). Our analysis relies on a modified method of moments based on graph observables.