@OwainEvans_UK By random initialization do you mean the initial weights of the untrained model before pretraining?
same thread: 1947732531241255041 1947733079109357894 1947733863339569231 1947735173535027483 1947737645192491114 1947737987401584994 1947739273521991737 1947739644965360065 1947748393604026568 1947749011831853438 1947759872961089806 1947761637148266708 1947762454295154968 1947767313576694215 1947769325500436605 1947774909289242816 1948168296026788093 1948169109352616298 1948589330404179997
Reproduced against link rot, credited and linked to its original. Yours and you’d rather it weren’t here? Open an issue.