Add rank column to table based on number of occurrences

I have a table and count the number of occurrences of a column value in a measure. This works fine.

CountOfColumn = COUNTROWS(FILTER(Table; Table[Column] = Table[Column]))

Now I would like to add a rank based on the count. so that the the entry with the most occurrences has rank 1, the next 2 etc.

This would make it possible to show the top n most frequent items.

I tried

RankByCountOfColumn = RANKX(Table; [CountOfColumn])

But the rank is always the same for each column although the counts are diffent.