Never Grow object inside the loop via c, cbind, rbind. The object will be copied every time. Instead, predefine the correct size (or overestimate a little if the result is fluid).
data.table .
library(data.table)
DT <- data.table(data)
DT[ref != target, list(bestdist = min(distance)), by = ref]
ref target ( ), ,
DT[as.character(ref) != as.character(target), list(bestdist = min(distance)), by = ref]