This paper presents an overview of the challenges of producing a list of retrieval results ranked according to perceptual similarity. We explain some of the problems in using a metric to measure peceptual similarity, and consider the arguments for the desirability of metrics for retrieval. We discuss the use of broader definitions of betweenness to produce such a ranking of retrieval results. We propose some initial ideas of a notion of projective betweenness that makes explicit the intuition that two referents should be used when producing a similarity ranking, and indicate how it might be used in relevance feedback.