Giant Pigeon and Small Person: Prompting Visually Grounded Models about the Size of Objects