University of Sussex
Browse

Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device

Download (396.64 kB)
journal contribution
posted on 2023-06-07, 18:22 authored by David J Brown, Tom MacPherson, Jamie WardJamie Ward
Sensory substitution devices convert live visual images into auditory signals, for example with a web camera (to record the images), a computer (to perform the conversion) and headphones (to listen to the sounds). In a series of three experiments, the performance of one such device (‘The vOICe’) was assessed under various conditions on blindfolded sighted participants. The main task that we used involved identifying and locating objects placed on a table by holding a webcam (like a flashlight) or wearing it on the head (like a miner’s light). Identifying objects on a table was easier with a hand-held device, but locating the objects was easier with a head-mounted device. Brightness converted into loudness was less effective than the reverse contrast (dark being loud), suggesting that performance under these conditions (natural indoor lighting, novice users) is related more to the properties of the auditory signal (ie the amount of noise in it) than the cross-modal association between loudness and brightness. Individual differences in musical memory (detecting pitch changes in two sequences of notes) was related to the time taken to identify or recognise objects, but individual differences in self-reported vividness of visual imagery did not reliably predict performance across the experiments. In general, the results suggest that the auditory characteristics of the device may be more important for initial learning than visual associations.

History

Publication status

  • Published

File Version

  • Published version

Journal

Perception

ISSN

0301-0066

Publisher

Pion Ltd

Issue

9

Volume

40

Page range

1120-1135

Department affiliated with

  • Psychology Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2012-08-09

First Open Access (FOA) Date

2016-03-22

First Compliant Deposit (FCD) Date

2017-03-07

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC