Class: Sonifier

Sonifier(config)

Image Sonifier

Generation of audio that represents an image.

Plays and holds notes on a synthesizer, then adjusts the volume of those notes depending on the brightness of red, green, and blue channels in each pixel in an image. By regularly re-sampling the image as it is changed, the sound heard will adjust as the relative colour brightnesses change within the currently visible image.

Source: GitHub repository

Requires Tone.js - <script src="https://cdnjs.cloudflare.com/ajax/libs/tone/13.1.4/Tone.js"></script>

Implementation notes

The default tones combine into a 'C 6/9' chord (which uses notes from the Pentatonic scale giving a drone-like sound):

  • Red: Major third
  • Green: Fifth
  • Blue: Minor third

Constructor

new Sonifier(config)

Constructs a Sonifier.

Parameters:
Name Type Description
config Object
Source:

Methods

(static) calculateBrightnessAverages(imagePixelData, imagePixelSamplingDistance) → {Object}

Calculates the average brightness of pixels in the supplied image data. Only samples every Nth pixel (in a grid across the whole image) to improve performance.

Parameters:
Name Type Description
imagePixelData ImageData

See: https://developer.mozilla.org/en-US/docs/Web/API/ImageData
imagePixelSamplingDistance Integer

How many pixels to sample, i.e. every Nth pixel
Source:
Returns:
  • properties for each channel (range 0-255): 'red', 'green', 'blue'
Type
Object

(static) getColorIndicesForCoord(x, y, imageWidth) → {Array}

Calculates the offset inside an ImageData array that the specified pixel data is. Returns an array of four offsets for the four bytes describing the single pixel (RGBA), e.g. getColorIndicesForCoord(x,y,w)[2] returns the pixel's blue value

See: https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Pixel_manipulation_with_canvas

Parameters:
Name Type Description
x Integer
y Integer
imageWidth Integer
Source:
Returns:
  • Absolute array offsets for the four bytes describing the single pixel (RGBA)
Type
Array

(static) toDecibelSine(value, dynamicRange) → {Number}

Calculates an appropriate decibel volume for the supplied value. Converts from pixel brightness (0 to 255) to decibel (-dynamicRange to 0).

Applies a sinusoidal curve so low & high values are compressed (less pronounced change) and mid-range values are expanded (more pronounced change). Rationale: brightness changes nearer very bright or very dark are less noticeable, so we want to reflect that in the resulting volume.

The default dynamic range is -30 to 0 dB. NB: -60 dB is absolute silence in Tone.js, we use -30 dB as the minimum so there is always some sound.

Parameters:
Name Type Description
value Number

Range: 0 (silence) - 255 (max)
dynamicRange Integer

Range of values to return to represent from silence to maximum volume: 0 - 60
Source:
Returns:
  • Range: -30 (silence) - 0 (max)
Type
Number

analyseCanvas(canvasElement, imageAnalysisStrategy)

Analyses once the image data in the HTML canvas in the specified HTML element and adjusts the appropriate sonifier volumes. NB: The sonifier must already be playing for any sound to be heard.

Supported image analysis strategies:

  • absolute = translate each channel brightness value to an equivalent volume value (default)
  • tiers = set the min/mid/max channel brightnesses to pre-defined volume tiers
Parameters:
Name Type Description
canvasElement Object

HTML Canvas Element
imageAnalysisStrategy String
Source:

analyseImageAbsolutes(imagePixelData)

Analyses once the supplied image data and adjusts the appropriate sonifier volumes. NB: The sonifier must already be playing for any sound to be heard.

Sets each channel volume based on its absolute average brightness.

This strategy is accurate, but many images contain such a range of colours that the channel averages end up in a similar range (50-180). We use a sinusoidal compression to expand this mid-range, but often the volume changes still don't seem very pronounced.

Parameters:
Name Type Description
imagePixelData ImageData

See: https://developer.mozilla.org/en-US/docs/Web/API/ImageData
Source:

analyseImageTiers(imagePixelData)

Analyses once the supplied image data and adjusts the appropriate sonifier volumes. NB: The sonifier must already be playing for any sound to be heard.

Has three set volume tiers (defined in Sonifier.tierVolumes) and uses only those volume levels for the RGB channel volumes based on which is brightest, middle, and dimmest. In addition, an overall volume is set based on the absolute brightness of the brightest channel - this provides some variation for black & white images (which have the same brightness for all three RGB channels).

This strategy gives more obvious volume changes, but is fairly artificial (e.g. all three may be similar brightness but the one slightly brighter gets a very high volume).

Parameters:
Name Type Description
imagePixelData ImageData

See: https://developer.mozilla.org/en-US/docs/Web/API/ImageData
Source:

play()

Starts playing the synthesizer notes for all (red/green/blue) channels.

Source:

playBlue()

Starts playing the synthesizer notes for the blue channel.

Source:

playGreen()

Starts playing the synthesizer notes for the green channel.

Source:

playRed()

Starts playing the synthesizer notes for the red channel.

Source:

pollCanvasStart(canvasElement, intervalSecs)

Starts regular analysis of the specified HTML canvas element at the specified time interval. NB: The sonifier must already be playing for any sound to be heard.

Parameters:
Name Type Default Description
canvasElement Object
intervalSecs Number 2

How often to analyse the canvas in seconds
Source:

pollCanvasStop()

Stops the regular analysis of the specified HTML canvas element. NB: The sonifier will continue to play the existing notes until stopped.

Source:

release()

Stops playing the synthesizer notes for all (red/green/blue) channels.

Source:

releaseBlue()

Stops playing the synthesizer notes for the blue channel.

Source:

releaseGreen()

Stops playing the synthesizer notes for the green channel.

Source:

releaseRed()

Stops playing the synthesizer notes for the red channel.

Source:

volumeBlue(vol)

Changes the blue channel synthesizer to the specified volume.

Parameters:
Name Type Default Description
vol Number 100

New volume: 0 (silence) - 255 (max)
Source:

volumeGreen(vol)

Changes the green channel synthesizer to the specified volume.

Parameters:
Name Type Default Description
vol Number 100

New volume: 0 (silence) - 255 (max)
Source:

volumeMaster(vol)

Changes the master volume to the specified volume.

Parameters:
Name Type Description
vol Number

New volume: -60 (silence) - 0 (max)
Source:

volumeRed(vol)

Changes the red channel synthesizer to the specified volume.

Parameters:
Name Type Default Description
vol Number 100

New volume: 0 (silence) - 255 (max)
Source: