Sonifying plants:
bioacoustic research with Plantsound

1.What is measured, and why through light

A plant is never still. Three classes of movement permanently overlap in foliage:

• Mechanical oscillations, from 0.1 to 3 Hz. Each leaf is a damped oscillator: a blade of mass m carried by a petiole of stiffness k. The slightest airflow, the convection of a radiator, a person walking across the room excites them. Their natural frequency depends on the plant's architecture: it is a morphological signature.
• Hydraulic movements, from a few tens of seconds to a few minutes. The turgor of motor cells, the opening and closing of stomata and water transport change the angle of the blades in slow steps. These variations follow the plant's water status.
• Growth movements, over hours: circumnutation, phototropism, nyctinasty. A stem permanently traces an ellipse around its axis; the leaves of many species rise in the evening and lower in the morning.

Commercial plant music devices measure an electrical current or resistance between two electrodes, one in the soil and one clipped to a leaf. This signal is real, but it is dominated by contact moisture, clip pressure and the static electricity of the room. Plantsound makes a different choice: it measures kinematics, the movement itself, without contact. It is the quantity most directly linked to the mechanical and water status of the plant, and the only one a phone measures without an accessory.

These movements are smaller than a pixel. A leaf 40 cm from the lens moves by a few tens of micrometres. No object tracking sees them. They do, however, change the light intensity received by the photosites lying on a leaf edge: an edge that slides by a tenth of a pixel changes the luminance of the pixel it crosses by nearly a tenth of its contrast. That change is what Plantsound measures.

2.The optical measurement chain

2.1 Sampling

Each frame is compressed by our algorithm before analysis. This is not a shortcut: for a mean luminance measurement, resolution brings nothing, whereas averaging many photosites reduces shot noise by the square root of the number of pixels. The sampling rate f_s is measured continuously on the frame clock, and all filters are recomputed when it changes: a phone that drops from 30 to 24 frames per second in low light does not shift the bands.

2.2 Sixty-four contrast-weighted zones

The image is divided into an 8 × 8 grid of zones. For each zone z, the mean luminance L_z(t) and the local contrast c_z, the mean gradient of the zone, are computed. A flat patch of leaf does not change luminance when it moves; only edges do. A zone whose contrast is below 2.5 grey levels is excluded from every decision: it will reveal no movement, however long it is observed.

2.3 Common-mode rejection

What varies in all zones at once is not the plant: it is automatic exposure, light flicker or a slight shift of the phone. Just as an electrocardiograph rejects mains hum, Plantsound subtracts from each zone the instantaneous mean of all zones:

2.4 The fast channel and its reference band

The signal u_z passes through two biquadratic band-pass filters (RBJ form):

• the useful band, centred on f₀ = 0.3 Hz with a quality factor Q = 0.4, a wide band covering most leaf oscillations between 0.08 and 2.5 Hz;
• the reference band, centred on 9 Hz with Q = 0.8. No plant moves at this frequency: everything in it is sensor noise. It serves as a yardstick.

The energies E_u and E_b of each band are integrated and then normalised by bandwidth. The reference band is fifteen times wider than the useful band; faced with white noise, it therefore collects fifteen times more energy through size alone. Comparing raw energies amounts to comparing window widths. Spectral densities are compared instead:

The curve that drives the music is the filtered output of the zone with the highest ratio R. It only switches zone when another one exceeds it by 35%: this hysteresis prevents the melody from jumping from one leaf to another at every fluctuation.

2.5 The slow channel

A biquad at 0.02 Hz sampled at 30 Hz is ill-conditioned: it drifts more than it filters. The slow channel therefore forms its band as the difference of two exponential moving averages, with time constants of 15 s and 120 s:

This initialisation detail is decisive. Starting from zero, the two averages converge towards the zone value at different speeds and draw a smooth ramp, identical in every zone. Neighbourhood correlation then reads 1.000: a perfect false positive. Any scene break (reframing, a light switched on) resets the slow channel and restarts its two-minute settling delay.

3.Qualifying the living

Detecting that something moves is not enough: a vibrating table, a fluctuating lamp or a breathing autofocus all produce energy in the useful band. What separates a plant from an object is the spatial structure of the movement.

3.1 Neighbourhood coherence

For each zone, the normalised correlation between its filtered signal and that of its immediate neighbours is computed, then the correlation with distant zones. The coherence retained for the scene is the median of these correlations over the usable zones. A median, not a mean: a reflection or a passing shadow in two zones carries away a mean, never a median.

3.2 Thresholds

The intensity that drives musical dynamics is the coherence mapped to the interval [0, 1]: a = min(1, C / 0.5).

3.3 The double gate

The movement measurement accepts anything that moves in a structured way; an image classifier readily recognises a plant in a photograph of a plant. The two measurements cover each other. The rule is asymmetric:

• to start, both plant movement AND a recognised plant are required;
• to continue, recognition is enough, with a 1.2 s grace delay that absorbs a flicker of the model.

This asymmetry matches biology: a plant keeps falling still, yet it remains itself, and its residual signal remains its own. Replacing the plant with a curtain does not extend the music: the model immediately stops recognising a plant.

4.Leaf colorimetry

For each zone, Plantsound computes the greenness index used in remote sensing (Green Leaf Index) from the mean red, green and blue channels:

The median GLI over the usable zones is displayed as a foliage indicator ("plant" criterion > 0.02). It is no longer a start condition: it rejected purple foliage (anthocyanins) and accepted a green curtain. Telling a plant from anything else is now the job of the vision model, which does it better. Colorimetry keeps an observational role and is the basis of a research hypothesis (§ 14, H5): the GLI of a given plant, tracked over time under stable lighting, measures yellowing before it is visible to the eye.

5.Recognising the morphology

Recognition runs entirely on the device. The model is BioCLIP-V0 (Apple + Neurosound), a contrastive image-text model. Its text half is used only once, offline: it converts the botanical labels of each family into vectors. The application only ships the visual half (21.8 MB) and the label vectors (a few tens of kilobytes). At each analysis, the image is projected into the same space and compared with the labels by cosine similarity:

This is zero-shot classification: no plant photo was used to train a specific layer. The profile is applied only when the verdict is clear, and it does not switch back at every analysis: two families neck and neck are worth no more than a coin toss, and the music must not change character every three seconds. No image leaves the phone for this step.

6.From signal to pitch

6.1 Normalisation by sliding bounds

A plant's signal has no absolute scale: its amplitude depends on distance, lighting and species. Plantsound normalises it with two bounds that widen instantly when the signal crosses them and narrow slowly otherwise:

The narrowing prevents an isolated burst from crushing everything that follows into the middle of the scale for good; its slowness prevents the bounds from catching up with the signal before it has finished rising. Both constants are set so that the melody covers the whole range over one minute of ordinary activity.

6.2 From degree to note

The degree d indexes the profile's scale, with n degrees per octave, above the tonic T and the base octave O:

The choice of scale settles a fundamental trade-off between fidelity and pleasantness. A chromatic scale follows the curve to the semitone; a pentatonic scale produces no dissonance, which makes any signal listenable, at the cost of a lower resolution. Plantsound settles this trade-off plant by plant (§ 9).

6.3 Intensity and space

7.Frequencies and tunings

The frequency of a note is computed from the tuning reference D, the frequency of A4:

The four tunings other than 440 come from wellness traditions: no controlled study attributes a specific physiological effect to them. They earn their place in Plantsound for a precise acoustic reason: they shift the whole score within the spectrum, and therefore its relationship to the resonances of the room and of any accompanying instrument. 417.60 Hz has a singular property: at −90.46 cents, its notes fall 9.5 cents above those of a 440 instrument one semitone lower. A B at 440 and a C at 417.60 sound almost in unison. The listening algorithm (§ 11) handles this ambiguity explicitly.

8.Musical time

8.1 Steps, density and silences

A profile's tempo is not a note rate: it is a step rate. At each step, spaced 60 / v seconds apart (v in steps per minute, with a minimum of 8), four filters decide whether the step sounds:

The more the plant moves, the more it plays. Silences are information: they are what distinguishes, by ear, an indoor tree from an aromatic herb.

8.2 Calm mode

An indoor plant, away from any draught, spends most of its time below the fast-channel threshold. Plantsound then switches to the slow channel, with asymmetric hysteresis:

• entry: 45 uninterrupted seconds of fast energy below 0.10, with the slow channel established;
• exit: immediate, as soon as fast energy reaches 0.18.

In calm mode, density drops to a quarter and the pitch of each note comes from the slow channel: notes arrive every five to fifteen seconds, at the pace at which turgor actually moves the blade. Energy is capped at 0.55 of the scale: still foliage never sounds as loud as agitated foliage, and the difference is audible.

9.From morphology to musical gesture

The profile table does not claim that a species "sings" in a given scale. It translates the form and physiological strategy of each family into a musical gesture, and each correspondence is justified in one sentence, displayed on screen. A succulent stores water, it is heavy and slow: low register, sustained notes, a continuous drone. A cactus is spiny and defensive: short, staccato notes separated by irregular silences. A fern repeats the same pattern at every scale (self-similar fronds): short three-note motifs, repeated higher and lower. A climber progresses: the register rises, then falls back. A moss, still to the eye, only varies at the margins: chromatic over a tiny range, so that semitones make audible deviations that any other scale erases.

Two regularities run through the table. Families with high structural investment and low metabolism (succulent, tree, moss) occupy the low register, play slowly and rest on a drone: their music has the inertia of their physiology. Families with thin, responsive organs (spiny cacti, aromatic herbs, flowers) occupy the high register and play fast: their music has the nervousness of their surface. The neutral profile is used when the family cannot be read: a pentatonic scale on handpan, which clashes with no signal.

Articulation matters more than scale. Two profiles sharing scale and timbre remain recognisable if one plays staccato and the other legato: it is the parameter that carries character.

10.The atmosphere

Plantsound queries an open weather service (Open-Meteo) every ten minutes, from a position rounded to a hundredth of a degree (about one kilometre). Six quantities are displayed; only two enter the sound. This split follows a physiological reasoning:

• Temperature and humidity already govern turgor and transpiration, and therefore the movement the camera measures. Injecting them a second time into the music counts the same cause twice.
• Atmospheric pressure and the UV index leave no trace in the movement of an indoor plant. They bring information the plant does not give. They touch neither the notes nor the tempo: they colour the room.

When a timbre has its own effects, the weather shifts them around a neutral point instead of overwriting them: reverb varies additively, cutoff multiplicatively, around 0.175 and 1975 Hz.

11.Listening to the room

The plant does not play alone: people often play with it. At start-up, then every sixty seconds of music, Plantsound falls silent and listens. The music fades out in 1.2 s, the microphone opens, and for at most ten seconds the application looks for a note sustained by a voice or an instrument. Falling silent to listen removes the echo problem at its root: a phone loudspeaker distorts and creates harmonics absent from the score, and no subtraction removes them with certainty.

11.1 Pitch by harmonic summation

The spectrum is computed over 8192 points (5.9 Hz per bin at 48 kHz, Blackman window). The noise floor is estimated from the spectrum itself, as the 20th percentile of a 48-bin neighbourhood, floored at −110 dB. Each bin receives an excess r_k above the floor plus 10 dB, capped at 40 dB. For each candidate fundamental frequency f₀, between 62 and 2100 Hz in steps of a third of a semitone:

11.2 Sustain and dominance

A frame only counts if it extends a note sustained for 0.25 s within half a semitone. A spoken syllable glides and lasts less: speech is rejected. A note is retained once it has accumulated three seconds and dominates the second one by a factor of 1.5. Two notes played equally designate no reference.

11.3 Reading the instrument's tuning

A tuned instrument shifts all its notes by the same offset from 440. This offset is measured over all sustained notes at once, through a circular mean with a period of 100 cents:

The note name is then read in the measured tuning, not in 440: at 426.70 Hz, C is 53 cents below standard C, closer to B. When two tunings compatible with the measurement name the notes differently (440 and 417.60 Hz), the doubt is settled in favour of the current tuning: the tuning does not flip from one listening window to the next. On synthetic signals and in the browser, the chain identifies a C at 440, 432 and 426.70, and a G at 429.62, with a tuning error below 3 cents.

12.The phone's sensors

Inertial sensors (accelerometer, gyroscope) do not enter the music. Common-mode rejection (§ 2.3) already absorbs the movements of a phone resting on a surface. Using them as an ego-motion reference is the next instrumental step (§ 14, H6).

13.Making it visible: cymatics and morphogenesis

Plantsound's animations react to the notes actually played, at the moment they are heard, not to the raw signal. Three of them translate the music into physical or botanical laws.

Cymatics. A free square plate vibrating in a mode (n, m) shows nodal lines where sand gathers. Their shape is given by:

Phyllotaxis. Each note gives birth to an element placed at the golden angle, 137.507°, from its predecessor: the arrangement of leaves and of sunflower seeds, which maximises exposure to light. Morphogenesis: a Mandelbulb whose power depends on the family (round lobes for a succulent, multiple spikes for a cactus), and whose phase advances with the notes.

14.Research programme

Plantsound measures, in thousands of homes, a quantity that plant biomechanics research usually observes in the laboratory with dedicated cameras: the fine kinematics of foliage. Each session produces a time-stamped series, associated with a family, a weather record and a position. We state six hypotheses that can be tested with this instrument.

The meaning of a sonification

A sonification is not a decoration of the signal: it is an instrument of analysis. The human ear detects a rhythmic regularity, a pitch drift or a change of texture over durations where the eye sees only a noisy curve. By translating plant kinematics into pitches, densities and silences, Plantsound makes perceptible phenomena that unfold at the edge of visibility. The next step is to close the loop: to measure whether prolonged exposure of a plant to its own sonification, through the vibration of air and support, in turn changes its kinematics. Thigmomorphogenesis establishes that repeated mechanical stimulation shortens and thickens stems; we predict that low-frequency acoustic stimulation, synchronised with the plant's movement, produces a measurable effect on petiole stiffness after four weeks. Plantsound provides both the stimulus and the measuring device.

References

1. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W. T. (2012). Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Transactions on Graphics, 31(4).
2. Louhaichi, M., Borman, M. M., Johnson, D. E. (2001). Spatially located platform and aerial photography for documentation of grazing impacts on wheat. Geocarto International, 16(1). GLI index.
3. Vasu, P. K. A., et al. (2024). MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training. CVPR.
4. Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML. CLIP model.
5. Darwin, C. (1880). The Power of Movement in Plants. London: John Murray. Circumnutation.
6. Moulia, B., Coutand, C., Julien, J.-L. (2015). Mechanosensitive control of plant growth: bearing the load, sensing, transducing, and responding. Frontiers in Plant Science, 6:52.
7. Braam, J. (2005). In touch: plant responses to mechanical stimuli. New Phytologist, 165(2). Thigmomorphogenesis.
8. Chladni, E. F. F. (1787). Entdeckungen über die Theorie des Klanges. Leipzig.
9. Douady, S., Couder, Y. (1992). Phyllotaxis as a physical self-organized growth process. Physical Review Letters, 68(13).
10. Hermann, T., Hunt, A., Neuhoff, J. G. (eds.) (2011). The Sonification Handbook. Berlin: Logos.
11. Bristow-Johnson, R. Audio EQ Cookbook: formulae for audio EQ biquad filter coefficients. Note published by the W3C Audio Working Group. RBJ filters.
12. Pollack, G. H. (2013). The Fourth Phase of Water. Seattle: Ebner & Sons. Origin of the H3O2− name.