flucoma-sc/release-packaging/HelpSource/Classes/FluidHPSS.schelp

TITLE:: FluidHPSS
SUMMARY:: Harmonic-Percussive Source Separation Using Median Filtering
CATEGORIES:: Libraries>FluidDecomposition
RELATED:: Guides/FluCoMa, Guides/FluidDecomposition

A FluidHPSS object performs Harmonic-Percussive Source Separation (HPSS) on the an audio input. The class performs HPSS as described in its original form footnote::
Fitzgerald, Derry. 2010. ‘Harmonic/Percussive Separation Using Median Filtering’. In Proceedings DaFx 10. https://arrow.dit.ie/argcon/67.
:: as well as a variation on the extension propsoed by Driedger et al. footnote::
Driedger, Jonathan, Meinard Uller, and Sascha Disch. 2014. ‘Extending Harmonic-Percussive Separation of Audio Signals’. In Proc. ISMIR. http://www.terasoft.com.tw/conf/ismir2014/proceedings/T110_127_Paper.pdf.
::

The algorithm takes an audio in, and divides it into two or three outputs, depending on the mode: LIST::
	## an harmonic component;
	## a percussive component;
	## a residual of the previous two if the flag is set to inter-dependant thresholds. See the modeFlag below.::

It is part of the Fluid Decomposition Toolkit of the FluCoMa project. footnote::
This  was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).
::

More information on median filtering, and on HPSS for musicianly usage, are availabe in LINK::Guides/FluCoMa:: overview file.
CLASSMETHODS::

METHOD:: ar
	The audio rate version of the object.

ARGUMENT:: in
	The input to be processed.

ARGUMENT:: harmFiltSize
	The size, in spectral frames, of the median filter for the harmonic component. Must be an odd number, >= 3.

ARGUMENT:: percFiltSize
	The size, in spectral bins, of the median filter for the percussive component. Must be an odd number, >=3

ARGUMENT:: modeFlag
	The way the masking is applied to the original spectrogram. (0,1,2)
	table::
	## 0 || The traditional soft mask used in Fitzgerald's original method of 'Wiener-inspired' filtering. Complimentary, soft masks are made for the harmonic and percussive parts by allocating some fraction of a point in time-frequency to each. This provides the fewest artefacts, but the weakest separation. The two resulting buffers will sum to exactly the original material.
	## 1 || Relative mode - Better separation, with more artefacts. The harmonic mask is constructed using a binary decision, based on whether a threshold is exceeded at a given time-frequency point (these are set using htf1, hta1, htf2, hta2, see below). The percussive mask is then formed as the inverse of the harmonic one, meaning that as above, the two components will sum to the original sound.

	## 2 || Inter-dependent mode - Thresholds can be varied independently, but are coupled in effect. Binary masks are made for each of the harmonic and percussive components, and the masks are converted to soft at the end so that everything null sums even if the params are independent, that is what makes it harder to control. These aren't guranteed to cover the whole sound; in this case the 'leftovers' will placed into a third buffer.
	::

ARGUMENT:: htf1
	In modes 1 and 2, the frequency of the low part of the threshold for the harmonic filter (0-1)

ARGUMENT:: hta1
	In modes 1 and 2, the threshold of the low part for the harmonic filter. That threshold applies to all frequencies up to htf1: how much more powerful (in dB) the harmonic median filter needs to be than the percussive median filter for this bin to be counted as harmonic.

ARGUMENT:: htf2
	In modes 1 and 2, the frequency of the hight part of the threshold for the harmonic filter. (0-1)

ARGUMENT:: hta2
	In modes 1 and 2, the threshold of the high part for the harmonic filter. That threshold applies to all frequencies above htf2. The threshold between htf1 and htf2 is interpolated between hta1 and hta2. How much more powerful (in dB) the harmonic median filter needs to be than the percussive median filter for this bin to be counted as harmonic.

ARGUMENT:: ptf1
	In mode 2, the frequency of the low part of the threshold for the percussive filter. (0-1)

ARGUMENT:: pta1
	In mode 2, the threshold of the low part for the percussive filter. That threshold applies to all frequencies up to ptf1. How much more powerful (in dB) the percussive median filter needs to be than the harmonic median filter for this bin to be counted as percussive.

ARGUMENT:: ptf2
	In mode 2, the frequency of the hight part of the threshold for the percussive filter. (0-1)

ARGUMENT:: pta2
	In mode 2, the threshold of the high part for the percussive filter. That threshold applies to all frequencies above ptf2. The threshold between ptf1 and ptf2 is interpolated between pta1 and pta2. How much more powerful (in dB) the percussive median filter needs to be than the harmonic median filter for this bin to be counted as percussive.

ARGUMENT:: winSize
	The window size in samples. As HPSS relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty

ARGUMENT:: hopSize
	The window hop size in samples. As HPSS relies on spectral frames, we need to move the window forward. It can be any size but low overlap may create audible artefacts.

ARGUMENT:: fftSize
	The inner FFT/IFFT size. It should be at least 4 samples long; at least the size of the window; and a power of 2. Making it larger than the window size provides interpolation in frequency.


RETURNS::
	An array of three audio streams: [0] is the harmonic part extracted, [1] is the percussive part extracted, [2] is the rest. The latency between the input and the output is ((harmFiltSize + (winSize / hopSize) - 1) *  hopSize) samples.


Discussion::
	HPSS works by using median filters on the spectral magnitudes of a sound. It hinges on a simple modelling assumption that tonal components will tend to yield concentrations of energy across time, spread out in frequency, and percussive components will manifest as concentrations of energy across frequency, spread out in time. By using median filters across time and frequency respectively, we get initial esitmates of the tonal-ness / transient-ness of a point in time and frequency. These are then combined into 'masks' that are applied to the orginal spectral data in order to produce a separation.

	The modeFlag parameter provides different approaches to combinging estimates and producing masks. Some settings (especially in modes 1 & 2) will provide better separation but with more artefacts. These can, in principle, be ameliorated by applying smoothing filters to the masks before transforming back to the time-domain (not yet implemented).


EXAMPLES::

CODE::
	//load a soundfile to play
	b = Buffer.read(s,File.realpath(FluidHPSS.class.filenameSymbol).dirname.replace("Classes","AudioFiles/Tremblay-AaS-SynthTwoVoices-M.wav"));

	// run with basic parameters (left is harmonic, right is percussive)
	{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1))}.play

	// run in mode 1
	{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),17,31,1,0.05,40,0.1,-40)}.play

	// run in mode 2m listening to
	//the harmonic stream
	{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),17,31,2,0.05,40,0.1,-40, 0.1, -10, 0.2, 10)[0].dup}.play
	// the percussive stream
	{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),17,31,2,0.05,40,0.1,-40, 0.1, -10, 0.2, 10)[1].dup}.play
	// the residual stream
	{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),17,31,2,0.05,40,0.1,-40, 0.1, -10, 0.2, 10)[2].dup}.play

	// null test (the process add a latency of ((harmFiltSize + (winSize / hopSize) - 1) *  hopSize) samples
	{var sig = PlayBuf.ar(1,b.bufnum,loop:1); [FluidHPSS.ar(sig,17,31, winSize:1024,hopSize:512,fftSize:2048).sum - DelayN.ar(sig, 1, ((31 + 1) *  512 / s.sampleRate))]}.play
::