FFT3DFilter

Abstract

author: Alexander G. Balakhnin aka Fizick
version: 1.9.1
download: http://avisynth.org.ru/
category: Misc Plugins
requirements: YV12 or YUY2 Colorspace
license: GPL

Introduction

FFT3DFilter is 3D Frequency Domain filter - strong denoiser and moderate sharpener.

Technical info

FFT3DFilter uses Fast Fourier Transform method for image processing in frequency domain. It is based on some advanced mathematical algorithmes of optimal filtration. It works not locally, but makes some delocalized (block) processing. In 3D mode, it results in effect similar to partial motion compensation. This filter can reduce noise without visible quality loss and artefactes, even with quite strong settings. It can greatly improve compression and reduce encoded file size.
Also it has option of limited sharpening without both noise amplifying and oversharpening (haloing).
Fiction? Try it yourself! :)

So, it is a spatial-temporal (3D) filter, and works as follows:

  1. get current and some previous frames;
  2. divide every frame to small overlapped blocks;
  3. get frequency spectrum by windowed forward 2D FFT transform of every block of these frames;
  4. apply some filter to frequency spectrum;
  5. optionally sharpen image (in 2D frequency domain) by amplifying of some frequencies;
  6. make inverse 2D FFT transform of cleaned spectrum for every block;
  7. do windowed summation of cleaned overlapped blocks to output frame.

Overlapped blocks and weighting windows are used to prevent blockiness (and ringing).
The blocks are overlapped by some value along their vertical and horizontal sizes.
The lesser overlap, the faster processing, but with more visible grid artifactes.

The overlapping definition is shown on picture.

overlap

Some analysis and synthesis weighting windows are used to get effective signal gain=1 after blocks summation.

Plugin has several processing modes at filter stage:

Using noise pattern

Since v1.5 it is possible to get noise pattern (shape) by spectrum analyisis of some empty block (without any objects), and then to reduce the noise with the same pattern in whole frame and in whole clip. It may be useful for removal of film (especialy amateur 8 mm) grain and analog TV capture interference.

Sharpening

At sharpening stage (after denoising) the plugin amplifies high spectrum (spatial, 2D) frequencies .
There is also sharpen-only mode without denoising (bt=-1).
Since version 1.1, some special limited sharpening method is used :

The sharpening strength is maximal for frequencies with middle-range amplitudes. Of course, you can control both these margins and general sharpening strength.

Since v.1.7, Gaussian High Pass Filter with variable cutoff frequency is used for sharpening.

Since v.1.9, plugin has special option dehalo for decreasing of strong frequencies, it may be used for spatial adaptive softening of oversharped pictures (halo removal). This mode may be combined with denoising and sharpening.

Syntax

FFT3DFilter(clip, float "sigma", float "beta", int "plane", int "bw", int "bh", int "bt", int "ow", int "oh", float "kratio", float "sharpen", float "scutoff", float "svr", float "smin", float "smax", bool "measure", bool "interlaced", int "wintype", int "pframe", int "px", int "py", bool "pshow", float "pcutoff", float "pfactor", float "sigma2", float "sigma3", float "sigma4", float "degrid", float "dehalo", float "hr", float "ht" )

All parameters are named.

Function parameters:

first parameter - input clip

sigma - given noise value for all (or highest) frequencies (float>0, default=2.0)
beta - noise margin (float>=1.0, default=1.0 for no noise left):
      control noise amount left in the restoration for Wiener filter, so max filter noise attenuation = (beta-1)/beta.
plane - processed color plane: 0 - luma(Y), 1 - chroma U, 2 - chroma V (default = 0)
bw - block width (integer, default = 48 since v.1.2)
bh - block height (integer, default = 48 since v.1.2)
bt - block temporal size, number of frames  (-1, 0, 1, 2 or 3, default = 3):
    0 - all previous frames (switch Kalman filter mode);
    1 - only current frame (spatial 2D Wiener filter);
    2 - previous and current frame (3D Wiener filter);
    3 - previous, current and next frame (3D Wiener filter)
    4 - two previous, current and next frame (3D Wiener filter)
    -1 - sharpen only (2D);
ow - overlap width (default=bw/3 since v.1.2)
oh - overlap height (default=bh/3 since v.1.2)
kratio - ratio of threshold to sigma to reset Kalman filter (default = 2.0):
      variation threshold = sigma*kratio, good value is about from 1.5 to 3.0;
sharpen - sharpening strength (default=0 - not sharpen)
      good values about 0.3 to 1.0 (negative values results in reverse effect)
scutoff - sharpening cutoff frequency, relative to max (default=0.3)
svr - sharpening vertical ratio (to horizontal) (default=1.0 - same as horizontal, 0 - no vertical sharpening)
smin - minimum limit (approximate noise margin) for sharpening stage (default=4.0)
smax - maximum limit (aproximate oversharping margin) for sharpening stage (default=20.0)
measure - select the most optimal (fastest) FFT method by speed measure (with longer init stage)
        instead of simple estimation (default=true since v.0.9.2)
interlaced - separate fields processing (default=false)
wintype - weighting windows type (default=0 again since v1.8):
    0 - same analysis and synthesis half-cosine window, used in all versions before 1.4;
    1 - intermediate between 0 and 2;
    2 - flat analysis window, rised cosine (Hanning) synthesis window.
pframe - noise pattern frame number (default=false)
px - noise pattern block horizontal X position (default=0)
py - noise pattern block vertical Y position (default=0)
    if px=px=0, then the pattern block is defined automatically with minimal power spectral density.
pshow - show noise pattern block and its properties (default=false)
pcutoff - noise pattern cutoff frequency (relative to max) (default=0.1)
pfactor - noise pattern denoise strength (0 to 1.0, default=0, this method disabled)
sigma2 - given noise value at second scale level frequencies (float>0, default=sigma)
sigma3 - given noise value at third scale level frequencies (float>0, default=sigma)
sigma4 - given noise value at lowest frequencies (float>0, default=sigma)
degrid - weigthing window compensation degree for grid decreasing (float>0, default=1.0)
dehalo - halo removal strength (float>0, default=0)
hr - halo approximate radius (float>0, default=2.0)
ht - halo approximate threshold (float>0, default=50.0)

The most important parameter is a given noise value sigma. You must use reasonable value based on a priori info for current clip.
Typical value for digital sources is about 1.5 to 2.5, and about 3 and above for analog captured video.
The good value of overlapping size is about quarter to half of block size. The half is the best, but slower.
Good values of block size are about 32 to 64.

Filter can produce some grid artifacts for large sigma and small bw, bh, ow, oh or small relative overlap sizes, especially with sharpening enabled.
Filter can produce ghosting for large sigma (and kratio) for 3D modes.

Weighting window wintype=0 can produce the worst grid artifactes, window type wintype=2 do not produce grid artifactes, but can produce some ringing, wintype=1 is intermediate case.

I recommend to use the weigthing window compensation with degrid=1 (since version 1.8), it improves the denoise quality and decreases the grid artifactes, especially for 2D.

Sharpening will result in worse clip compression.>

In order to use noise pattern method in place of ordinary (blind) method you must:
Firstly switch show mode pshow=true, and set some non-zero value pfactor=1.0.
Then select frame number and some block position, thus the block shown must not contain any objects beside typical noise pattern.
The switch off show mode pshow=false,
and set noise reduction strength, recomended value pfactor= 0.5 to 1.0.
The best windows type for this method is wintype=2.
The sigma and beta parameters are not used in this denoising method, but you can use this method in show mode pshow=true to estimate sigma value for ordinary denoising method (it is not strictly the same, but similar value).

There is also combined method, when you can directly set different noise values sigma for highest and sigma2, sigma3, sigma4 for lower frequencies. The pattern coefficients will be created internally from these sigmas values by interpolation. Set pfactor=0 for this method (internally it will be =1).

Features and limitations

  1. Filter works only in YV12 or YUY2 color format.
  2. Only specified single color plane is processed (Y, U or V).
  3. Since v1.3 it works with both progressive and interlaced clips.
  4. Tested with Avisynth v2.55, v2.56, v.2.5.7.
  5. Filter uses fast external FFTW library version 3 (http://www.fftw.org)
    as Windows binary DLL (compiled with gcc under MinGW by Alessio Massaro), which support for threads and have AMD K7 (3dNow!) support in addition to SSE/SSE2.
    It may be downloaded from ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip
    You MUST put FFTW3.DLL file from this package to some directory in path (for example, C:\WINNT\SYSTEM32).
    Filter will NOT work without it!
  6. FFTW works most efficiently for arrays whose size (bw, bh) can be factored into small primes (2, 3, 5, and 7), and otherwise it uses a slower general-purpose routine.
  7. First versions were really slooow! But recent versions are simple slow :-).
    Algorithm is improved and assembler 3DNow! and SSE instruction are used for some modes
  8. Algorithm is optimized by speed for forward sequential frames access only.
  9. The sharpening method is experimental, however is quite good since v1.1 (and v1.7).
  10. The noise pattern method is experimental too.
  11. degrid option is not implemented for Kalman filter.
  12. For noisy video it is useful to use median prefilter (for example DeGrainMedian) before FFT3Dfilter.

Simple sample scripts

To denoise progressive or fieldsbased (separated) clip luma:

Avisource("input.avi")
loadplugin("c:\plugins\fft3dfilter.dll")
FFT3DFilter(sigma=3)

Of course, you must replace "c:\plugins" by your plugins folder path (and set your video pathname too :). You can skip LoadPlugin command if you put fft3dfilter.dll file to autoloading plugins folder (C:\Program Files\Avisynth 2.5\plugins).

To sharpen only:

Avisource("input.avi")
loadplugin("fft3dfilter.dll")
FFT3DFilter(bt=-1, sharpen=0.7)

To denoise and slightly sharpen the interlaced clip:

Avisource("input.avi")
loadplugin("fft3dfilter.dll")
FFT3DFilter(sigma=3, sharpen=0.3, interlaced=true)

To denoise all color planes (both luma and chroma)

Note: The processing speed will be decreased more, so you can consider other (simpler and faster) filters using for chroma denosing (CNR2, DeGrainMedian, etc).

Correct way to denoise all color planes (script by AI)

Avisource("input.avi")
loadplugin("fft3dfilter.dll")
YToUV(fft3dfilter(sigma=3, plane=1).UToY,\
fft3dfilter(sigma=3, plane=2).VToY,\
fft3dfilter(sigma=2, plane=0))

It is especially important if you use motion compensation

Avisource("input.avi")
loadplugin("fft3dfilter.dll")
loadplugin("depan.dll")
motion=DepanEstimate(trust=2.5, fftw=true)
DepanInterleave(data=motion)
YToUV(fft3dfilter(sigma=3, plane=1).UToY,\
fft3dfilter(sigma=3, plane=2).VToY,\
fft3dfilter(sigma=2, plane=0))
SelectEvery(3,1)

New simplest way to denoise all color planes with same settings

Avisource("input.avi")
loadplugin("fft3dfilter.dll")
fft3dfilter(sigma=2, plane=4)

To decrease the existing horizontal halo only

Avisource("input.avi")
loadplugin("fft3dfilter.dll")
fft3dfilter(bt=-1, plane=0, dehalo=1.0, hr=2.0, ht=50, svr=0)

More info

The algorithm of Wiener filter is based on the 3D IIR/3D Frequency Domain Filter from:
MOTION PICTURE RESTORATION. by Anil Christopher Kokaram. Ph.D. Thesis. May 1993.
http://www.mee.tcd.ie/~ack/papers/a4ackphd.ps.gz in postscript format (use GSview with Ghostscript to read).
Search more info about Wiener and Kalman filters in special literature or Googles.

Filter discussion is at DOOM9 Avisynth forum, thread "New very slow FFT denoiser:"
http://forum.doom9.org/showthread.php?s=&threadid=85790.

There is also FFT3DGPUplugin (by tsp) with similar algo, but using modern videocard processor for FFT. tsp ported many fft3dfilter features. See http://forum.doom9.org/showthread.php?t=89941.

Version changes:

$Date: 2006/12/17 10:28:23 $