Image segmentation

This note presents the design of a scalable software package named ImagePy for analysing biological images. Our contribution is concentrated on facilitating extensibility and interoperability of the software through decoupling the data model from the user interface. Especially with assistance from the Python ecosystem, this software framework makes modern computer algorithms easier to be applied in bioimage analysis.

It implements the template matching function from the OpenCV library. The java interface of OpenCV was done through the javacv library. It is quite similar as the existing template matching plugin but runs much faster and users could choose among six matching methods: 

1.Squared difference

2.Normalized squared difference

3.Cross-correlation

4.Normalized cross-correlation

5.Correlation coefficient

6.Normalized correlation coefficient

The detailed algorithms could be found here.

The cvMatch_Template will search a specific object (image pattern) over an image of interest by the user-specified method. 

Neuroconductor is an open-source platform for rapid testing and dissemination of reproducible computational imaging software, specialized in brain medical imaging (MRI, fMRI, DTI, etc...) but that could be used on a wider range of images. The goals of the project are to:

• provide a centralized repository of R software dedicated to image analysis;
• disseminate quickly software updates;
• educate a large, diverse community of scientists using detailed tutorials and short courses;
• ensure quality via automatic and manual quality controls; and
• promote reproducibility of image data analysis.

Based on the programming language R, Neuroconductor starts with 68 inter-operable packages that cover multiple areas of imaging including visualization, data processing and storage, and statistical inference. Neuroconductor accepts new R package submissions, which are subject to a formal review and continuous automated testing.

CaPTk is a software platform for analysis of radiographic cancer images, currently focusing on brain, breast, and lung cancer. CaPTk integrates advanced, validated tools performing various aspects of medical image analysis, that have been developed in the context of active clinical research studies and collaborations toward addressing real clinical needs. With emphasis given in its use as a very lightweight and efficient viewer, and with no prerequisites for substantial computational background, CaPTk aims to facilitate the swift translation of advanced computational algorithms into routine clinical quantification, analysis, decision making, and reporting workflow. Its long-term goal is providing widely used technology that leverages the value of advanced imaging analytics in cancer prediction, diagnosis and prognosis, as well as in better understanding the biological mechanisms of cancer development.

NiftyNet is a TensorFlow-based open-source convolutional neural networks (CNNs) platform for research in medical image analysis and image-guided therapy. NiftyNet’s modular structure is designed for sharing networks and pre-trained models. Using this modular structure you can:

• Get started with established pre-trained networks using built-in tools;
• Adapt existing networks to your imaging data;
• Quickly build new solutions to your own image analysis problems.

HyphaTracker propose a workflow for time-resolved analysis of conidia germination. Each part of this workflow can also be used independnatly , as a toolbox. It has been tested on bright-field microscopic images of conidial germination. Its purpose is mainly to identify the germlings and to remove crossing hyphae, and measure the dynamics of their growth.

LimeSeg: A coarsed-grained lipid membrane simulation for 3D image segmentation **Download instruction:** There is no download but you can easily install this plugin via ImageJ update site. If you really need to download the jar file, access the file in the update site repository ([Link]())

"An open source machine learning framework for everyone "

TensorFlow™ is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains.

The MIPAV (Medical Image Processing, Analysis, and Visualization) application enables quantitative analysis and visualization of medical images of numerous modalities such as PET, MRI, CT, or microscopy. Using MIPAV's standard user-interface and analysis tools, researchers at remote sites (via the internet) can easily share research data and analyses, thereby enhancing their ability to research, diagnose, monitor, and treat medical disorders.

Multicut workflow for large connectomics data. Using luigi for pipelining and caching processing steps. Most of the computations are done out-of-core using hdf5 as backend and implementations from nifty

ANTs computes high-dimensional mappings to capture the statistics of brain structure and function.

Image Registration

Diffeomorphisms: SyN, Independent Evaluation: Klein, Murphy, Template Construction (2004)(2010), Similarity Metrics, Multivariate registration, Multiple modality analysis and statistical bias

Image Segmentation

Atropos Multivar-EM Segmentation (link), Multi-atlas methods (link) and JLF, Bias Correction (link), DiReCT cortical thickness (link), DiReCT in chimpanzees

vmtk is a collection of libraries and tools for 3D reconstruction, geometric analysis, mesh generation and surface data analysis for image-based modeling of blood vessels.

vmtk is composed of

• C++ classes (VTK and ITK -based algorithms)
• Python classes (high-level functionality - each class is a script)
• PypeS - Python pipeable scripts, a framework which enables vmtk scripts to interact with each other

SliceMap

Whole brain tissue slices are commonly used in neurobiological research for analyzing pathological features in an anatomically defined manner. However, since many pathologies are expressed in specific regions of the brain, it is necessary to have an annotation of the regions in the brain slices. Such an annotation can be done by manual delineation, as done most often, or by an automated region annotation tool.

SliceMap is a FIJI/ImageJ plugin for automated brain region annotation of fluorescent brain slices. The plugin uses a reference library of pre-annotated brain slices (the brain region templates) to annotate brain regions of unknown samples. To perform the region annotation, SliceMap registers the reference slices to the sample slice (using elastic registration plugin BUnwarpJ) and uses the resulting image transformations to morph the template regions towards the anatomical brain regions of the sample. The resulting brain regions are saved as FIJI/ImageJ ROI’s (Regions Of Interest) as a single zip-file for each sample slice.

More information can also be found in "SliceMap: an algorithm for automated brain region annotation", Michaël Barbier, Astrid Bottelbergs, Rony Nuydens, Andreas Ebneth, Winnok H De Vos, Bioinformatics, btx658, https://doi.org/10.1093/bioinformatics/btx658

SuRVoS: Super-Region Volume Segmentation workbench

A volume is first partitioned into Super-Regions (superpixels or supervoxels) and then interactively segmented by the user providing training annotations. SuRVoS can then learn from and extend the annotations to the whole volume.

This project was designed for vectorize and analyze the  blood vessels in the mouse brain.

This plugin requires the definition of seed point detection settings by the user (Semi-automated).

We have developed a novel approach, named DF-Tracing, to tackle this challenge. This method first extracts the neurite signal (foreground) from a noisy image by using anisotropic filtering and automated thresholding. Then, DF-Tracing executes a coupled distance-field (DF) algorithm on the extracted foreground neurite signal and reconstructs the neuron morphology automatically. Two distance-transform based “force” fields are used: one for “pressure”, which is the distance transform field of foreground pixels (voxels) to the background, and another for “thrust”, which is the distance transform field of the foreground pixels to an automatically determined seed point. The coupling of these two force fields can“push” a “rolling ball” quickly along the skeleton of a neuron, reconstructing the 3D cell morphology.

EBImage provides general purpose functionality for image processing and analysis. In the context of (high-throughput) microscopy-based cellular assays, EBImage offers tools to segment cells and extract quantitative cellular descriptors. This allows the automation of such tasks using the R programming language and facilitates the use of other tools in the R environment for signal processing, statistical modeling, machine learning and visualization with image data.

EBImage is available through the Bioconductor software project (www.bioconductor.org). Strengths Lightweight Suitable for automated, scripted analyses All functions are documented with examples Modular links to R and Bioconductor software, notably imageHTS and cellHTS2 Community support via the Bioconductor mailing list Reproducible (image) analysis using the Sweave report-writing system

Workflow of data-analysis strategies for image-based cell profiling.

1. Image analysis (image correction, image segmentation, feature extraction)
2. Image quality control
3. Preprocessing extracted features
4. Dimensionality reduction
5. Single-cell data aggregation
6. Measuring profile similarity
7. Assay quality assessment
8. Downstream analysis

Examples of publications where the workflow has been used at least partly:

• Image-based multivariate profiling of drug responses from single cells
• Phenotypic profiling of the human genome by time-lapse microscopy reveals cell division genes
• Mapping genetic interactions in human cancer cells with RNAi and multiparametric phenotyping
• Cell shape and the microenvironment regulate nuclear translocation of NF‐κB in breast epithelial and tumor cells
• Genome-wide RNAi Screening Identifies Protein Modules Required for 40S Subunit Synthesis in Human Cells
• Cell Painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes
• Systematic morphological profiling of human gene and allele function via Cell Painting

Microscopy Image Browser (MIB) is a high-performance Matlab-based software package for advanced image processing, segmentation and visualization of multi-dimensional (2D-4D) light and electron microscopy datasets.

MIB is a freely available, user-friendly software for effective image processing of multidimensional datasets that improves and facilitates the full utilization of acquired data and enables quantitative analysis of morphological features. Its open-source environment enables fine tuning and possibility of adding new plug-ins to customize the program for specific needs of any research project.

Spot detector detects and counts spots, based on wavelet transform.

- Detects spots in noisy images 2D/3D.
- Depending on objective, spots can be nuclei, nucleus or cell
- Versatile input: sequence or batch of file.
- Detects spot in specific band/channel.
- Multi band labeling: automaticaly creates ROIs from one band and count in the same or an other band.
- Filters detection by size.
- Sort detection by ROIs
- Output data in XLS Excel files: number of detection by ROIs, and each detection location and size.
- Outputs withness image with ROIs and detection painted on it.
- Outputs binary detection image.
- Displays detections
- Displays tags

This is the "prediction step" of the Pyxit segmentation model builder. It is a learnable segmentation algorithm based on ground-truth images and segmentation mask. It learns a multiple output pixel classification algorithm. It downloads from Cytomine-Core annotation images+alphamasks from project(s), build a segmentation (pixel classifier) model which is saved locally. Typical application: tumor detection in tissues in histology slides. 

This is a learnable segmentation algorithm based on ground-truth images and segmentation mask. It learns a multiple output pixel classification algorithm. It downloads from Cytomine-Core annotation images+alphamasks from project(s), build a segmentation (pixel classifier) model which is saved locally. Typical application: tumor detection in tissues in histology slides. It is based on "Fast Multi-Class Image Annotation with Random Subwindows and Multiple Output Randomized Trees" http://orbi.ulg.ac.be/handle/2268/12205 and was used in "A hybrid human-computer approach for large-scale image-based measurements using web services and machine learning" http://orbi.ulg.ac.be/handle/2268/162084?locale=en

This is the source code and data page of a distributed parallel algorithm 2683 for segmentation of large fluorescence microscopy images.

Image-processing algorithms developed at the MOSAIC Group for fluorescence microscopy. Tools included:

• 2D/3D single-particle tracking tool which can be used to track bright spots in 2D/3D movies over time.
• Optimal filament segmentation of 2D images. 
• Curvature filters for image filtering, denoising, and restoration. 
• Image naturalization for image enhancement based on gradient statistics of natural-scence images. 
• Tool for automatically send and distribute jobs on clusters and get back the results.
• Multi-region image segmentation of 2D and 3D images without needing to know the number of regions beforehand. 
• Squassh for globally optimal segmentation of piecewise constant regions in 2D and 3D images and for object-based co-localization analysis. 
• Tool for inferring spatial interactions between patterns of objects in images or between coordinates read from a file.
• Tool for robust, histogram-based background subtraction well suited to correct for inhomogeneous illumination artifacts.
• A tool to estimate the Point-Spread Function of the microscopy out of 2D fluorescence images.
• A tool to measure the 3D Point-Spread Function of a confocal microscope from an image stack.
• Addition of synthetic Poisson-distributed noise to an image in order to simulate shot noise of various signal-to-noise ratios. 
• Convolution of an image with a Bessel function in order to simulate imaging with a microscope. 
• A utility to detect bright spots in images and estimate their center. 
• A utility to create manual segmentations to be used as ground truth to test and benchmark automatic segmentation algorithms.
• A tool for replacing one color in an image with another color.

Free-D is a three-dimensional (3D) reconstruction and modeling software. It allows to generate, process and analyze 3D point and surface models from stacks of 2D images. Free-D is an integrated software, offering in a single graphical user interface all the functionalities required for 3D modeling. It runs on Linux, Windows, and MacOS. Free-D is developed by the Modeling and Digital Imaging team of the Institut Jean-Pierre Bourgin, INRA Versailles, France.

Free-D (http://free-d.versailles.inra.fr/) is a 3D reconstruction and modeling software. It is multiplatform, free (but not open source) tool for academic research and teaching.

Here is how to proceed, using Free-D:

1. Segmentation:

* load (a collection of) individual 3d stacks

* (optional for serial sections) perform a 2D registration to align image slices

* segment/reconstruct 3D contours using snakes

* segment 3D spots

2. Construct average cell:

* normalize the contours to compute a average cell, by registering/warping 3D contours/surfaces

3. Quantification:

* project each individual cell to the average one

* build density maps to analyze (cartography)

A few notes for current software version (till 10/2016):

* input file format: tiff (not able to import bioformats)

* currently results are saved in customized format, but there is an exportor to convert this format into fiji readable one

* import already generated contours is on the software's TODO list

It is a tool to visualize and annotate volume image data of electron microscopy. Users can annotate objects (e.g. neurons) and skeleton structures. It provides the ability to overlaying the image data with user annotations, representing the spatial structure and the connectivity of labeled objects, and displaying a three dimensional model of it. It can be extended by plugins written in python. A similar, web-based implementation is being developed at webknossos.info. Example datasets are also available.

MorphoLibJ is a library of plugin for ImageJ with functionalities for image processing such as filtering, reconstructing, segmenting, etc... Tools are based on Mathematical morphology with more rigorous mathematical approach than in the standard tools of ImageJ in particular for surface (or perimeter) measurements which are usually based on voxel counting.  

http://imagej.net/MorphoLibJ#Measurements

Among the features:

Morphological operations :  Dilation, Erosion, Opening,  Closing , Top hat (white and black), Morphological gradient (aka Beucher Gradient), Morphological Laplacian, Morphological reconstruction, Maxima/Minima , Extended Maxima/Minima -Watershed (classic or controlled) -Image overlay -Image labelling -Geodesic diameter -Region Adjacency Graph -Granulometry curves, morphological image analysis.

When trying to isolate objects, one strategy might be to use regular morphological operations (opening/closing) to remove small objects that are not of interest. In case small objects are made of a large number of pixels, this operation might impair the remaining objects' contours. An alternative strategy might be to use morphological reconstruction. In short, seed is placed on the image, on objects, then conditional dilation is performed from those seeds.

Here is how to proceed, using MorphoLibJ:

1. Open an image
2. Use the multi-point selection tool and place seeds on objects of interest
3. Create a new image of same size, black background
4. Transfer the selection to the new image (Edit/Selection/Restore selection)
5. Draw (make sure you're using white foreground) the multiple point selection
6. Launch the Morphological reconstruction plugin: Plugins > MorphoLibJ > Morphological reconstruction

ABSnake can segment complex structures in 2D images as well as 3D or temporal images. It uses a new active contour model based on a geometrical approach for correctly following invaginated structures.

MosaicIA is a tool to analyze the spatial distribution of objects in images. It estimates from an observed particle or object distribution what hypothetical interaction between the objects is most likely to have created this distribution.

In this case study, MATLAB, the Image Processing and Signal Processing toolboxes were used to determine the green intensities from a small portion of a microarray image containing 4,800 spots. A 10x10 pattern of spots was detected by averaging rows and columns to produce horizontal and vertical profiles. Periodicity was determined automatically by autocorrelation and used to form an optimal length filter for morphological background removal. A rectangular grid of bounding boxes was defined. Each spot was individually addressed and segmented by thresholding to form a mask. The mask was used to isolate each spot from surrounding background. Individual spot intensity was determined by integrating pixel intensities. Finally, integrated intensities were tabulated and saved to a data file for subsequent statistical analysis to determine which genes matter most.

A workflow template to analyze subcellular structures in fluorescence 2D/3D microscopy images based on a Fiji plugin **Squassh** is described in Rizek et al (2014).

The workflow employs detecting, segmenting, and quantifying subcellular structures. For segmentation, it accounts for the microscope optics and for uneven image background. Further analyses include both colocalization and shape analyses. However, it does not work directly for time-lapse data. A brief summary note can be found here.

An example macro introduced in the documentation page of the ImageJ plugin Trainable Weka Segmentation (in Fiji, it's bundled). A segmentation protocol based on machine learning. Full macro is available in the "Download" Link. 

This plugin can be trained to learn from the user input and perform later the same task in unknown (test) data. Weka: it makes use of all the powerful tools and classifiers from the latest version of Weka. Segmentation: it provides a labeled result based on the training of a chosen classifier. Trainable Weka Segmentation Complete macro example is at the end of the page.

This article Baslat et al. presents a method to compute Lymphatic Vessel Density on an image of the whole slide (a workflow documented as text).

Vessels are obtained with a Maximum Entropy Thresholding applied on the excess Red channel (2 times the red values minus blue+green value). Stroma tissue is obtained with a Moment Preserving Thresholding on the blue channel.

CellX is an open-source software package of workflow template for cell segmentation, intensity quantification, and cell tracking on a variety of microscopy images with distinguishable cell boundary.

Installation and step-by-step usage details are described in Mayer et al (2013). 

After users provide a few annotations of cell sizes and cell boundary profiles, it tries to match boundary profile pattern on cells thus provide segmentation and further tracking. It works the best on cells without extreme shapes and with a rather homogeneous boundary pattern. It may not work well on images with cells of sizes only a few pixels. Its output comprises control images for visual validation, text files for post-processing statistics, and MATLAB objects for advanced subsequent analysis.

This method allows to compute a threshold that preserves the moments of an image. In ImageJ/Fiji, you can access it in Image->Ajust->Threshold and choose Moments in the list. In Aphelion, the tool is in Segmentation->Threshold->AphImgMomentThreshold The original paper is 2449

SlideToolkit is a collection of command-line tools to assist with the automated histology analysis of whole-slide images. The publication linked in the "reference" details the actual workflow. 

This includes tools to organize the data, perform tiling and subsequent batch processing of the generated tiles in a cell profiler pipeline. All the tools are designed to run on a single PC or on a HPC system. The scripts in the toolkit are on github under MIT licence.

This workflow classifies, or segments, the pixels of an image given user annotations. It is especially suited if the objects of interests are visually (brightness, color, texture) distinct from their surrounding. Users can iteratively select pixel features and provide pixel annotations through a live visualization of selected feature values and current prediction responses. Upon users' satisfaction, the workflow then predicts the remaining unprocessed image(s) regions or new images (as batch processing). Users can export (as images of various formats): selected features, annotations, predicted classification probability, simple segmentation, etc. This workflow is often served as one of the first step options for other workflows offered by ilastik, such as object classification, automatic tracking.

Requires Matlab Runtime Environment or Matlab. Source code (m-files) are downloaded. Software availability: AVeMap was developed under MATLAB (MathWorks). It is available as an executable, multiplatform program, together with source codes and documentation here, and the source code is also available as Supplementary Software. For practical reasons, this executable version, which does not require MATLAB, runs on a single processor. For users who want to customize the software and/or need the power of parallel computing, an installation of MATLAB with its 'parallel' and 'image processing' toolboxes is needed. Note that, even with the executable version, the velocity fields are stored for further analysis. The add-on AVeMap+ uses these AVeMap-computed velocity fields to generate heat map tables. It is available with the same link.

In this example, cells are grown as a tissue monolayer. Rather than identifying individual cells, this pipeline quantifies the area occupied by the tissue sample.

Download package also contains example images. 

This workflow classifies objects based on object-level features (e.g. intensity based, morphology based, etc) and user annotations. It needs segmentation images besides the raw image data. Segmentation images can be obtained from ilastik pixel classification, or binary segmentation images from other tools. Within the object classification, one can prefilter objects through thresholds (on pixel probability image) or object sizes (on segmentation image). Outputs are predicted classification label images. Selected features can also be exported. Advanced users also have possibilities to add customized (object) features for classification in a simple plugin fashion through python scripts.

This workflow is used to track multiple (appear/disappear, dividing and merging) objects in presumably big 2D+t or 3D+t datasets. It is best suitable for roundish objects or spots. Tracking is done through segmentation, which can be obtained from ilastik pixel classification, or imported from other tools. Users should provide a few object level labels, and the software predicts results on the rest of the image or new images with similar image characteristics. As a result, all objects get assigned random IDs at the first frame of the image sequence and all descendants in the same track (also children objects such as daughter cells) inherit this ID.

The original paper describes a method to analyze mitochondrial morphology in 2D and 3D.

Task

Quantify the length of microtubules (MT) and the MT average density per cell.

Workflow descriptions

Simple two step workflow, allowing visual & manual correction of microtubule between the 2 steps. Batch measurement of microtubule lengths for multiple images is achieved by segmenting the MTs and then their skeletonizations. The number of pixels in the microtubule is proportional to their length, so the length can be estimated.

Script

Workflow is written as an ImageJ macro (Fiji) with following steps:

1. The enhancement of tubular structure by computing eigenvalues of the hessian matrix on a Gaussian filtered version of the image ( sigma 1 pixel), as implemented in the tubeness plugin.

2. The tubules were then thresholded , and structures containing less than 3 pixels were discarded.

3. If needed, a visual check and correction of segmented microtubule is then performed.

4. After correction, segmented MTs were then reduced to a 1-pixel thick line using the skeletonize plugin of Fiji. The length of the skeletonized microtubules was then directly proportional to their length.

5. Data were grouped by condition and converted back to micrometers units under Matlab for the statistical tests.

Pitfalls

Commented but not that general without editing some fields in the macros.

Sample Data

Sample data and workflow (see above URL) can be accessed by - login: biii - password Biii!

Misc

3D version also available here. Use of components Skeletonize and Tubeness Filter

Tracking of focal adhesions includes a number of challenges:

1. Detection of focal adhesion regions in areas of highly variable background
2. Separation of "clumped" adhesions in different objects.
3. Dynamics: Focal adhesions dynamically, grow, shrink, change their shape, they can fuse with neighboring adhesions or one adhesion can be split into multiple children.

Würflinger et al (2011) describe how to detect focal adhesion objects and how to track them over time. Interestingly, tracking results are fed back to segmentation to improve separation of clumped adhesions.

The authors implemented the workflow in Matlab, but do not provide a ready-to-use script.

This macro segments blood vessels in a 3D stack. It is suited for well-contrasted images (low background) and works better if the width of the vessels of interest is reasonably uniform.

Sample image: 1

sample image: 2