Future Work

From Utah Center for Neuroimage Analysis

Contents 1 Bugs 1.1 ir-stom cropping 1.2 ir-stom speed 1.3 ir-stom parameters 1.4 Multichannel Registration 1.5 General fixes 1.6 Output Formats 1.7 Stos registration quality 1.8 ir-tweak 64-bit 1.9 Speed 1.10 General fixes 2 Features 2.1 Delaunay triangle mesh transform 2.2 stos transform import 2.3 RGB ir-assemble and ir-stom 2.4 rotation handle 3 Other things to consider 3.1 Internationalization 3.2 Installers 3.3 Documentation 3.4 Build instructions 3.5 ITK patch for GetInverse()

Bugs

This is a placeholder page for bugs until we get a more structured system.

ir-stom cropping

We want to produce images of each section registered into the volume. Currently ir-stom crops a pixel from the volume if it falls outside the volume even if it could be registered to the starting section. In addition the cropping behavior of ir-stom forces us to exclude marginal sections that can be registered but cannot serve as reference sections due to dirt, folds, etc... The viking viewer includes pixels if they can be mapped from the section they appear on following the registration chain to the starting section, even if other sections further into the volume cannot map that pixel. This is a more desirable behavior for us. I am not sure how easy it will be to get this behavior from ir-stom, but if transforms have to be reimplemented the Viking viewer already has a working reference implementation.

ir-stom speed

(Currently being looked into by Brad)

ir-stom is currently single threaded. Additionally it maps pixels down each step in the registration chain instead of adding the transforms together to produce a single transformation step. This makes ir-stom slow. If we want to produce high resolution images using ir-stom for optimized viewers and to track processes we need this to be faster.

ir-stom parameters

ir-stom currently requires us to build the entire volume when it is invoked. It would be better to have a parameter that enables us to only create the new sections we've added to the volume.

Multichannel Registration

We need to have the ability to perform multi-channel registration (i.e. pulling along more than one channel of data to simultaneously register them). We can currently do this by manually splitting out the channels in photoshop and loading each channel, registering them, and reassembling the output. This is tedious. We want to load RGB images and choose which channels are displayed and then carry forward all channels in registration. A very robust solution allows any number of channels, but I think that falls in the future work category and I'm not sure if any image formats support N number of channels.

General fixes

1. ir-assemble needs to preserve color channels instead of only writing greyscale. ( I haven't looked into this one much. I mostly want to be able to apply .mosaic transformations to colored tiles and retain color in the output. Many labs produce color images instead of greyscale and I've hit this issue trying to help them use our tools.)
2. ir-fft should read a directory or a mosaic file instead of typing each file on the command line. This is in case others ask for our code and we don't want to build a custom pipeline for their scope, or if their scope doesn't have stage-position data. In the case of Windows there is a command-line buffer limit which is quickly exceeded. We should also have ir-fft read a .mosaic file.
3. ir-fft does not align images properly in large mosaics. Integrating Joel's tension code from ir-refine-translate into ir-fft would probably fix this. Again, this is for other users who may ask to use the code after the paper goes out.

Output Formats

• Resolved by Brad

The current system of building an image pyramid for each tile of each section is ideal for assembling the volume quickly. It is not ideal for viewing the volume. At low magnification we have to load 1000+ images into our viewer. At high magnification we have to load up to four 4080x4080 tiles with 15% overlap. Neither is optimal for network performance.

I would like to resolve this by making ir-stom and ir-assemble produce tiles of fixed size for multiple resolutions along with an xml snippet describing the layout of the tiles. By always loading tiles that are 512x512 for every resolution we keep the network load constant regardless of resolution and do not pay the bandwidth costs of loading overlapping tiles. I (James) have a detailed spec written up for how these tools should work.

Stos registration quality

• Resolved by Paul*

The neuroma data is not registering properly. This seems to be because stos-brute is not getting the initial translate and rotation parameters correct. We suspect the underlying issue is registration is thrown off by black background pixels which are generated by ir-assemble. Tolga suggested replacing background with white noise as a workaround, but that did not fix the problem. This is probably because the images are downsampled which reduces the noise to a uniform grey background. Joel says there is a masking system built in to stos-brute already, and that it is utilized when a .mosaic file is provided as input. The downside to this is that we no longer use absolute paths to tiles in the .mosaic files and there is no option to specify the directory which tiles live in as parameters to stos-brute. The other downside is that this requires running assemble on the .mosaic each time we invoke a .stos tool which is very slow. The current thinking is that we add an option to ir-assemble to generate a mask (Hopefully at the same time we assemble the mosaic so the slow assemble code only has to be invoked once.) We then pass this bit-mask into the stos tools, which will need to be modified to support this so the tools know which pixels to ignore during registration.

ir-tweak 64-bit

• Resolved by Paul*

We need an ir-tweak that does not crash with large datasets. Ideally Paul will find a solution, possibly pyramid based, that scales regardless of the size of the images we are registering. An image I (James) need to register for my dissertation proposal which ir-tweak cannot load is 16,500 x 18,000 pixels. This is not an uncommon size. ir-tweak seems to fail even if I downsample by 8 or more, but can load the image from the .mosaic file. However we'd like a solution that scales. I am willing to build image pyramids for these data as we do with EM data if that will allow us to register. We still need to have full-res registered assembled images for output though.

Speed

• Resolved by Paul*

Stos-grid takes a long time on a 1000 tile dataset even if it is downsampled by a factor of 32. (To a size of about 3700x4100 pixels, 46% of which are empty space). Judging from the results I (James) got on the mouse trial dataset I think we're going to want another level of refinement using a downsample of 16. I'd like to get Paul's estimate on different stages of optimizing the code. Optimizing is an open ended goal, so if he could profile it and break down what he could do and how long it would take that would be a great step so we can prioritize those items. I'm in support of parrellelizing it with Cuda if possible. Debugging these datasets takes a long time currently, and the only small dataset we have is successful. Carl may be able to capture a few tiles of a small irregular shaped volume if that is essential for debugging.

General fixes

1. *Resolved by removing preview images* Prepend the filename for the final preview image outputs from ir-stos-grid with the name of the mosaic file so i can quickly look through all of the results of running ir-stos-brute.
2. *Resolved by Brad* Make 64-bit builds work for each major platform: Windows, Mac, Linux for the ir-tools. Ideally the build script would be able to build binaries for each platform without tweaking. It may do this already, but currently only Mac binaries are available to the Marc lab.
3. *Resolved by Paul/Brad* Update ITK patch to work with latest version of ITK

Features

Dear Santa,

Delaunay triangle mesh transform

We'd like to change the underlying registration transform from polynomial warps to local linear warps. Jamie is writing a spec for this so Paul should consult with him before starting. In short, I want to start the registration by defining four points at the corners of the images(texture). This defines two triangles which we use a linear mapping inside to warp the mapping image to control image. The user then defines additional points which subdivide the triangles to perform local refinements. This lets us use a few points if the image is mostly registered, allows us to jump folds in the tissue, and we can use many points in a local region if it is very distorted.

stos transform import

Implement .stos transform import in ir-tweak for manual correction of ir-stos-... registration results.

RGB ir-assemble and ir-stom

One of the groups the Marc Lab introduced to the tools has a dataset in color. We can generate transforms with the grayscale images, but we want to be able to write color files after we have the .mosaic and .stos files.

rotation handle

Add global rotation widget to ir-mosaic, because current control point method is practically unusable.

Other things to consider

Internationalization

1. Fix filename handling problems as they become apparent (international filenames). UTF-8 filenames work out of the box on OSX and Linux, not on Windows. Helper functions for UTF-8 filenames have been added to the_utils.hxx. Internally, std::string, the_text_t and char * text variables should be assumed to hold UTF-8 content.
2. the_ascii_font_t class is incapable of displaying UTF-8 text, perhaps it should be replaced with a more established OpenGL text rendering method (3rd party lib)

Installers

1. ir-tweak and ir-mosaic need proper installers for Windows and Linux.
2. ir-tools are command-line utilities and it is unclear how they should be installed on OSX, Windows or Linux.
3. It is possible that CMake/CPack may be used for generating the installers, this needs to be investigated.

Documentation

1. There have been additions and changes to ir-tools, enhancements to ir-tweak and ir-mosaic. All of this needs to be documented.
2. Techniques, best practices, etc... should be documented as well. Image processing pipeline falls into this category.

Build instructions

1. SVN checkout instructions and explanation of each directory in the ImageReconstruction SVN is necessary.
2. Build instructions need to be updated, they are 2 years out of date by now.

ITK patch for GetInverse()

Patching itk::Transform::GetInverse() is still required.

1. It may be possible to avoid this by implementing a Facade pattern.
2. Ideally, Facade pattern could be avoided by convincing ITK developers to accept our changes. This would require a technical report paper for Insight Journal describing the changes, and adding unit tests for the new GetInverse functionality.