Choosing a tokenizer means choosing a generator

A tokenizer compresses an image into a representation that a generator can model. It is tempting to choose that tokenizer by reconstruction quality, then treat it as a fixed part of the pipeline. My NeurIPS paper tests this assumption by comparing tokenizers, generators and sampler settings in a controlled ChestMNIST study.

The tokenizer rankings change with the generator. Sampler selection also changes the apparent ranking of generators. A tokenizer that reconstructs images well is therefore not necessarily the best choice for generation in this setting.

For a similar experiment, I would compare complete tokenizer-generator-sampler combinations within the intended inference budget, selecting sampler settings on held-out validation data. That recommendation is an interpretation of this study, rather than a universal rule about image generation.

The experiments concern low-resolution, unconditional images and non-clinical metrics. They do not establish clinical usefulness. Most configurations have limited evidence about variation between training runs.

The manuscript gives the comparisons and limitations. The experiment repository contains result files and figure scripts;medtokenizers andmedlatents contain the model implementations.