Hemithioindigo (HTI) photoswitches have a tremendous potential for biological and supramolecular applications due to their absorptions in the visible-light region in conjunction with ultrafast photoisomerization and high thermal bistability. Rational tailoring of the photophysical properties for a specific application is the key to exploit the full potential of HTIs as photoswitching tools. Herein we use time-resolved absorption spectroscopy and Hammett analysis to discover an unexpected principal limit to the photoisomerization rate for donor-substituted HTIs. By using stationary absorption and fluorescence measurements in combination with theoretical investigations, we offer a detailed mechanistic explanation for the observed rate limit. An alternative way of approaching and possibly even exceeding the maximum rate by multiple donor substitution is demonstrated, which give access to the fastest HTI photoswitch reported to date.