This is because on the 4th crusade they attacked Constantinople. The Crusader and Venetian leadership decided on the outright conquest of Constantinople in order to settle debts, and drew up a formal agreement to divide the Byzantine Empire between them. So the 4th one was not made to the Holy Land.
