I traveled to Austria this summer, for vacations. It was impossible to avoid visiting the Zentralfriedhof of Vienna, where one can contemplate the grave of a great viennese scientist: Ludwig Boltzmann. His ideas in mathematical physics were amazingly original, beautiful, and deep. He understood the power of melting probability and partial differential equations before the rigorous developments of these fields in mathematics and physics. His elementary derivation of the entropy formula from the combinatorics of microstates is also striking, years before the formulation of the quantum hypothesis by Max Planck and the concept of information by Claude Shannon. Unfortunately, his ideas were not well received at that time, and he eventually committed suicide in Italy after a severe depression. Nevertheless, his work had an enormous impact on science. In 2010, Cédric Villani received a Fields Medal partly for his study of the Boltzmann equation of kinetic theory of gases. You may take a look at his paper on the Cercignani conjecture and the more recent paper by Desvillettes, Mouhot, and Villani.
In the nineteenth century, probability theory was not yet developed, and stochastic models were called statistical models. This is why we say statistical mechanics and statistical physics. You may take a look at the pretty interesting small old book by Paul and Tatiana Ehrenfest.
