When the Commonwealth ended and King Charles came back to power, many men changed their political colour (like the Vicar of Bray) and many inns hastily changed their names to loyalist forms such as "The Royal Lion", "The Royal Oak" (and "The King's Head"?—perhaps not). However, this one, though changing its colour, had always been a Lion: it called itself at various times the Golden Lion, Red Lion, White Lion, and only became the Royal Lion after the Prince of Wales slept in it in 1856. So the locals called it The Confused Lion.

The earlier Lion stood farther back from the street; in front of it were its yard and two houses. These were "scorched and injured" by the great fire of 1844, so the Golden Lion bought and demolished them; and in 1909 it absorbed the adjoining New Inn.

The American-born painter James McNeill Whistler, working in London and Paris, used to come and stay in Lyme, especially since its sea air seemed to be his wife Beatrix's cancer. ("Trixie" was the widow of the architect E. W. Godwin, who built Whistler's house; she married Whistler in 1888, and died of the cancer five years later.) He made many sketches and, in 1895, two notable portraits, "The Master Smith of Lyme Regis" and "The Little Rose of Lyme Regis" (which had also been the name of a Lyme ship three centuries earlier). Whistler's little rose was Rose Rendall, small daughter of a Broad Street grocer. He saw her from a window of the Royal Lion, and went out and told her he intended to paint her. Think what would happen to him if he did that nowadays! What happened then was that she ran away, thinking he meant to apply a coat of paint to her, like a doll. (I know at least one other person who had the same reaction when told, at the age of four, that she was going to be painted.) After it was explained to Rose that he would paint a picture of her, he rewarded her with a doll from Paris, which is now shown in the Lyme museum.