Day and Night, 1938, by MC Escher. Images courtesy of Collection Gemeentemuseum Den Haag/the MC Escher Company

The artist who created some of the most memorable images of the 20th century was never fully embraced by the art world. There is just one work by Maurits Cornelis Escher in all of Britain’s galleries and museums, and it was not until his 70th birthday that the first full retrospective exhibition took place in his native Netherlands. Escher was admired mainly by mathematicians and scientists, and found global fame only when he came to be considered a pioneer of psychedelic art by the hippy counterculture of the 1960s. His prints adorn albums by Mott the Hoople and the Scaffold, and he was courted unsuccessfully by Mick Jagger for an album cover and by Stanley Kubrick for help transforming what became 2001: A Space Odyssey into a “fourth-dimensional film”.

But Escher did not belong to any movement. In a 1969 letter to a friend, he observed testily that “the hippies of San Francisco continue to print my work illegally”. (Many of his letters are reproduced in the standard reference book, Escher: The Complete Graphic Work, edited by JL Locher, which includes a full biography and analytical essays by Escher and others.) He had been sent a catalogue for a California “Free University” that contains “three reproductions of my prints alternating with photographs of seductive naked girls”. This would have seemed distasteful to the rather formal Escher, who bridled when Jagger addressed him by his first name in a fan letter. According to Patrick Elliott’s catalogue essay, “Escher and Britain”, for the new exhibition at the Scottish National Gallery of Modern Art, The Amazing World of MC Escher, the artist replied to the musician’s assistant: “Please tell Mr Jagger I am not Maurits to him.”

To his family and childhood friends Maurits was affectionately known as Mauk. He was born to George and Sara Escher in 1898 in Leeuwarden. The youngest of his civil-engineer father’s five sons (two from a previous marriage), Mauk was a sickly child who was interested in carpentry and took music lessons, but failed his final school exams, except for mathematics. His father noted fondly in his diary that the young man consoled himself “by drawing and making a linocut of a sunflower”.

Drawing Hands, 1948, by MC Escher

Escher then studied for a few years at the School for Architecture and Decorative Arts in Haarlem, but he abandoned architecture to try to carve out a career as a graphic artist. It quickly went well. By the end of the 1920s, during which he had travelled extensively in Italy and Spain, and met and married his wife, Jetta, Escher was exhibiting his work regularly in Holland, and, in 1934, he won his first American exhibition prize. But it was only two years later that Escher really became Escher. That year he went to the Alhambra Palace in Granada, Spain, and carefully copied some of its geometric tiling. His work gradually became less observational and more formally inventive. As Escher later explained, it also helped that the architecture and landscape of his successive homes in Switzerland, Belgium and Netherlands were so boring: he “felt compelled to withdraw from the more or less direct and true-to-life illustrating of my surroundings”, embracing what he called his “inner visions”.

Those visions fed what would become Escher’s most celebrated works. In 1948, he made Drawing Hands, the image of two hands, each drawing the other with a pencil. It is a neat depiction of one of Escher’s enduring fascinations: the contrast between the two-dimensional flatness of a sheet of paper and the illusion of three-dimensional volume that can be created with certain marks. In Drawing Hands, space and the flat plane coexist, each born from and returning to the other, the black magic of the artistic illusion made creepily manifest. The following, from a later Escher essay, could easily serve as a gloss on this image:

The artist still has the feeling that moving his pencil over the paper is a kind of magic art. It is not he who determines his shapes; it seems rather that the stupid flat shape at which he painstakingly toils has its own will (or lack of will), that it is this shape which decides or hinders the movement of the drawing hand, as though the artist were a spiritualist medium.

Escher’s lifelong subject, in a way, was the dramatised artificiality of the created image. (The art historian EH Gombrich wrote that Escher’s work “presents so many interesting comments on the puzzles of representation”.) Of his 1945 picture Balcony, with its weird bulging central distortion, Escher commented: “Surely it is a bit absurd to draw a few lines and then claim ‘This is a house.’” The theme of Balcony, he said, was “this odd situation”

The situations only became odder. Escher methodically pushed representative techniques to their limits. Some earlier images suggest a particular interest in perspective, for instance the “bird’s-eye view” of his Tower of Babel (1928) – which in retrospect seems like an unparadoxical rehearsal for his later adventures in impossible architecture – or a forest of columns in a colonnade for Nocturnal Rome (1934). A later image, Depth (1955), is an entirely fictional investigation into the formal possibilities of perspective: an array of what look like monstrous robotic fish-aeroplanes, receding implicitly into infinite space. (In a letter, Escher explained the carefully thought-out features that contribute to the depth effect, including “Rhythmic positioning of each fish at the intersections of a cubic threefold-rotation-point system”. Well, naturally.)

Hand with a Reflecting Sphere, 1935, by MC Escher

In the late 1930s, Escher also became obsessed by the “regular division of the plane”, in which shapes (often fish, lizards or birds) are tiled across a flat plane in such a way that the spaces between them make other, recognisable shapes. (This technique was directly inspired by the Alhambra.) Day and Night (1938) features black and white bird forms arranged in this way over a chequerboard countryside. In many of these images the distinction between foreground and background is obliterated: the viewer can choose to see one or other set of shapes as foreground at will

But if Escher’s work had been nothing more than a pedantic meta-artistic commentary on the plasticity of techniques, it would by now have been forgotten. He gave a revealing lecture in 1953 that distinguished between “feeling people” – artists who concentrate on the human form – and “thinking people”, artists like himself who are “reality enthusiasts”, interested in “the language of matter, space and the universe”. Escher’s greatest pictures are not simply geometric exercises; they marry formal astonishment with a vivid and idiosyncratic vision.

Take House of Stairs (1951), with its nightmarish interior (inspired by Escher’s own school’s stairways) and its pseudo-human-faced articulated centipedes crawling through its architectural phantasmagoria. (Escher invented those creatures, he explained wryly, “as a result of dissatisfaction concerning nature’s lack of any wheel-shaped living creatures endowed with the power of propulsion by means of rolling themselves up”.) Or Belvedere (1958), with its impossible ladder and a collection of those jesters, knaves and contemplators who would come to populate Escher’s most extraordinary invented places. (The long-gowned woman in Belvedere is copied directly from The Garden of Earthly Delights by Hieronymus Bosch.)

Most dazzling, perhaps, is the celebrated Ascending and Descending (1960), with its two ranks of human figures trudging forever upwards and eternally downwards respectively on an impossible four-sided eternal staircase. It is the most recognisable of Escher’s “impossible objects” images, which were inspired by the British mathematician Roger Penrose and his father, the geneticist Lionel Penrose. Fascinated by House of Stairs, the Penroses published a paper in 1956 in the British Journal of Psychology entitled “Impossible Objects: A Special Type of Visual Illusion”. Receiving an offprint a few years later, Escher wrote to Lionel expressing his admiration for the “continuous flights of steps” in the paper, and enclosing a print of Ascending and Descending. (The paper also included the “tri-bar” or Penrose triangle, which is constructed impossibly from three 90-degree angles: in 1961 Escher built his never-ending Waterfall using three of them.)

The mathematical trickery in Ascending and Descending’s staircase is not the subject of the image. Escher was never a surrealist. But in this picture, it becomes clear that he was a kind of existentialist. He had long admired Dostoyevsky and Camus, and in a letter to a friend while he was working on Ascending and Descending he explained: “That staircase is a rather sad, pessimistic subject, as well as being very profound and absurd. With similar questions on his lips, our own Albert Camus has just smashed into a tree in his friend’s car and killed himself. An absurd death, which had rather an effect on me. Yes, yes, we climb up and up, we imagine we are ascending; every step is about 10 inches high, terribly tiring – and where does it all get us? Nowhere.”

Ascending and Descending, 1960, by MC Escher

This dreamscape of futility is perfected by the two figures who are not on the eternal staircase. One gazes up at his condemned fellows from a side terrace; one sits glumly on the lower stairs. “Two recalcitrant individuals refuse, for the time being, to take any part in this exercise,” Escher commented. “They have no use for it at all but no doubt sooner or later they will be brought to see the error of their nonconformity.” Escher’s art at its best, then, is not just surprising but also surprisingly readable, putting him in the company of the great allegorical printmakers such as Albrecht Dürer.

Since Escher’s death in 1972, his most famous images have become ubiquitous. New fuel for his popular cult was provided by Douglas Hofstadter’s interdisciplinary fantasia of a book, Gödel, Escher, Bach (1979), which seduced generations of curious students in the following decades. (Escher adored Bach.) Fittingly, given the artist’s mathematical playfulness, some of the richest tributes to his work in modern times have come in the world of video games. In the beautiful Echochrome (2008), players set out to free an eternally walking human from a succession of Escherian landscapes by rotating the point of view until the “trick” of perspective locks into place.

In a 1963 lecture on “the impossible”, Escher declared: “If you want to express something impossible, you must keep to certain rules. The element of mystery to which you want to draw attention should be surrounded and veiled by a quite obvious, readily recognisable commonness.” This is arguably as true of fiction or music as it is of Escher’s brand of geometric sorcery. And it also, in a way, sums up the genius of Escher himself, an orderly man who made inexhaustibly extraordinary things.