On the shoulders of giants: David Marr

Visual Design: Elif Taştekin with Midjourney ⁠Proofreading: İdil Cervatoğlu

⁠"If I have seen further than others, it is by standing on the shoulders of giants." - Isaac Newton ⁠ ⁠In the history of cognitive science, and perhaps in the history of science as a whole, few have attained the respect and influence that belonged to David Courtenay Marr. In his brief life of just 35 years, if we exclude his undergraduate studies, in just about ten to fifteen years, the contributions that David Marr made to science and what he achieved are neither something most of us could dream of, nor realize even if we dared to dream, nor have the “luck” to accomplish even if we were capable. After his short life, David Marr would be remembered as one of the founders, perhaps the only founder, of computational neuroscience, and in addition, he would become one of the most frequently mentioned names within the interdisciplinary field of cognitive science.

Unfortunately, we know very little about David’s childhood and elementary school years. However, as far as we know, he attended one of England’s most established schools, Rugby School, on a scholarship. Founded in 1567, Rugby School, even before David was born, had been a place where many "great names" spent their adolescence. After completing his first year and successfully passing his exams, he made his subject choices for a two-year education according to his interests, as required by the educational system of Rugby School. He chose mathematics and physics, and since he needed to select a language course as well, he added Russian. After these two years, known as "A Levels," students underwent another year of "S Level" and scholarship education. Additionally, those who wished could receive an extra term of education to write a thesis.

David's interest in analytical disciplines such as mathematics and physics was evident even then. Yet, perhaps unexpectedly, his interest in neuroscience, the discipline on which he would later build his career, was also apparent in those years. David, along with his best friends at high school, Peter Williams and William Y. Arms—who would later become a professor of computer science at Cornell University—had read and discussed Grey Walter’s 1953 book "The Living Brain," which addressed topics like the evolution of the brain, the invention of EEG, and how patterns in the brain could provide information about a person's mind. Young David was excited by the idea of developing a "mathematical theory of the brain," advocating that it was a feasible notion.

One of David’s three main passions in life—his field of study and “flying” being the other two—was music, which began to emerge during his high school years. In 1961, David, who played the clarinet in the British National Youth Orchestra, would maintain his connection to music for the rest of his life. He met his later close friend Tony Pay in this orchestra, and years later, David would recall the shock of meeting Tony Pay: "For the first time in my life, someone was clearly better than me at something I did. This caused me incredible traumatic feelings and terrible insecurity. So much so that I couldn't stand being obviously the second best and left the Youth Orchestra after a year." He would later add that looking back, he recognized an originality in his own playing style, which he could not appreciate at the time.

In 1963, David began his undergraduate studies in mathematics at Trinity College, Cambridge, on a scholarship. After completing his bachelor's degree three years later, he contacted Giles Brindley in the physiology department to express his desire to do his Ph.D. with him on the brain-related work he had been contemplating for a long time. Brindley advised him to spend a year reading extensively in the field to gain knowledge, which David dedicated a year to. Among his readings at that time were the works of Ramon y Cajal and the 1967 book by Eccles, Ito, and Szentágothai, "The Cerebellum as a Neuronal Machine." After this intense period of reading, Marr felt ready for his theory, which resulted in three separate papers published in 1969, 1970, and 1971. These papers, in essence, attempted to answer one of the fundamental questions of neurobiology by focusing on three different neuroanatomical structures: How does the brain work?

These three papers, in their entirety, actually constituted an expanded version of Marr’s doctoral thesis. The first focused on the function and architecture of the cerebellum, the second on the cerebrum, and the third on the hippocampus or "archicortex," as it was termed at the time.

In the first paper on the cerebellum, David Marr proposed a function for the neural circuits described in the book by Eccles, Ito, and Szentágothai. Although the book depicted many pathways and their interactions, it did not present a theory of what they actually did. This subject was one of the major debates in neurophysiology at the time. In particular, two excitatory pathways interacting with Purkinje cells were the focus of the debate: the mossy fibre-granule cell-parallel fibre pathway (let's call it the 'first pathway' for the sake of this writing) and the inferior olive-climbing fibre pathway (let's call this the 'second pathway'). Purkinje cells received excitatory input from both pathways. David Marr suggested that the second pathway carried a directive from the cerebrum about a certain movement pattern, so to speak, an "order," and that the first pathway transmitted information about the "context" in which this "movement pattern" was fired. Thus, the information about a specific "action" and the "context" in which it occurred was encoded together, leading to a strengthening of the connection over time and the "contextual information becoming automated enough to trigger the action." For instance, when learning to ride a bicycle or swim, the process of initially being aware of our movements and gradually performing them until they become automated and form a sort of "muscle memory" is akin to this. In short, David Marr argued that the function of the cerebellum was motor learning, offering a formal theory to explain the existing neural architecture data. An important detail to note about this theory is that it also claimed that Hebbian learning took place in the brain, a concept introduced before Bliss and Lomo demonstrated long-term potentiation occurring in a Hebbian manner in the brain for the first time in 1973.

David Marr's idea was quite new and different for the neurobiology of the time. However, it did not attract much attention in the years it was first published, unlike the attention it would garner later. Furthermore, because of its assumptions about Hebbian learning, some even claimed it was "ungrounded." The theory, later known as the Motor Learning Theory, was further developed and modified over the years by the works of Albus and Ito.

David Marr's 1971 theory of the hippocampus was based primarily on the findings published in 1957 by William Scoville and Brenda Milner. Cases like H.M., which were later corroborated by other cases, showed that damage to the hippocampus led to problems with short-term memories and the formation of new memories, yet did not affect long-term memories. Moreover, in his 1970 paper on the neocortex, which I will discuss shortly, he stated that the brain could not store any random set of associations, therefore the information to be encoded in long-term memory had to be filtered and differentiated according to certain parameters or constraints from the total information before being encoded into long-term memory. David Marr would focus on these as the "functions to be explained" by the hippocampus. To address this problem, or rather to make the problem "addressable," he made various assumptions, some "intuitive" and others based on the neuroanatomy of the time. Some of these assumptions involved, for example, the amount of total information to be stored, the number of synapses in a nerve cell, etc. Marr called the "input units that needed to be recorded" "events." Assuming that information is transferred to long-term memory, to the neocortex, during sleep, and that no more than one "event" is encoded per second, he postulated that approximately 10⁵ events would be encoded per day. Considering these and other similar constraints, David thought that his theory needed to contain three layers to perform the necessary operations. In this respect, the theory would be likened to the perceptron by some. A small note might be helpful here: Marr's proposed theory was not about a short-term memory involved in tasks like keeping a phone number in mind for a brief period but was about the process later called "memory consolidation," which is the transformation of information into a stable and permanent memory over time.

Lastly, in his 1970 neocortex theory, David Marr focused on the question of how the cerebrum "discovers" categories and represents the "external" world. This theory was vital in terms of the question of how the brain operates because some details that were assumed in both the hippocampus and cerebellum theories could only be addressed with the cerebrum theory. Yet, a gap regarding the interaction between the cerebrum and hippocampus was noticeable in the theory; although Marr had mentioned writing an article on this, as far as we know, it was never written. In the cerebrum paper, Marr made two definitions, one for internal input, which was already learned, and another for external input, which was "new." Drawing from cluster analysis methods in the field of numerical taxonomy of the time, Marr postulated that the similarities between these external inputs and internal inputs were first calculated, and then a conditional probability about which class(es) the "new" belonged to was deduced from this calculation, and classification was done accordingly. Additionally, he stated that these conditional probabilities alone would not be sufficient and could only be explanatory "if the external world is objectively, spatially consistent and orderly," thereby touching on a philosophically debated and intriguing subject.

In these three papers, the dominant assumptions, fundamental propositions, and presentation of various predictions derived from the theory in order of importance also attracted attention for their significance in terms of academic writing and theory development. However, the question of how "current" and relevant Marr's articles remain against new evidence and findings after all this time is still a subject of debate. But perhaps more importantly, and still a dominant reason for the praise of these papers today, is not the specific propositions of Marr or the specific details of his theory, but the clarity of his reasoning on what needs to be and how things should be for a physiological circuit to process a function or fulfil a task. These papers can still provide a great deal of ideas and insight for those interested in theoretical neurobiology on "how to approach a problem."

During his Ph.D., David continued to focus on these topics. The work of Giles Brindley on learning and the visual system in cats, particularly his thesis on "modifiable synapses," had influenced Marr and his theory greatly utilized this concept. Moreover, Marr had the opportunity to play music with Brindley, who was also closely associated with music, allowing them to develop their relationship further. Yet, from a theoretical perspective, Marr would later talk about his approach during his Ph.D. with these words: "I believed that truth was essentially neuronal in this field, and the purpose of my research was to uncover the structure of the nervous system." Although traces of Marr's later views, especially the Levels of Analysis, can be seen in his Ph.D., it is possible to say that he would later diverge from his old views in terms of the truth not being neuronal.

Having completed his master's in mathematics and his doctorate in neurophysiology, David reached the end of his "education" life in 1971. Immediately afterwards, he joined the laboratory of Francis Crick, one of the discoverers of the structure of the DNA molecule and later a Nobel laureate who had shifted his interest to neurobiology, the MRC Molecular Biology Laboratory. Francis and David had known each other before. One day, when Francis complained to Alan Hodgkin, who won the Nobel Prize with Andrew Huxley for their models on the action potential, that most of the theoretical neurobiology studies he was reading were baseless, Hodgkin asked, "What do you think about David Marr?" After reaching out and meeting him, Francis, who had not yet decided what to work on after his Ph.D., opened the doors of his laboratory to David. However, what was truly decisive in David's life at the time was the conference held at Boston University School of Medicine on 24–26 May, as the last link in a chain of coincidences. Benjamin Kaminer, a key figure in organizing this conference, knew Sydney Brenner and Seymour Papert from South Africa. Sydney Brenner was a molecular biologist working on C. elegans in Francis Crick's laboratory, whose work would later earn him a Nobel Prize. Seymour Papert, who was at MIT at the time, published the famous study on perceptrons with Marvin Minsky in 1967. It occurred to Benjamin, who had just started working at Woods Hole Massachusetts, that bringing together the work teams of Sydney and Seymour could be scientifically very beneficial. When he shared this idea with Sydney, Sydney mentioned David Marr, who had recently joined their laboratory and was working on theoretical aspects of the brain, and requested that he be invited as well. Eventually, the conference took place between 24–26 May with the participation of big names like David Marr, Sydney Brenner, Francis Crick, Marvin Minsky, Seymour Papert, Stephen Blomfield, David Hubel, Torsten Wiesel, Horace B. Barlow, and more. David spent a considerable amount of time with Marvin and Seymour at the conference, engaging in many discussions. As a result, he was invited to the MIT Artificial Intelligence Lab, followed by a letter sent on November 21, 1972: "This is a formal invitation for you to come for three or six months or more if you want. The salary is $1,250 a month. Requirement: Do something terrific!"

David accepted this invitation and went to the MIT AI Lab in 1973 with a research proposal in hand. His project, named KLUMSY, aimed to address questions such as how a "simple" sensorimotor system like a robotic arm operates, and "how it contains information and moves according to that information." Although quite detailed, his proposal was not accepted. Consequently, David's interest began to shift gradually. The academic environment he joined surprised him in one aspect; there was little knowledge about neurophysiological details or about what the brain did. Moreover, there didn't seem to be much interest or effort in this direction. Most people approached the subjects from the perspective of computer science, and although everyone was very competent in this regard, the interdisciplinary interaction David was looking for was not there. Those who were somewhat interested in these topics were only aware of Piaget, which Seymour Papert had talked about during their brief time working together. David, who wanted to change this and work more interdisciplinarily, would be quite successful in this endeavor in the years ahead, meeting many scientists who would help him with this effort. Setting everything else aside, David, who was greatly influenced by the energetic, discussion, and idea-exchange-friendly environment at MIT, wrote in a letter to his Ph.D. advisor Brindley in 1973 that "the environment is so impressive that I have decided to stay for one or two more years."

During this time, his interest increasingly shifted to the field of vision. This shift was certainly influenced by the exchange of ideas and long conversations he had with individuals like Jack Pettigrew, Carl Hewitt, and Edwin Land. He even had the opportunity to write and publish a couple of articles in collaboration with a few people working in the field of vision. His personal readings and thoughts were also increasingly taking shape. These could be seen in both his later articles and the "AI Memos," the name given to the presentations they generally made among themselves at the AI Lab. David often prepared AI Memos and shared his ideas with those interested at MIT. In 1975, he decided to work permanently at MIT and accepted a professorship offer from the psychology department. Right after, he established a "vision group" within the AI Lab with students who wanted to work in the field of vision and do their theses in this area. He worked with names like Tomaso Poggio, Shimon Ullman, Ellen Hildreth, and Keith Nishihara. Among them, Tomaso Poggio became one of his closest associates, a good colleague and a good friend. Tommy was someone with whom David could share his passion for "flying." David, who had been flying as a hobby since his youth, encouraged Tommy to take a few flying lessons, and they were able to share this passion. In addition, the Levels of Analysis that David would detail in his Vision book were first introduced in an article titled "From Understanding Computation to Understanding Neural Circuitry" published with Tomaso Poggio in 1976. The biggest common ground in their scientific approaches was their interest in "vision," both having a solid mathematical background and more importantly, they both valued neurophysiological and psychophysical evidence. Among the "vision group," the pair began to be referred to as the "dynamic duo."

David Marr learned he had leukemia when he went to the hospital on December 2, 1977, after his fever did not subside for two weeks. The rest of David's life was characterized by a determined effort not to let his scientific passion wane and to battle a quite serious illness. Despite the ups and downs of his disease, the vision group had published about 120 articles during those three years. Moreover, David also embarked on writing "Vision," one of the seminal works in the field, in 1979. The book, subtitled "A Study of Visual Information Processing in Humans," was essentially a compilation of the findings from his work since he began to be interested in vision and creating a comprehensive theory of vision as possible. Perhaps even more importantly than the theory, the section on "Levels of Analysis" was noteworthy.

According to this, to "truly understand" any information processing system, we needed to examine it at three levels, and three different dimensions. The first was the "Computational Level." The main question at this level was what the function of the said system was, what input it took and what output it gave, what kind of function it calculated. The name of this level might seem odd to some, as it was often associated with mathematical modeling, but here mathematics was not a parameter because the answers at the other levels could also be mathematically modeled. The second level was the "Algorithmic" or "Representational Level" (Algorithmic / Representational). This level concerned the question of what algorithms or types of algorithms the system used to perform the "computation" on the incoming input. Finally, David suggested that we should ask how this algorithm or algorithms were physically implemented. The "Implementation Level" was the stage where this question was explored.

At the same time, Marr's "Three-Level Hypothesis," also known as Levels of Analysis, provided a methodological framework for all of cognitive science, not just theoretical neurobiology or studies in vision. Although other scientists and philosophers, such as Chomsky with his competence-performance distinction, had made similar proposals, Marr's framework would become the most well-known over time.

As the name suggests, Marr developed his comprehensive theory of vision in this book. And according to David Marr's vision theory, vision occurred as a result of three sequential stages. The first was the "Primary Sketch." In this step, the "edges" were initially identified. Then in the "2.5D Sketch" stage, surfaces and gradients were determined. Finally, in the "3D Sketch" stage, a three-dimensional representation of the object independent of the observer's viewpoint was created.

In 1979, David Marr and Tomaso Poggio went to visit Francis Crick, who had moved to the Salk Institute, in La Jolla, California. The theoretical discussions with Francis during their month-long stay were actually the inspiration for the dialogue in the epilogue of this book. This section would become uniquely important as many professors recommended it for those interested in cognitive science.

Unfortunately, David would not live to see his book in its final form, and it would only be published two years after his death. On November 16, 1980, a remarkably sunny and warm Sunday, around 11 in the morning, a nurse entered David Marr's room and called out to him. His wife, Lucia, was also in the room. When David did not respond, Lucia thought he was playing a small joke, as he liked to joke with the nurses, and initially did not take it seriously. But when David did not answer the nurse's calls not once, not twice, but numerous times, they realized the gravity of the situation. David had suddenly fallen into a coma that day and passed away the next.

Although David did not live to see those days, in one of the first reviews published after the book was released, Cristopher Longuet-Higgins would say, "David Marr was already a legend among neuroscientists when he died at the age of 35." Forty years later, I see no problem in revising this statement to describe him not only among neuroscientists but as "one of the legendary figures in the history of science." May he rest in peace... ⁠

⁠His Publications: ⁠ ⁠(1969) A theory of cerebellar cortex. The Journal of Physiology. 202: 437-70.

(1970) A theory for cerebral neocortex. Proceedings of the Royal Society of London. Series B, Biological Sciences. 176: 161-234.

(1970) How the cerebellum may be used. Nature. 227: 1224-8. (With Stephen Blomfield)

(1971) Simple memory: a theory for archicortex. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences. 262: 23-81.

(1974) The computation of lightness by the primate retina. Vision Research. 14: 1377-88.

(1975) Approaches to biological information processing. Science, 190:875–876.

(1976) Early processing of visual information. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences. 275: 483-519.

(1976) Cooperative computation of stereo disparity. Science (New York, N.Y.). 194: 283-7. (With Tomaso Poggio)

(1976) Analyzing natural images: a computational theory of texture vision. Cold Spring Harbor Symposia On Quantitative Biology. 40: 647-62.

(1977) Analysis of occluding contour Proceedings of the Royal Society of London - Biological Sciences. 197: 441-475.

(1977) From understanding computation to understanding neural circuitry Neurosciences Research Program Bulletin. 15: 470-488. (With Tomaso Poggio)

(1977) Artificial intelligence-A personal view Artificial Intelligence. 9: 37-48.

(1978) Visual information processing: artifical intelligence and the sensorium of sight. Technological Review. 81:28-49. (With Keith Nisihara)

(1978) Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Society of London. Series B, Biological Sciences. 200: 269-94. (With Keith Nisihara)

(1978) Analysis of a cooperative stereo algorithm. Biological Cybernetics. 28: 223-39. (With Guenther Palm and Tomaso Poggio)

(1979) A computational theory of human stereo vision. Proceedings of the Royal Society of London. Series B, Biological Sciences. 204: 301-28. (With Tomaso Poggio)

(1979) Bandpass channels, zero-crossings, and early visual information processing. Journal of the Optical Society of America. 69: 914-6. (With Shimon Ullman and Tomaso Poggio)

(1980) Theory of edge detection. Proceedings of the Royal Society of London. Series B, Biological Sciences. 207: 187-217. (With Ellen Hildreth)

(1980) Visual information processing: the structure and creation of visual representations. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences. 290: 199-218.

(1980) Smallest channel in early human vision. Journal of the Optical Society of America. 70: 868-70. (With Tomaso Poggio and Ellen Hildreth)

(1981) Directional selectivity and its use in early visual processing. Proceedings of the Royal Society of London. Series B, Biological Sciences. 211: 151-80. (With Shimon Ullman)

(1982) Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. San Francisco: W. H. Freeman and Company.

(1982) Representation and recognition of the movements of shapes. Proceedings of the Royal Society of London. Series B, Biological Sciences. 214: 501-24. (With Lucia Vaina) ⁠

⁠Further Reading

Vaina, L., & Marr, D. (2017). Computational theories and their implementation in the brain: The legacy of David Marr.

Vaina, L. (1991). From the Retina to the Neocortex: Selected Papers of David Marr. Boston: Birkhauser Boston.

Edelman, S. (2012). Vision, Reanimated and Reimagined. Perception, 41(9), 1116–1127. https://doi.org/10.1068/p7274

Poggio, T. (2012). The Levels of Understanding Framework, Revised. Perception, 41(9), 1017–1023. https://doi.org/10.1068/p7299

Quinlan, P. (2012). Marr’s Vision 30 Years on: From a Personal Point of View. Perception, 41(9), 1009–1012. https://doi.org/10.1068/p4109ed

Willshaw, D. J., & Buckingham, J. T. (1990). An assessment of Marr's theory of the hippocampus as a temporary memory store. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 329(1253), 205–215. https://doi.org/10.1098/rstb.1990.0165

Willshaw, D., Hallam, J., Gingell, S., & Lau, S. L. (1997). Marr's theory of the neocortex as a self-organizing neural network. Neural computation, 9(4), 911–936. https://doi.org/10.1162/neco.1997.9.4.911

Tyrrell, T., & Willshaw, D. (1992). Cerebellar cortex: its simulation and the relevance of Marr's theory. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 336(1277), 239–257. https://doi.org/10.1098/rstb.1992.0059

Willshaw DJ, Dayan P, Morris RGM. 2015 Memory, modelling and Marr: a commentary on Marr (1971) ‘Simple memory: a theory of archicortex’. Phil. Trans. R. Soc. B 370: 20140383. http://dx.doi.org/10.1098/rstb.2014.0383