Connectionism, Artificial Neural Networks, and Systematicity
Connectionism is often evaluated through the critique advanced by Jerry Fodor and Zenon Pylyshyn in Connectionism and Cognitive Architecture: A Critical Analysis (1988), alongside the response developed by Paul Smolensky in The Constituent Structure of Connectionist Mental States: A Reply to Fodor and Pylyshyn (1988). The debate centres on whether connectionist models can adequately account for the structured nature of human cognition.
From a connectionist perspective, mental representations are understood as distributed patterns of activation across networks, where meaning emerges from the structure of these patterns rather than from discrete symbolic tokens (Buckner & Garson, 2025). A key challenge to this approach is the systematicity argument developed by Fodor and Pylyshyn (1988), which claims that human cognition is inherently structured and compositional. According to this view, the ability to entertain a given thought implies the capacity to entertain systematically related thoughts, suggesting an underlying symbolic architecture that classical models are better suited to explain.
In response, Smolensky (1988) argues that connectionist systems can account for systematic and compositional properties without requiring explicit symbolic rules. He proposes that such structure can emerge from subsymbolic, distributed processes within neural networks. Together, these positions frame an ongoing debate over whether cognitive architecture is fundamentally symbolic or whether systematic cognition can be fully explained through connectionist mechanisms.
What is Connectionism, ANN, and Systematicity?
Within cognitive science, connectionism is a movement that leverages artificial neural networks (ANNs) to examine aspects of human cognition and intellectual abilities (Buckner & Garson, 1997). ANNs are computer systems inspired by early models of sensory processing in the brain (Krogh, 2008). Neural networks consist of a large number of interconnected units (nodes) arranged in structured layers that simulate simple neuron activity, akin to how the human brain processes data. These units are segregated into three classes:
- Input nodes
- Hidden nodes
- Output nodes
ANN models compute outputs by taking data from other nodes or external sources via input nodes, weighting each input, and summing them to determine an output (Krogh, 2008). Outputs and patterns of activation are determined by the weights (connection strengths) between nodes. Representation in connectionism is distributed across patterns of activation across nodes, mathematically expressed as vectors of numbers that encode a system’s state (Buckner & Garson, 1997). In classic ANN models, if the total input exceeds a threshold, the output of the unit is one; otherwise, it is zero (Krogh, 2008).
By applying algorithms that mimic processes of real neurons, ANNs “learn” to solve problems, often achieving results associated with human intellect (Krogh, 2008). Experiments using ANN models have demonstrated the ability to learn skills such as:
- Facial recognition
- Reading
- Grammatical structure detection
ANNs learn by adjusting their node weights using backpropagation to increase output accuracy. This involves:
- Sending inputs forward to generate a prediction
- Comparing the prediction with the correct answer
- Calculating error
- Modifying node weights to reduce error through gradient descent
While ANN provides a computational framework for understanding cognitive processes, critics such as Fodor and Pylyshyn (1988) question whether these systems can capture the systematic structure of human cognition. Systematicity refers to the idea that if a mind can entertain structured thoughts composed according to certain rules, it can also entertain other closely related thoughts (Fodor & Pylyshyn, 1988).
For example, if a person can understand:
“John loves Mary”
they can also understand:
“Mary loves John”
These related thoughts demonstrate that mental representations are composed of components (“John,” “Mary,” and “love”) that can be recombined in structured ways. Fodor and Pylyshyn argue that this reliable pairing of cognitive capacities reflects an underlying combinatorial structure rather than an accidental feature of cognition.
Evaluating Fodor and Pylyshyn’s Systematicity Argument
Fodor and Pylyshyn (1988) argue that ANNs lack the symbolic and syntactic architecture required to demonstrate systematicity of thought. They maintain that classical symbolic cognitive architecture is necessary to explain the core properties of human thought, and that connectionist models fail to replicate these properties independently. This supports the Language of Thought (LOT) hypothesis, which claims that cognitive processes operate on mental representations whose compositional relations generate meaning.
According to Fodor and Pylyshyn (1988), classical models involve:
- Combinatorial mental representations
- Structure-sensitive mental processes
Thrane (1988) defines symbolic representation as a relational process in which one entity (“A”) represents another (“B”) within a shared context. This generates mental representations expressed through symbols whose meanings derive from interpretation of external objects.
Combinatorial mental representations are structured symbols whose meanings are determined by both semantic content and syntactic arrangement of constituent parts, much like sentences in a language (Fodor & Pylyshyn, 1988). Structure-sensitive mental processes are operations that depend on syntactic structure rather than overall similarity or patterns of activation.
Together, these elements form the basis of the LOT hypothesis and explain how a finite set of symbols can generate an infinite number of thoughts.
Fodor and Pylyshyn argue that connectionist models struggle because they lack this combinatorial structure. In connectionist systems:
- Representations are distributed across nodes
- Meaning arises from activation patterns rather than discrete symbolic parts
As a result, ANN models struggle to explain why certain cognitive capacities necessarily co-occur. For example, if an ANN learns to represent:
“John loves Mary”
it does not automatically follow that it can also represent:
“Mary loves John”
unless it has been explicitly trained on that example.
Each configuration must be learned as a distinct pattern rather than derived from shared structure. Fodor and Pylyshyn therefore argue that connectionist models treat representations as unstructured lists that cannot recombine shared components. Without language-like syntax, connectionist models struggle to distinguish representational roles and contents, resulting in an architecture that lacks the formal compositionality required for systematicity.
Smolensky’s Counter to Fodor and Pylyshyn
Smolensky (1988) rejects the claim that connectionist models cannot account for systematicity. Instead, he argues that systematicity can emerge from subsymbolic connectionist representations rather than symbolic ones.
Subsymbolic representation refers to distributed encoding of knowledge across patterns of activation (Kelley, 2003). Information is not stored in any single location but distributed across network weights, allowing systems to function as autonomous learning systems operating in parallel.
Rather than encoding explicit discrete items, subsymbolic models capture information implicitly through overall patterns of weighted connections that determine learned responses (Kelley, 2003). Smolensky argues that connectionist systems can encode symbols and roles within distributed structures that represent complex scenarios.
Tensor-Product Variable Binding (TPVB)
Smolensky (1990) proposed tensor-product variable binding (TPVB) as a mechanism through which connectionist models can encode structured symbolic information within subsymbolic representations.
In TPVB frameworks:
- Roles (subject, verb, object) are represented as vectors
- Fillers (“John,” “Mary,” “love”) are also represented as vectors
- These vectors are mathematically combined into structured bindings
For example, in the sentence:
“John loves Mary”
- “John” fills the subject role
- “loves” is the verb
- “Mary” fills the object role
TPVB combines these role–filler relationships mathematically so that the system preserves relational structure. Through this strategy, connectionist systems can achieve systematicity because reassigning fillers to different roles, such as changing:
“John loves Mary”
to:
“Mary loves John”
requires only minor changes in vector bindings rather than entirely new symbolic rules (Cummins et al., 2001).
Tensor-product representations represent a major theoretical advance because they demonstrate that connectionist systems can exhibit systematic behaviour without relying on language-like symbolic syntax. Smolensky further argues that compositional structure in cognition is inherently graded rather than perfectly discrete. According to this view:
- Mental representations are approximations of underlying activation patterns
- Symbols are emergent rather than fundamental
- Systematicity arises through learned similarities in distributed representations
rather than through an internal symbolic language.
Conclusion
Connectionism provides a framework for understanding cognition as emerging from distributed patterns of activation across neural networks. ANNs learn through backpropagation algorithms that enable them to analyse data and solve problems without explicit symbolic rules. This has led to successful applications such as:
- Facial recognition
- Reading
- Grammatical structure detection
However, Fodor and Pylyshyn (1988) argue that these systems cannot adequately explain the systematic and compositional structure of human cognition. Drawing on the Language of Thought hypothesis, they maintain that cognition requires classical symbolic architectures involving formal rule-based manipulation of representations. In their view, connectionist systems fail to explain why cognitive capacities occur in structured and interdependent clusters.
Smolensky (1988), however, challenges this conclusion by arguing that systematicity can emerge from subsymbolic distributed representations. Through tensor-product variable binding, connectionist systems can encode roles and fillers within activation vectors while preserving relational structure without relying on symbolic syntax.
Ultimately, the debate highlights a central question in cognitive science:
Is cognition fundamentally symbolic and rule-governed, or is it subsymbolic and emergent from neural network dynamics?