In a recent Domus piece on In-Screen Sports Graphics, Max Gadney reports on a talk given by Ryan Ismert at Strata. Ismert works at Sportsvision, which makes the graphics that appear in most US TV sports telecasts.
In the piece, Gadney sets outs Sportsvision’s work as a model for other infoviz practices, especially those aimed at business decision makers and designers of public space. He offers some compelling evidence of Sportsvision’s success at integrating rich sensor-derrived data into concise and comprehensible on-screen graphics.
Then Gadney’s takes things a little farther. He asks “how do we go about getting the freeflowing, more subjective data that might better communicate life in buildings, football and business?” In sport he’s thinking about problems like getting the system to capture the subtleties of the fluid “whirling patterns of play” demonstrated in a FC Barcelona football match. He wants systems like Sportsvision’s on-screen infoviz to be able to understand and represent such seemingly ineffable properties (and, by analogy, for our business and building modeling systems to be able to do the same) not just hard numbers.
And Gadney has a suggestion for an answer: “The answer, for both football and buildings, will emerge from a more holistic, performative sophistication in collection and visualisation, as a ‘total design’”. He suggests that this “total design” could be based on “parametric models describing the interdependency between performative elements in buildings”:
“Parametric models indicate how a change to one component of a structure causes ripples of changes through all the other connected elements, mapped across structural loads but also environmental characteristics, financial models and construction sequencing. FC Barcelona’s activity is also clearly parametric in this sense. It cannot be understood through sensors tracking individuals but only through assembling the whole into one harmonious, interdependent system.”
Here, I think, is where Gadney gets into trouble. The problem with this idea of “total design” is that it bleeds into Cybernetics on the one side and AI on the other and hence falls into the problems that haunt both of those disciplines.
What is the victory condition for an advanced parametric model of the subtle strands of inter-relations between players on a football pitch?
Start with the Cybernetics option. Is the goal predicting the future of the game? Predicting when and if individual goals will be scored? If so, that falls into the classic modeling problem that doomed large-scale cybernetics efforts such as World3 and George Van Dyne’s work in the Colorado grasslands. Capturing increasing amounts of data doesn’t cause your model of the complex system to converge on the real world behavior. Instead it causes it to either act stochastically (as in Van Dyne’s case) or fail catastrophically because of flawed assumptions (as in the case of World3’s failure to predict the green revolution and the liberalization of global trade).
The failures of these statistics-based complex systems models lead directly to the rise of chaos theory and a new modesty across disciplines like ecology. The scientists involved learned respect for how little rich data actually helps in the problem of modeling complex systems. However, many other industries did not learn such respect. One of the most prominent of these was the financial industry which built probably the most sophisticated and detailed data-driven systems modeling tools in the world. As is now emerging these tools were a major contributor to the overdeveloped sense of confidence and control that plagued financial industry operators, leading directly to the 2008 crisis.
On the AI side, we have another proposal for the goal of such a “total design”. Maybe the victory condition is that the system shares our aesthetic appreciation of the game? The traditional aim of AI is to reproduce human capacities in the machine. Towards that aim we might use increased tracking of players and increasingly sophisticated models of game dynamics to make our computational systems into passionate fans of games, fans that can appreciate the complex shifting patterns of FC Barcelona’s tik-tak passing the way we do.
However, how you would measure (or even clearly define) machine “appreciation” is a philosophical problem that has plagued hard AI proponents since the beginning of the discipline. And, I would argue, a problem on which they’ve made basically no progress in that time. The reason for that lack of progress to my mind is that the core of AI itself is a bad metaphor. As Bruce Sterling argued compellingly in his Long Now talk, The Singularity: Your Future as a Black Hole: “we don’t know what cognition is and we don’t even really know what computation is”. So how can we expect to jump straight into subtle problems like building aesthetic appreciation into computation. Even if you’re not convinced by the Serle arguments against hard AI, I think you’d have to see this as something of an obstacle to setting something like this as the goal of a sport- or business- or building-infoviz system, despite the potential poetic beauty of doing so. (I wrote more about this problem and how I think it’s evolving in the context of present day technology in my post, AI Unbundled).
All of that said, what is the goal of such a “total design” system if it cannot be complex systems management or AI aesthetic appreciation? An alternate goal for such a system was articulated by Doug Engelbart at SRI in the 60s: that of human augmentation. Computational systems should seek to augment human experiences and abilities: learning, recall, access, communication.
What would an infoviz system look like that was aimed at the Augment goals? Rather than offering an illusory sense of control or the pathetic fallacy of some machinic aesthetic understanding, such a system would aim to enhance what human beings gain from sport: a sense of the beauty of perfected human movement, the thrill of competition, especially when rooted in the emotion-sharing and amplification herd-behaviour of crowds, etc. I don’t know how you’d go about using data to augment these human experiences, but I know that you’d be much better off with David Foster Wallace’s NY Times essay on Roger Federer as Religious Experience as a starting point than Cybernetics or AI.