|
James A. LandayCHI becoming even more irrelevant?August 29, 2010Tweet It’s that time of year when many researchers are working hard to make their mad dash to the September CHI Conference papers submission deadline... it had me thinking (as I too make the dash with my collaborators), "Does CHI really matter?" You've seen my other complaints about CHI & UIST, but I'm really wondering if the work at CHI has any impact on industry. To justify itself as a "science", the CHI community seems to have become mostly a group of people who run studies about other people's work (e.g., let's go see what we can analyze about Facebook, Twitter, Wikipedia, etc.) and very little of the community seems to care about creating new visions/prototypes for the future that eventually industry might pick up (often much later with a path back to research that is hard to trace). Is industry the only place that can create interesting new ideas, so CHI needs to just become the backwater of studying other people's creations? Has CHI become what I see? Am I wrong? Is there a place for both kinds of work (studies / systems) at CHI? What do you think? |
blog
James, there is merit to your idea here, but let me throw in a different perspective: since users are very creative in appropriating technologies to serve their own ends, it can be very fruitful and productive of new ideas to study what people are actually doing with other people's creations. This carries on the iterative spiral of studying behavior, designing to it, studying behavior with that design, designing to it. . . So for me, it is a question of balance between studying behavior and creating new design ideas. I don't think everybody has to do both; I wouldn't want to go back to the point where, if you didn't translate your research findings into a prototype of some kind, you pretty much weren't going to get your paper accepted. So put me in the camp in favor of a balance between both kinds of work!
Just a quick comment: I'd love to see some empirical data on the type of publications that tend to be accepted at CHI. Perhaps we should categorize the last 10 or 15 years of CHI pubs in terms of whether they were offering a novel system or studying an existing one and explore the trend.
The systems and networking communities are undergoing the same realization it seems. I'm glad that at least we're all in this together. The bar for creating a notable system, be it to demonstrate an interaction, a new systems principle, or whatever is too high. Even if you do build a usable system, you'll only attract a limited set of users to use it -- e.g. grad students, or turkers. And forget about content or social processes -- can anything academics ever build compete with Wikipedia in this sense. And why should industry even care about academic systems? Google, and Facebook, and Microsoft have their own research staff who research and publish their own papers.
I think part of the problem is that computer scientists take it for granted that anything they do will be taken up by industry. I think its time to grow up and move on as a field towards basic research that may impact industry 5/10/20 years in the future. Is this a ludicrous idea, and why do the other sciences cope just fine when their projects have few immediate applications? Why is it that our field is so short-sighted, and fashion driven? Maybe its because in most CS sub-fields there is a very fine line between engineering and computer science, and most CS projects tend to have a bit of both.
@Ivan is partially right that it is hard and rare for academic research to have real impact in the real world. However, we should always strive for real impact, because without it, we are just a closed system, talking amongst ourselves.
I think that's James' worry. CHI cannot afford to disconnect with industry. The fact Apple hardly, if at all, send anyone to CHI is worrying to me. It sure points to how we might be losing grip on issues relating to hci system design.
I agree.
I agree too.
There is a growing realization that the western concept of "creating new visions" and innovation has largely outlived its usefulness. Look at Steven Johnson's new book, though this idea has been around since Koestler's Act of Creation at least. Framing the issue this way is problematic. Consider how would a zen master might respond.
Jonathan, please enlighten us. How might a zen master respond?
Obliquely. Tomorrow, the February issue of CACM is due out with my Viewpoint covering the key issue around CHI as I see it, or saw it when I submitted the piece in 2009 (they have a backlog). I won't repeat that here, but it does seem that people are not analyzing the issues deeply. The bar for creating a notable system is too high? Some notable applications have been created and shared in days. Consider how much harder it was when CHI emerged, before the Internet or Web, when journals were just in libraries and few people who did not attend a conference could find the proceedings, when platforms were more numerous and not interoperable at all, and development tools were few and far between. What has really changed? Share your hypotheses.
Jonathan, I just read your CACM article (for others: http://cacm.acm.org/magazines/2011/2/104400-technology-co..., though you need to get past the pay wall). Some really good points & perspective, but I must confess that given your comments on this post, I was expecting to read a different article.
First, I still don't understand your point about visions and innovation as outliving its usefulness (see Jonathan's previous comment). Why would it be useful previously, but not anymore? Elaborating on that point would be helpful.
Second, it seems like James has not been so much concerned with the difficulty of building systems, but the difficulty in justifying them to the research community. Hence system builders are sometimes asked to e.g. systematically compare them to existing, but marginally related systems in hugely time-consuming studies to produce results that can be passed off as scientifically valid, but is mostly a distraction as far as pushing the idea/vision forward goes.
This then puts system builders in a bind. On one hand, the university incentive structure emphasizes publication counting (seriously problematic IMHO), putting a handicap on systems research in relation to, say, analyses of Wikipedia (to pick on some of my previous work). On the other hand, it prevents systems builders from following through on their work enough to communicate effectively with industry, because they're so damn caught up in comparing-the-incomparable. New ideas suffer because they cannot be easily integrated.
So my reading of James' concerns is about the relative payoff in academia for doing systems work as it changes over time. He is concerned that the declining relative payoff is driving novelty out of CHI.
Do you think that moving to a different communication format, ala conferences/journals, is actually a sufficient move to address the above situation? I can put together a sort of argument along those lines, but it seems pretty weak when considering the incentive structures around organizations supporting research.
I'm really looking forward to see how the CSCW '12 process works. In your CACM piece, you mentioned that there have been many attempts to reformulate the conference process, but none of them radical; do you see the CSCW move to be bolder?
Briefly, as next week I'm co-chairing iConference 2011, a larger conference than recent CSCWs.
First, the CACM article is on my easily-found web site. Does UW have a site license for the ACM Digital Library?
Next, systems papers don't have acceptance troubles. For example, papers keyworded Tools/Toolkits/Programming Environments had 32% and 40% acceptance rates at CHI 2011 and CSCW 2011, far above the average. 'Software Engineering,' rarely associated strongly with CSCW, had 32% CSCW 2011 acceptance. Everyone whinges about persecution, but with skyrocketing submissions, raw numbers in almost every area rise. Are systems folks happy with almost twice the average acceptance rate?
I doubt it, getting to a point of the CACM article. Systems people -- and designers, usability folks, qualitative folks, etc. -- feel discriminated against because when we reject 75%-80% of the papers, we reject work that deserves to be discussed, and the authors know it. They know that decisions around the border are random, which is statistically verifiable. My central point is that this is bad for our sense of community. Everyone has a sense of grievance and entitlement. Even me, I think academics discriminate against papers studying use in organizations, but my statistical case might be weak if I looked deeply. How do systems builders think they look to everyone else, complaining despite high acceptance rates? The big happy community disintegrates. That's what I've seen over 30 years. So it goes.
Moving to a different communication format? I didn't advocate abandoning workshops, conferences, journals. Technology can supplement them and we should focus on identifying appopriate roles for each. If the CACM Viewpoint wasn't clear, my web page links to a longer iConference 2010 paper. More CHI-specific data were presented at a UW Research Conversation last year, which may survive online.
Finally, although building and deploying systems seems easier now on average, appearing innovative is harder because one has hundreds of thousands of competitors, all with the Internet as a publicity medium. Before, fewer were doing it and if someone else did what you were doing, the odds of you hearing about it were often small. Your work could be appreciated locally. Few had a hope of appealing to more than dozens of users. Some projects were incubated at universities for many years before commercialization efforts. My views of the broad shift are in an Interactions Timelines article, March 2007, titled 'Living without parental controls: The future of HCI,' DOI on my web page. If you can't access it I'll send you a copy.
Cheers, see some of you at iConference 2011.
Jonathan