%0 Journal Article %J European Physical Journal Plus %D 2024 %T Gravity Spy: Lessons Learned and a Path Forward %A Michael Zevin %A Corey B. Jackson %A Zoheyr Doctor %A Yunan Wu %A Carsten Østerlund %A L. Clifton Johnson %A Christopher P. L. Berry %A Kevin Crowston %A Scott B. Coughlin %A Vicky Kalogera %A Sharan Banagiri %A Derek Davis %A Jane Glanzer %A Renzhi Hao %A Aggelos K. Katsaggelos %A Oli Patane %A Jennifer Sanchez %A Joshua Smith %A Siddharth Soni %A Laura Trouille %A Marissa Walker %A Irina Aerith %A Wilfried Domainko %A Victor-Georges Baranowski %A Gerhard Niklasch %A Barbara Téglás %X

The Gravity Spy project aims to uncover the origins of glitches, transient bursts of noise that hamper analysis of gravitational-wave data. By using both the work of citizen-science volunteers and machine-learning algorithms, the Gravity Spy project enables reliable classification of glitches. Citizen science and machine learning are intrinsically coupled within the Gravity Spy framework, with machine-learning classifications providing a rapid first-pass classification of the dataset and enabling tiered volunteer training, and volunteer-based classifications verifying the machine classifications, bolstering the machine-learning training set and identifying new morphological classes of glitches. These classifications are now routinely used in studies characterizing the performance of the LIGO gravitational-wave detectors. Providing the volunteers with a training framework that teaches them to classify a wide range of glitches, as well as additional tools to aid their investigations of interesting glitches, empowers them to make discoveries of new classes of glitches. This demonstrates that, when giving suitable support, volunteers can go beyond simple classification tasks to identify new features in data at a level comparable to domain experts. The Gravity Spy project is now providing volunteers with more complicated data that includes auxiliary monitors of the detector to identify the root cause of glitches.

%B European Physical Journal Plus %V 139 %P Article 100 %8 01/2024 %G eng %R 10.1140/epjp/s13360-023-04795-4 %0 Conference Proceedings %B Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems %D 2023 %T AngleKindling: Supporting Journalistic Angle Ideation with Large Language Models %A Petridis, Savvas %A Diakopoulos, Nicholas %A Crowston, Kevin %A Hansen, Mark %A Henderson, Keren %A Jastrzebski, Stan %A Nickerson, Jefrey V %A Chilton, Lydia B %B Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems %G eng %U https://savvaspetridis.github.io/papers/anglekindling.pdf %R 10.1145/3544548.3580907 %0 Journal Article %J Classical and Quantum Gravity %D 2023 %T Data quality up to the third observing run of Advanced LIGO: Gravity Spy glitch classifications %A Glanzer, J %A Banagiri, S %A Coughlin, S B %A Soni, S %A Zevin, M %A Berry, C P L %A Patane, O %A Bahaadini, S %A Rohani, N %A Crowston, K %A Østerlund, C %K FOS: Physical sciences %K General Relativity and Quantum Cosmology (gr-qc) %K Instrumentation and Methods for Astrophysics (astro-ph.IM) %B Classical and Quantum Gravity %V 40 %G eng %N 6 %R 10.1088/1361-6382/acb633 %0 Journal Article %J International Journal of Digital Curation %D 2021 %T Assessment, Usability, and Sociocultural Impacts of DataONE: A Global Research Data Cyberinfrastructure Initiative %A Robert J. Sandusky %A Suzie Allard %A Lynn Baird %A Leah Cannon %A Kevin Crowston %A Amy Forrester %A Bruce Grant %A Rachael Hu %A Robert Olendorf %A Danielle Pollock %A Alison Specht %A Carol Tenopir %A Rachel Volentine %X

DataONE, funded from 2009-2019 by the U.S. National Science Foundation, is an early example of a large-scale project that built both a cyberinfrastructure and culture of data discovery, sharing, and reuse. DataONE used a Working Group model, where a diverse group of participants collaborated on targeted research and development activities to achieve broader project goals. This article summarizes the work carried out by two of DataONE’s working groups: Usability & Assessment (2009-2019) and Sociocultural Issues (2009-2014). The activities of these working groups provide a unique longitudinal look at how scientists, librarians, and other key stakeholders engaged in convergence research to identify and analyze practices around research data management through the development of boundary objects, an iterative assessment program, and reflection. Members of the working groups disseminated their findings widely in papers, presentations, and datasets, reaching international audiences through publications in 25 different journals and presentations to over 5,000 people at interdisciplinary venues. The working groups helped inform the DataONE cyberinfrastructure and influenced the evolving data management landscape. By studying working groups over time, the paper also presents lessons learned about the working group model for global large-scale projects that bring together participants from multiple disciplines and communities in convergence research.

%B International Journal of Digital Curation %V 16 %8 04/2021 %G eng %N 1 %R 10.2218/ijdc.v16i1.678 %0 Journal Article %J Classical and Quantum Gravity %D 2021 %T Discovering features in gravitational-wave data through detector characterization, citizen science and machine learning %A S Soni %A C P L Berry %A S B Coughlin %A M Harandi %A C B Jackson %A K Crowston %A C Østerlund %A O Patane %A A K Katsaggelos %A L Trouille %A V-G Baranowski %A W F Domainko %A K Kaminski %A M A Lobato Rodriguez %A U Marciniak %A P Nauta %A G Niklasch %A R R Rote %A B Téglás %A C Unsworth %A C Zhang %B Classical and Quantum Gravity %V 38 %G eng %N 19 %R 10.1088/1361-6382/ac1ccb %0 Journal Article %J Innovation: Organization & Management %D 2021 %T Examining Open Innovation in Science (OIS): What Open Innovation can and cannot offer the science of science %A Susanne Beck %A Marcel LaFlamme %A Carsten Bergenholtz %A Marcel Bogers %A Tiare-Maria Brasseur %A Marie-Louise Conradsen %A Kevin Crowston %A Diletta Di Marco %A Agnes Effert %A Despoina Filiou %A Lars Frederiksen %A Thomas Gillier %A Marc Gruber %A Carolin Haeussler %A Karin Hoisl %A Olga Kokshagina %A Maria-Theresa Norn %A Marion Poetz %A Gernot Pruschak %A Laia Pujol Priego %A Agnieszka Radziwon %A Alexander Ruser %A Henry Sauermann %A Sonali Shah %A Julia Suess-Reyes %A Christopher L. Tucci %A Philipp Tuertscher %A Jane Bjørn Vedel %A Roberto Verganti %A Jonathan Wareham %A Sunny Mosangzi Xu %X

Scholars across disciplines increasingly hear calls for more open and collaborative approaches to scientific research. The concept of Open Innovation in Science (OIS) provides a framework that integrates dispersed research efforts aiming to understand the antecedents, contingencies, and consequences of applying open and collaborative research practices. While the OIS framework has already been taken up by science of science scholars, its conceptual underpinnings require further specification. In this essay, we critically examine the OIS concept and bring to light two key aspects: 1) how OIS builds upon Open Innovation (OI) research by adopting its attention to boundary-crossing knowledge flows and by adapting other concepts developed and researched in OI to the science context as exemplified by two OIS cases in the area of research funding; 2) how OIS conceptualises knowledge flows across boundaries. While OI typically focuses on well-defined organizational boundaries, we argue that blurry and even invisible boundaries between communities of practice may more strongly constrain flows of knowledge related to openness and collaboration in science. Given the uptake of this concept, this essay brings needed clarity to the meaning of OIS, which has no particular normative orientation toward a close coupling between science and industry. We end by outlining the essay’s contributions to OI and the science of science, as well as to science practitioners.

%B Innovation: Organization & Management %G eng %R 10.1080/14479338.2021.1999248 %0 Journal Article %J Physical Review D %D 2019 %T Classifying the unknown: Discovering novel gravitational-wave detector glitches using similarity learning %A Scott Coughlin %A Sara Bahaadini %A Neda Rohani %A Michael Zevin %A Patane, Oli %A Mahboobeh Harandi %A Corey Brian Jackson %A Noroozi, V. %A Sarah Allen %A Areeda, J. %A Coughlin, M. %A Ruiz, P. %A Berry, C. P. L. %A Kevin Crowston %A Aggelos Katsaggelos %A Andrew Lundgren %A Carsten Østerlund %A Joshua Smith %A Laura Trouille %A Vicky Kalogera %X

The observation of gravitational waves from compact binary coalescences by LIGO and Virgo has begun a new era in astronomy. A critical challenge in making detections is determining whether loud transient features in the data are caused by gravitational waves or by instrumental or environmental sources. The citizen-science project Gravity Spy has been demonstrated as an efficient infrastructure for classifying known types of noise transients (glitches) through a combination of data analysis performed by both citizen volunteers and machine learning. We present the next iteration of this project, using similarity indices to empower citizen scientists to create large data sets of unknown transients, which can then be used to facilitate supervised machine-learning characterization. This new evolution aims to alleviate a persistent challenge that plagues both citizen-science and instrumental detector work: the ability to build large samples of relatively rare events. Using two families of transient noise that appeared unexpectedly during LIGO's second observing run, we demonstrate the impact that the similarity indices could have had on finding these new glitch types in the Gravity Spy program.

%B Physical Review D %V 99 %P 082002 %G eng %N 8 %R 10.1103/PhysRevD.99.082002 %0 Journal Article %J Computers in Human Behavior %D 2017 %T Gamers, citizen scientists, and data: Exploring participant contributions in two games with a purpose %A Nathan Prestopnik %A Kevin Crowston %A Wang, Jun %X

Two key problems for crowd-sourcing systems are motivating contributions from participants and ensuring the quality of these contributions. Games have been suggested as a motivational approach to encourage contribution, but attracting participation through game play rather than intrinsic interest raises concerns about the quality of the contributions provided. These concerns are particularly important in the context of citizen science projects, when the contributions are data to be used for scientific research. To assess the validity of concerns about the effects of gaming on data quality, we compare the quality of data obtained from two citizen science games, one a “gamified” version of a species classification task and one a fantasy game that used the classification task only as a way to advance in the game play. Surprisingly, though we did observe cheating in the fantasy game, data quality (i.e., classification accuracy) from participants in the two games was not significantly different. As well, data from short-time contributors was also at a usable level of accuracy. Finally, learning did not seem to affect data quality in our context. These findings suggest that various approaches to gamification can be useful for motivating contributions to citizen science projects.

%B Computers in Human Behavior %V 68 %P 254–268 %G eng %R 10.1016/j.chb.2016.11.035 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/chb2016.pdf %0 Journal Article %J Communications of the Association for Information Systems %D 2015 %T Assessing IS research impact %A Niederman, Fred %A Kevin Crowston %A Koch, Hope %A Krcmar, Helmut %A Powell, Philip %A E. Burton Swanson %X

Based on the International Conference on Information Systems’ (ICIS) 2013 senior scholars’ forum, this paper shares insights on IS research impact assessment. We define research impact as conducting research that makes a difference to individuals, businesses, industries and societies. While assessment groups like AACSB (the Association to Advance Collegiate Schools of Business) want scholars to make an impact, sometimes they operationalize impact in ways that may encourage scholars to pursue research goals tangential to making a difference. The purpose of this paper is to stimulate thinking in the IS community on creating research assessment techniques that encourage our scholars to make a difference.

%B Communications of the Association for Information Systems %V 36 %& Article 7 %R 10.17705/1CAIS.03607 %0 Conference Proceedings %B iConference %D 2014 %T Exploring data quality in games with a purpose %A Nathan Prestopnik %A Kevin Crowston %A Wang, Jun %X

A key problem for crowd-sourcing systems is motivating contributions from participants and ensuring the quality of these contributions. Games have been suggested as a motivational approach to encourage contribution, but attracting participation through game play rather than scientific interest raises concerns about the quality of the data provided, which is particularly important when the data are to be used for scientific research. To assess whether these concerns are justified, we compare the quality of data obtained from two citizen science games, one a “gamified” version of a species classification task and one a fantasy game that used the classification task only as a way to advance in the game play. Surprisingly, though we did observe cheating in the fantasy game, data quality (i.e., classification accuracy) from participants in the two games was not significantly different. As well, the quality of data from short-time contributors was at a usable level of accuracy. These findings suggest that various approaches to gamification can be useful for motivating contributions to citizen science projects.

%B iConference %C Berlin, Germany %8 3/2014 %R 10.9776/14066 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/gamedataquality_cameraready_4.pdf %0 Conference Paper %B CHI '13 Extended Abstracts on Human Factors in Computing Systems %D 2013 %T Forgotten island: A story-driven citizen science adventure %A Nathan Prestopnik %A Souid, Dania %Y Mackay, Wendy E. %Y Brewster, Stephen %Y Bødker, Susanne %X

Forgotten Island, a citizen science video game, is part of an NSF-funded design science research project, Citizen Sort. It is a mechanism to help life scientists classify photographs of living things and a research tool to help HCI and information science scholars explore storytelling, engagement, and the quality of citizenproduced data in the context of citizen science.

%B CHI '13 Extended Abstracts on Human Factors in Computing Systems %I ACM Press %C Paris, France %P 2643–2646 %8 4/2013 %@ 9781450319522 %U http://delivery.acm.org/10.1145/2480000/2479484/p2643-prestopnik.pdf %! CHI EA '13 %R 10.1145/2468356.2479484 %0 Conference Paper %B Forty-sixth Hawai'i International Conference on System Sciences (HICSS-46) %D 2013 %T Motivation and data quality in a citizen science game: A design science evaluation %A Kevin Crowston %A Nathan Prestopnik %B Forty-sixth Hawai'i International Conference on System Sciences (HICSS-46) %C Wailea, HI %8 1/2013 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/hicss2013citizensort_cameraready.pdf %0 Conference Proceedings %B iConference 2012 %D 2012 %T Citizen science system assemblages: Understanding the technologies that support crowdsourced science %A Nathan Prestopnik %A Kevin Crowston %X We explore the nature of technologies to support citizen science, a method of inquiry that leverages the power of crowds to collect and analyze scientific data. We evaluate these technologies as system assemblages, collections of interrelated functionalities that support specific activities in pursuit of overall project goals. The notion of system assemblages helps us to explain how different citizen science platforms may be comprised of widely varying functionalities, yet still support relatively similar goals. Related concepts of build vs. buy and web satisfiers vs. web motivators are used to explore how different citizen science functionalities may lead to successful project outcomes. Four detailed case studies of current citizen science projects encompassing a cross-section of varying project sizes, resource levels, technologies, and approaches to inquiry help us to answer the following research questions: 1) What do typical system assemblages for citizen science look like? 2) What factors influence the composition of a system assemblage for citizen science? 3) What effect does the assemblage composition have on scientific goals, participant support, motivation, and satisfaction? and 4) What are the design implications for the system assemblage perspective on citizen science technologies? %B iConference 2012 %C Toronto, Ontario %8 2/2012 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/citizensciencesystemassemblage.pdf %0 Conference Paper %B Group '12 Conference %D 2012 %T Purposeful gaming & socio-computational systems: A citizen science design case %A Nathan Prestopnik %A Kevin Crowston %B Group '12 Conference %C Sanibel Island, FL, USA %8 10/2012 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/citizensort_cameraready.pdf %0 Generic %D 2011 %T Citizen Science System Assemblages: Toward Greater Understanding of Technologies to Support Crowdsourced Science %A Nathan Prestopnik %A Kevin Crowston %8 6/2011 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/system_assemblage_0.pdf %0 Generic %D 2011 %T Citizen science system assemblages: Toward greater understanding of technologies to support crowdsourced science %A Nathan Prestopnik %A Kevin Crowston %X We explore the nature of technologies to support citizen science, a method of inquiry that leverages the power of crowds to collect and analyze scientific data. We evaluate these technologies as system assemblages, collections of interrelated functionalities that support specific activities in pursuit of overall project goals. The notion of system assemblages helps us to explain how different citizen science platforms may be comprised of widely varying functionalities, yet still support relatively similar goals. Related concepts of build vs. buy, support for science vs. support for participants, and web satisfiers vs. web motivators are used to explore how different citizen science functionalities may lead to successful project outcomes. Four detailed case studies of current citizen science projects encompassing a cross-section of varying project sizes, resource levels, technologies, and approaches to inquiry help us to answer the following research questions: 1) What factors influence the composition of a system assemblage for citizen science? 2) What do typical system assemblages for citizen science look like? 3) What effect does the assemblage composition have on scientific goals, participant support, motivation, and satisfaction? and 4) What are the design implications for the system assemblage perspective on citizen science technologies? %I Syracuse University School of Information Studies %8 06/2011 %G eng %9 Unpublished working paper %1 CSCW 2012 submission, reworked to iConference 2012 submission %> https://crowston.syr.edu./sites/crowston.syr.edu/files/system_assemblage.pdf %0 Unpublished Work %D 2011 %T Exploring Collective Intelligence Games With Design Science: A Citizen Science Design Case %A Nathan Prestopnik %A Kevin Crowston %X Citizen science is a form of collective intelligence where members of the public are recruited to contribute to scientific investigations. Citizen science projects often use web-based systems to support collaborative scientific activities, but finding ways to attract participants and confirm the veracity of the data produced by non-scientists are key research questions. We describe a series of web-based tools and games currently under development to support taxonomic classification of organisms in photographs collected by citizen science projects. In the design science tradition, the systems are purpose-built to test hypotheses about participant motivation and techniques for ensuring data quality. Findings from preliminary evaluation and the design process itself are discussed. %1 Submitted to CI 2012 conference %> https://crowston.syr.edu./sites/crowston.syr.edu/files/designing%20citizen%20science%20games.pdf %0 Conference Paper %B “Computing for Citizen Science” workshop at the IEEE eScience Conference %D 2011 %T Gaming for (citizen) science: Exploring motivation and data quality in the context of crowdsourced science through the design and evaluation of a social-computational system %A Nathan Prestopnik %A Kevin Crowston %K Citizen Science %K data quality %K Design %K Design Science %K Games %K Gaming %K Motivation %K Participation %K Social Computational Systems %X In this paper, an ongoing design research project is described. Citizen Sort, currently under development, is a web-based social-computational system designed to support a citizen science task, the taxonomic classification of various insect, animal, and plant species. In addition to supporting this natural science objective, the Citizen Sort platform will also support information science research goals on the nature of motivation for social-computation and citizen science. In particular, this research program addresses the use of games to motivate participation in social-computational citizen science, and explores the effects of system design on motivation and data quality. A design science approach, where IT artifacts are developed to solve problems and answer research questions is described. Research questions, progress on Citizen Sort planning and implementation, and key challenges are discussed. %B “Computing for Citizen Science” workshop at the IEEE eScience Conference %C Stockholm, Sweden %8 12/2011 %U http://itee.uq.edu.au/~eresearch/workshops/compcitsci2011/index.html %> https://crowston.syr.edu./sites/crowston.syr.edu/files/gamingforcitizenscience_ver6.pdf %0 Conference Proceedings %B Proceedings of the European Conference on Information Systems (ECIS) %D 2001 %T Information and communication technologies in the real estate industry: Results of a pilot survey [Research in progress] %A Rolf Wigand %A Kevin Crowston %A Sawyer, Steve %A Allbritton, Marcel %E Smithson, Steve %E Gricar, Joze %E Podlogar, Mateja %E Avgerinou, Sophia %K Real Estate %X We have been studying the growing use of information and communication technologies (ICT) in the residential real estate industry and the effects of this use on how realtors work. Earlier stages of our project involved qualitative research to develop a better understanding of the industry, the work of realtors and their use of ICT. In this paper we report on the results of qualitative research and a pilot of a survey intended to gather large-scale data on realtors and ICT use. %B Proceedings of the European Conference on Information Systems (ECIS) %C Bled, Slovenia %P 339-343 %G eng %> https://crowston.syr.edu./sites/crowston.syr.edu/files/ecis2001.pdf %0 Journal Article %J Management Science %D 1999 %T Tools for inventing organizations: Toward a handbook of organizational processes %A Malone, Thomas W. %A Kevin Crowston %A Lee, Jintae %A Pentland, Brian %A Dellarocas, Chrysanthos %A Wyner, George %A Quimby, John %A Osborn, Charley %A Bernstein, Avi %A Herman, George %A Klein, Mark %A O'Donnell, Elissa %K Coordination %K Handbook %K Process %X This paper describes a novel theoretical and empirical approach to tasks such as business process redesign and knowledge management. The project involves collecting examples of how different organizations perform similar processes, and organizing these examples in an on-line "process handbook." The handbook is intended to help people: (1) redesign existing organizational processes, (2) invent new organizational processes (especially ones that take advantage of information technology), and (3) share ideas about organizational practices. A key element of the work is an approach to analyzing processes at various levels of abstraction, thus capturing both the details of specific processes as well as the "deep structure" of their similarities. This approach uses ideas from computer science about inheritance and from coordination theory about managing dependencies. A primary advantage of the approach is that it allows people to explicitly represent the similarities (and differences) among related processes and to easily find or generate sensible alternatives for how a given process could be performed. In addition to describing this new approach, the work reported here demonstrates the basic technical feasibility of these ideas and gives one example of their use in a field study. %B Management Science %V 45 %P 425–443 %G eng %N 3 %R 10.1287/mnsc.45.3.425 %> https://crowston.syr.edu./sites/crowston.syr.edu/files/ms99.pdf %0 Book Section %B Computational Organization Theory %D 1994 %T Evolving novel organizational forms %A Kevin Crowston %E Carley, Kathleen M. %E Prietula, Michael J. %K Coordination %X A key problem in organization theory is to suggest new organizational forms. In this paper, I suggest the use of genetic algorithms to search for novel organizational forms by reproducing some of the mechanics of organizational evolution. Issues in using genetic algorithms include identification of the unit of selection, development of a representation and determination of a method for calculating organizational fitness. As an example of the approach, I test a proposition of Thompson's about how interdependent positions should be assigned to groups. Representing an organization as a collection of routines might be more general and still amenable to evolution with a genetic algorithm. I conclude by discussing possible objections to the application of this technique. %B Computational Organization Theory %I Lawrence Erlbaum %C Hillsdale, NJ %P 19-38 %@ 0-8058-1406-X %G eng %> https://crowston.syr.edu./sites/crowston.syr.edu/files/CCSWP185.html %0 Conference Paper %B Computational Organization Design: 1994 AAAI Spring Symposium %D 1994 %T Using a Process Handbook to design organizational processes %A Dellarocas, C %A Lee, Jintae %A Malone, Thomas W. %A Kevin Crowston %A Pentland, Brian %E Hulthage, Ingemar %K Handbook %K Process %B Computational Organization Design: 1994 AAAI Spring Symposium %I AAAI Press %P 55-56 %@ 9780929280806