ICSOB 2023 Best Paper: Understanding User Feedback in Software Ecosystems
Over the years, ISPMA has supported the excellence in academic research by sponsoring the Best Paper Award at the International Conference on Software Business (ICSOB). This distinguished event bring together the brightest minds in software business and software product management research domains. We are delighted to extend our warmest congratulations to Bachan Ghimire, Ze Shi Li, and Daniela Damian for their outstanding paper, “Understanding User Feedback in Software Ecosystems: A Study on Challenges and Mitigation Strategies.” This paper has made a significant contribution to the field of Software Product Management, offering valuable insights that benefit both the academic community and industry practitioners. Bachan Ghimire, the primary author, has agreed to share the key findings of this research in our blog. For further details about the paper and the conference, we invite you to visit the ICSOB website. You can also contact Bachan directly at for more information.
Author: Bachan Ghimire, Research Affiliate | University of Victoria
In the rapidly evolving world of software ecosystems, understanding user feedback is a complex yet crucial task. The challenges are manifold: consolidating feedback from various channels for a single platform, distinguishing user feedback based on the core platform, partner apps, or multiple applications, and the general lack of motivation among users to provide feedback in Software Ecosystems (SECOs). These challenges make it difficult to generalize common problems and devise effective mitigation strategies. Furthermore, the absence of studies focusing on end-user problems in software ecosystems underscores the need for research in this area.
In the study, we collected 2.4 million user reviews from 283 platforms. We identified 40,000 reviews about Software Ecosystems (SECOs) using manual and automated techniques. We created a training dataset of 5,000 SECO reviews, labeled by problem types. We trained an XGBoost model to categorize reviews into six problem categories with 93% accuracy. We used sentiment analysis for negative reviews and statistical techniques (TF-IDF and Chi-Squared analysis) to find problem-associated features. We also conducted semi-structured interviews with platform owners to understand problem handling. This approach helped us deeply understand user feedback in software ecosystems.
Our research identified six main categories of challenges within Software Ecosystems (SECOs): Integration, Customer Support, Design & Complexity, Privacy & Security, Cost & Pricing, and Performance & Compatibility. Each of these categories represents a unique set of issues faced by users. For instance, users have reported problems such as dysfunctional API errors, a preference for live customer support over chat-based interactions, and even switching to competitors due to the complexity of the interface.
There are also concerns about scams and fake reviews in marketplaces, unexpected charges to bank accounts, and issues related to device-specific and operating system-specific compatibility. Interestingly, our findings show that developers tend to respond less frequently to SECO reviews compared to general reviews. Moreover, SECO-related reviews have seen significant growth in the past five years, a trend that has been amplified by the integration demands brought on by the COVID-19 pandemic.
To effectively address these challenges, we recommend several strategies. Platforms should consider adopting an API-first mentality, fostering a supportive community, implementing stringent vetting processes in marketplaces, and employing live feedback tracking tools. Additionally, considering the development of cross-platform software, establishing comprehensive documentation and guidelines, and maintaining transparent policies in user data management could also prove beneficial. These recommendations aim to improve the user experience and address the challenges faced by both end-users and developers in software ecosystems.
Looking ahead, there are several exciting directions for future research in the area of user feedback in software ecosystems. These include broadening the scope of our study to include more sources of feedback, refining our machine learning model with a larger dataset, and identifying new categories of challenges as software ecosystems evolve. We also see potential in evaluating the effectiveness of our suggested mitigation strategies in real-world scenarios, conducting longitudinal studies to track changes over time, and exploring the impact of external factors such as market trends or regulatory changes. These avenues promise to deepen our understanding of user feedback in software ecosystems and help us develop more effective strategies to address the challenges faced by end-users and developers.
