Project Description
Computer Science (CS) educators frequently develop new methodologies, languages, and programming environments to teach novice programmers the fundamental concepts of CS. A recent trend has focused on new environments that reduce the initial challenges associated with the heavy syntax focus of traditional environments. There are numerous Initial Programming Environments (IPEs) available that have been created for student use that in some cases have fostered self-discovery and inquiry-based exploration. In this qualifier paper, three IPEs are surveyed: Scratch, Lego Mindstorms LabVIEW, and App Inventor. These three IPEs were selected due to the high number of students and teachers currently using them, as well as my personal experience with these environments through past teaching activities. The creative and graphical nature of these three IPEs have been shown to increase student and teacher interest in CS through adoption in classrooms worldwide at the K-12 levels, as well as interest in introductory university courses. An additional reason for selecting these environments for the focused study is that I have assisted in offering professional development workshops for teachers and student summer camps utilizing these environments.
Although the graphical nature of these IPEs can be helpful for learning concepts in CS, a small group of students is being left out from learning experiences and engagement in CS. Graphical environments often require the use of both a mouse and keyboard, which motorically challenged users often are unable to operate. Based on research performed and presented in this paper, a Vocal User Interface (VUI) is a viable solution that offers a “Programming by Voice” (PBV) capability (i.e., a capability to describe a program without the use of a keyboard or mouse). There are two primary disadvantages with VUIs as a technology to address the limitations of motorically challenged users: 1) vocal strain can emerge for technical solutions that require a deep amount of vocal interactions, and 2) the process of integrating voice controls into a legacy application (e.g., an IPE) is a very time consuming process. There are existing vocal tools (e.g., the generic Vocal Joystick) that could be integrated into various applications; however, due to the needs of IPEs and the duration of IPE usage, the Vocal Joystick is not feasible due to the potential vocal strain, which is why a more command-driven approach may offer benefits for the PBV concept. The command-driven approach leads to a time-consuming process in terms of adapting legacy applications, particularly, if multiple applications (like the three IPEs previously mentioned) require specialized VUIs. Each environment has its own layout and its own commands; therefore, each application requires a different VUI. In order to create a more generic solution, Model- Driven Engineering (MDE) and Domain-Specific Languages (DSLs) can be applied to create a semiautomated process allowing a level of abstraction that captures the specific needs of each IPE. From the specification of each IPE, a customized VUI can be generated that integrates with the legacy application in a non-invasive manner.
Videos
Publications
An Empirical Evaluation of a Vocal User Interface for Programming by Voice
Amber Wagner and Jeff Gray, International Journal of Information Technologies and Systems Approach (Special Issue on Human-Computer Interaction, Carina Gonzalez, editor), , vol. 8, no. 2, May 2015, pp. 47-63.
Myna: Empowering Motorically Challenged Students to Program by Voice
Amber Wagner, SIGCSE 2014 Student Research Competition, Atlanta, Georgia, March 2014.
Awarded second place in Graduate Category
Automation of VUI to GUI Mapping
Amber Wagner, SIGCHI 2013 Doctoral Consortium, Paris, France, April 2013.
Programming by Voice: A Hands-Free Approach for Motorically Challenged Children
Amber Wagner, Ramaraju Rudraraju, Srinivasa Datla, Avishek Banerjee, Mandar Sudame, and Jeff Gray, ACM SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, May 2012, pp. 2087-2092.
Conferences/Abstracts
Observations of First Learners with Different Capabilities
Amber Wagner, Jeff Gray SIGCSE 2014, Atlanta, GA, March 2014.
An Empirical Evaluation of Myna – A Vocal User Interface
Amber Wagner, Jeff Gray SEWIC 2013, Guntersville, AL, November 2013.
Myna: Empowering Motorically Challenged Students to Program by Voice
Amber WagnerSEWIC 2013, Guntersville, AL, November 2013.
Creating a Vocal User Interface for Initial Programming Environments
Amber Wagner, Jeff Gray 2013 UA Systems Scholars Institute, Huntsville, AL, May 2013.
Creating a VUI for Initial Programming Environments
ACM Mid-Southeast Conference, November 2012
Awarded first place/Doctoral Division
Contributors
- Dr. Gary Edwards
- Dr. Jeff Gray
- Ramaraju Rudraraju
- Srinivasa Datla
- Avishek Banerjee
- Mandar Sudame
- Christopher Hodapp
- Josh Pepperman
- Kenny Kung
- Josh Wolfe
- Thomas Lewallen
- Zane Johnston
- Dr. Sandra Nichols
