CRACS - Indexed Articles in Journals

Permanent URI for this collection

http://localhost:4000/handle/123456789/123

Browse

Browsing CRACS - Indexed Articles in Journals by Author "5125"
  • Item
    Automated Assessment in Computer Science Education: A State-of-the-Art Review
    ( 2022) Álvaro Figueira ; José Paulo Leal ; José Carlos Paiva ; 5088 ; 5125 ; 6251
    Practical programming competencies are critical to the success in computer science (CS) education and goto-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through a broad range of programming activities that learners must perform themselves. It is not reasonable to consider that teachers could evaluate all attempts that the average learner should develop multiplied by the number of students enrolled in a course, much less in a timely, deep, and fair fashion. Unsurprisingly, exploring the formal structure of programs to automate the assessment of certain features has long been a hot topic among CS education practitioners. Assessing a program is considerably more complex than asserting its functional correctness, as the proliferation of tools and techniques in the literature over the past decades indicates. Program efficiency, behavior, and readability, among many other features, assessed either statically or dynamically, are now also relevant for automatic evaluation. The outcome of an evaluation evolved from the primordial Boolean values to information about errors and tips on how to advance, possibly taking into account similar solutions. This work surveys the state of the art in the automated assessment of CS assignments, focusing on the supported types of exercises, security measures adopted, testing techniques used, type of feedback produced, and the information they offer the teacher to understand and optimize learning. A new era of automated assessment, capitalizing on static analysis techniques and containerization, has been identified. Furthermore, this review presents several other findings from the conducted review, discusses the current challenges of the field, and proposes some future research directions.
  • Item
    Automated Assessment in Computer Science Education: A State-of-the-Art Review
    ( 2022) Álvaro Figueira ; José Paulo Leal ; José Carlos Paiva ; 5088 ; 5125 ; 6251
    Practical programming competencies are critical to the success in computer science (CS) education and goto-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through a broad range of programming activities that learners must perform themselves. It is not reasonable to consider that teachers could evaluate all attempts that the average learner should develop multiplied by the number of students enrolled in a course, much less in a timely, deep, and fair fashion. Unsurprisingly, exploring the formal structure of programs to automate the assessment of certain features has long been a hot topic among CS education practitioners. Assessing a program is considerably more complex than asserting its functional correctness, as the proliferation of tools and techniques in the literature over the past decades indicates. Program efficiency, behavior, and readability, among many other features, assessed either statically or dynamically, are now also relevant for automatic evaluation. The outcome of an evaluation evolved from the primordial Boolean values to information about errors and tips on how to advance, possibly taking into account similar solutions. This work surveys the state of the art in the automated assessment of CS assignments, focusing on the supported types of exercises, security measures adopted, testing techniques used, type of feedback produced, and the information they offer the teacher to understand and optimize learning. A new era of automated assessment, capitalizing on static analysis techniques and containerization, has been identified. Furthermore, this review presents several other findings from the conducted review, discusses the current challenges of the field, and proposes some future research directions.
  • Item
    BabeLO-An Extensible Converter of Programming Exercises Formats
    ( 2013) Ricardo Queirós ; José Paulo Leal ; 5695 ; 5125
    In the last two decades, there was a proliferation of programming exercise formats that hinders interoperability in automatic assessment. In the lack of a widely accepted standard, a pragmatic solution is to convert content among the existing formats. BabeLO is a programming exercise converter providing services to a network of heterogeneous e-learning systems such as contest management systems, programming exercise authoring tools, evaluation engines and repositories of learning objects. Its main feature is the use of a pivotal format to achieve greater extensibility. This approach simplifies the extension to other formats, just requiring the conversion to and from the pivotal format. This paper starts with an analysis of programming exercise formats representative of the existing diversity. This analysis sets the context for the proposed approach to exercise conversion and to the description of the pivotal data format. The abstract service definition is the basis for the design of BabeLO, its components and web service interface. This paper includes a report on the use of BabeLO in two concrete scenarios: to relocate exercises to a different repository, and to use an evaluation engine in a network of heterogeneous systems.
  • Item
    Bibliometric Analysis of Automated Assessment in Programming Education: A Deeper Insight into Feedback
    ( 2023) Álvaro Figueira ; José Paulo Leal ; 5088 ; 5125
    Learning to program requires diligent practice and creates room for discovery, trial and error, debugging, and concept mapping. Learners must walk this long road themselves, supported by appropriate and timely feedback. Providing such feedback in programming exercises is not a humanly feasible task. Therefore, the early and steadily growing interest of computer science educators in the automated assessment of programming exercises is not surprising. The automated assessment of programming assignments has been an active area of research for over a century, and interest in it continues to grow as it adapts to new developments in computer science and the resulting changes in educational requirements. It is therefore of paramount importance to understand the work that has been performed, who has performed it, its evolution over time, the relationships between publications, its hot topics, and open problems, among others. This paper presents a bibliometric study of the field, with a particular focus on the issue of automatic feedback generation, using literature data from the Web of Science Core Collection. It includes a descriptive analysis using various bibliometric measures and data visualizations on authors, affiliations, citations, and topics. In addition, we performed a complementary analysis focusing only on the subset of publications on the specific topic of automatic feedback generation. The results are highlighted and discussed.
  • Item
    Bibliometric Analysis of Automated Assessment in Programming Education: A Deeper Insight into Feedback
    ( 2023) Álvaro Figueira ; José Paulo Leal ; 5088 ; 5125
    Learning to program requires diligent practice and creates room for discovery, trial and error, debugging, and concept mapping. Learners must walk this long road themselves, supported by appropriate and timely feedback. Providing such feedback in programming exercises is not a humanly feasible task. Therefore, the early and steadily growing interest of computer science educators in the automated assessment of programming exercises is not surprising. The automated assessment of programming assignments has been an active area of research for over a century, and interest in it continues to grow as it adapts to new developments in computer science and the resulting changes in educational requirements. It is therefore of paramount importance to understand the work that has been performed, who has performed it, its evolution over time, the relationships between publications, its hot topics, and open problems, among others. This paper presents a bibliometric study of the field, with a particular focus on the issue of automatic feedback generation, using literature data from the Web of Science Core Collection. It includes a descriptive analysis using various bibliometric measures and data visualizations on authors, affiliations, citations, and topics. In addition, we performed a complementary analysis focusing only on the subset of publications on the specific topic of automatic feedback generation. The results are highlighted and discussed.
  • Item
    A CLP-based tool for computer aided generation and solving of maths exercises
    ( 2003) José Paulo Leal ; 5125
    We propose an interesting application of Constraint Logic Programming to automatic generation and explanation of mathematics exercises. A particular topic in mathematics is considered to investigate and illustrate the advantages of using the CLP paradigm. The goal is to develop software components that make the formulation and explanation of exercise's easier. We describe exercises by grammars which enables us to get specialized forms almost for free, by imposing further conditions through constraints. To define the grammars we concentrate on the solving procedures that are taught instead of trying to abstract an exercise template from a sample of similar exercises. Prototype programs indicate that Constraint Logic Programming frameworks may be adequate to implement such a tool. These languages have the right expressiveness to encode control on the system in an elegant and declarative way.
  • Item
    A CLP-based tool for computer aided generation and solving of maths exercises
    ( 2003) José Paulo Leal ; 5125
    We propose an interesting application of Constraint Logic Programming to automatic generation and explanation of mathematics exercises. A particular topic in mathematics is considered to investigate and illustrate the advantages of using the CLP paradigm. The goal is to develop software components that make the formulation and explanation of exercise's easier. We describe exercises by grammars which enables us to get specialized forms almost for free, by imposing further conditions through constraints. To define the grammars we concentrate on the solving procedures that are taught instead of trying to abstract an exercise template from a sample of similar exercises. Prototype programs indicate that Constraint Logic Programming frameworks may be adequate to implement such a tool. These languages have the right expressiveness to encode control on the system in an elegant and declarative way.
  • Item
    crimsonHex: a learning objects repository for programming exercises
    ( 2013) José Paulo Leal ; Ricardo Queirós ; 5125 ; 5695
    A repository of learning objects is a system that stores electronic resources in a technology-mediated learning process. The need for this kind of repository is growing as more educators become eager to use digital educational contents and more of it becomes available. The sharing and use of these resources relies on the use of content and communication standards as a means to describe and exchange educational resources, commonly known as learning objects. This paper presents the design and implementation of a service-oriented repository of learning objects called crimsonHex. This repository supports new definitions of learning objects for specialized domains and we illustrate this feature with the definition of programming exercises as learning objects and its validation by the repository. The repository is also fully compliant with existing communication standards and we propose extensions by adding new functions, formalizing message interchange and providing a REST interface. To validate the interoperability features of the repository, we developed a repository plug-in for Moodle that is expected to be included in the next release of this popular learning management system. Copyright (c) 2012 John Wiley & Sons, Ltd.
  • Item
    Defining programming problems as learning objects
    ( 2009) Ricardo Queirós ; José Paulo Leal ; 5695 ; 5125
    Standards for learning objects focus primarily on content presentation. They were already extended to support automatic evaluation but it is limited to exercises with a predefined set of answers. The existing standards lack the metadata required by specialized evaluators to handle types of exercises with an indefinite set of solutions. To address this issue existing learning object standards were extended to the particular requirements of a specialized domain. A definition of programming problems as learning objects, compatible both with Learning Management Systems and with systems performing automatic evaluation of programs, is presented in this paper. The proposed definition includes metadata that cannot be conveniently represented using existing standards, such as: the type of automatic evaluation; the requirements of the evaluation engine; and the roles of different assets - tests cases, program solutions, etc. The EduJudge project and its main services are also presented as a case study on the use of the proposed definition of programming problems as learning objects.
  • Item
    Defining programming problems as learning objects
    ( 2009) Ricardo Queirós ; José Paulo Leal ; 5695 ; 5125
    Standards for learning objects focus primarily on content presentation. They were already extended to support automatic evaluation but it is limited to exercises with a predefined set of answers. The existing standards lack the metadata required by specialized evaluators to handle types of exercises with an indefinite set of solutions. To address this issue existing learning object standards were extended to the particular requirements of a specialized domain. A definition of programming problems as learning objects, compatible both with Learning Management Systems and with systems performing automatic evaluation of programs, is presented in this paper. The proposed definition includes metadata that cannot be conveniently represented using existing standards, such as: the type of automatic evaluation; the requirements of the evaluation engine; and the roles of different assets - tests cases, program solutions, etc. The EduJudge project and its main services are also presented as a case study on the use of the proposed definition of programming problems as learning objects.
  • Item
    A distributed system for learning programming on-line
    ( 2012) José Paulo Leal ; Ricardo Queirós ; 5125 ; 5695
    Several Web-based on-line judges or on-line programming trainers have been developed in order to allow students to train their programming skills. However, their pedagogical functionalities in the learning of programming have not been clearly defined. EduJudge is a project which aims to integrate the "UVA On-line Judge", an existing on-line programming trainer with an important number of problems and users, into an effective educational environment consisting of the e-learning platform Moodie and the competitive learning tool QUESTOURnament. The result is the EduJudge system which allows teachers to apply different pedagogical approaches using a proven e-learning platform, makes problems easy to search through an effective search engine, and provides an automated evaluation of the solutions submitted to these problems. The final objective is to provide new learning strategies to motivate students and present programming as an easy and attractive challenge. EduJudge has been tried and tested in three algorithms and programming courses in three different Engineering degrees. The students' motivation and satisfaction levels were analysed alongside the effects of the EduJudge system on students' academic outcomes. Results indicate that both students and teachers found that among other multiple benefits the EduJudge system facilitates the learning process. Furthermore, the experiment also showed an improvement in students' academic outcomes. It must be noted that the students' level of satisfaction did not depend on their computer skills or their gender.
  • Item
    EmoSpell, a Morphological and Emotional Word Analyzer
    ( 2018) José Paulo Leal ; 5125
    The analysis of sentiments, emotions, and opinions in texts is increasingly important in the current digital world. The existing lexicons with emotional annotations for the Portuguese language are oriented to polarities, classifying words as positive, negative, or neutral. To identify the emotional load intended by the author, it is necessary to also categorize the emotions expressed by individual words. EmoSpell is an extension of a morphological analyzer with semantic annotations of the emotional value of words. It uses Jspell as the morphological analyzer and a new dictionary with emotional annotations. This dictionary incorporates the lexical base EMOTAIX. PT, which classifies words based on three different levels of emotions-global, specific, and intermediate. This paper describes the generation of the EmoSpell dictionary using three sources: the Jspell Portuguese dictionary and the lexical bases EMOTAIX. PT and SentiLex-PT. Additionally, this paper details the Web application and Web service that exploit this dictionary. It also presents a validation of the proposed approach using a corpus of student texts with different emotional loads. The validation compares the analyses provided by EmoSpell with the mentioned emotional lexical bases on the ability to recognize emotional words and extract the dominant emotion from a text.
  • Item
    Ensemble - an E-Learning Framework
    ( 2013) Ricardo Queirós ; José Paulo Leal ; 5695 ; 5125
    E-Learning frameworks are conceptual tools to organize networks of e-learning services. Most frameworks cover areas that go beyond the scope of e-learning, from course to financial management, and neglects the typical activities in everyday life of teachers and students at schools such as the creation, delivery, resolution and evaluation of assignments. This paper presents the Ensemble framework - an e-learning framework exclusively focused on the teaching-learning process through the coordination of pedagogical services. The framework presents an abstract data, integration and evaluation model based on content and communications specifications. These specifications must base the implementation of networks in specialized domains with complex evaluations. In this paper we specialize the framework for two domains with complex evaluation: computer programming and computer-aided design (CAD). For each domain we highlight two Ensemble hotspots: data and evaluations procedures. In the former we formally describe the exercise and present possible extensions. In the latter, we describe the automatic evaluation procedures.
  • Item
    Fostering Programming Practice through Games
    ( 2020) José Paulo Leal ; Ricardo Queirós ; José Carlos Paiva ; 5125 ; 5695 ; 6251
    Loss of motivation is one of the most prominent concerns in programming education as it negatively impacts time dedicated to practice, which is crucial for novice programmers. Of the distinct techniques introduced in the literature to engage students, gamification, is likely the most widely explored and fruitful. Game elements that intrinsically motivate students, such as graphical feedback and game-thinking, reveal more reliable long-term positive effects, but those involve significant development effort. This paper proposes a game-based assessment environment for programming challenges, built on top of a specialized framework, in which students develop a program to control the player, henceforth called Software Agent (SA). During the coding phase, students can resort to the graphical feedback demonstrating how the game unfolds to improve their programs and complete the proposed tasks. This environment also promotes competition through competitive evaluation and tournaments among SAs, optionally organized at the end by the teacher. Moreover, the validation of the effectiveness of Asura in increasing undergraduate students' motivation and, consequently, the practice of programming is reported.
  • Item
    Gamification of Learning Activities with the Odin service
    ( 2016) José Paulo Leal ; Ricardo Queirós ; José Carlos Paiva ; 5125 ; 5695 ; 6251
    Existing gamification services have features that preclude their use by e-learning tools. Odin is a gamification service that mimics the API of state-of-theart services without these limitations. This paper presents Odin as a gamification service for learning activities, describes its role in an e-learning system architecture requiring gamification, and details its implementation. The validation of Odin involved the creation of a small e-learning game, integrated in a Learning Management System (LMS) using the Learning Tools Interoperability (LTI) specification. Odin was also integrated in an e-learning tool that provides formative assessment in online and hybrid courses in an adaptive and engaging way.
  • Item
    GEdIL-Gamified Education Interoperability Language
    ( 2020) José Paulo Leal ; José Carlos Paiva ; Ricardo Queirós ; 5125 ; 6251 ; 5695
    The paper introduces Gamified Education Interoperability Language (GEdIL), designed as a means to represent the set of gamification concepts and rules applied to courses and exercises separately from their actual educational content. This way, GEdIL allows not only for an easy yet effective specification of gamification schemes for educational purposes, but also sharing them among instructors and reusing in various courses. GEdIL is published as an open format, independent from any commercial vendor, and supported with dedicated open-source software.
  • Item
    Integrating the LMS in Service Oriented eLearning Systems
    ( 2011) Ricardo Queirós ; Luís Lopes ; José Paulo Leal ; 5695 ; 5126 ; 5125
    Learning management systems are routinely used for presenting, solving and grading exercises with large classes. However, teachers are constrained to use questions with pre-defined answers, such as multiple-choice, to automatically correct the exercises of their students. Complex exercises cannot be evaluated automatically by the LMS and require the coordination of a set of heterogeneous systems. For instance, programming exercises require a specialized exercise resolution environment and automatic evaluation features, each provided by a different type of system. In this paper, the authors discuss an approach for the coordination of a network of eLearning systems supporting the resolution of exercises. The proposed approach is based on a pivot component embedded in the LMS and has two main roles: (1) provide an exercise resolution environment, and (2) coordinate communication between the LMS and other systems, exposing their functions as web services. The integration of the pivot component in the LMS relies on Learning Tools Interoperability (LTI). This paper presents an architecture to coordinate a network of eLearning systems and validate the proposed approach by creating such a network integrated with LMS from two different vendors.
  • Item
    Learning path personalization and recommendation methods: A survey of the state-of-the-art
    ( 2020) José Paulo Leal ; 5125
    A learning path is the implementation of a curriculum design. It consists of a set of learning activities that help users achieve particular learning goals. Personalizing these paths became a significant task due to differences in users' limitations, backgrounds, goals, etc. Since the last decade, researchers have proposed a variety of learning path personalization methods using different techniques and approaches. In this paper, we present an overview of the methods that are applied to personalize learning paths as well as their advantages and disadvantages. The main parameters for personalizing learning paths are also described. In addition, we present approaches that are used to evaluate path personalization methods. Finally, we highlight the most significant challenges of these methods, which need to be tackled in order to enhance the quality of the personalization.
  • Item
    Managing experiments on cognitive processes in writing with HandSpy
    ( 2013) José Paulo Leal ; 5125
    Experiments on cognitive processes require a detailed analysis of the contribution of many participants. In the case of cognitive processes in writing, these experiments require special software tools to collect gestures performed with a pen or a stylus, and recorded with special hardware. These tools produce different kinds of data files in binary and proprietary formats that need to be managed on a workstation file system for further processing with generic tools, such as spreadsheets and statistical analysis software. The lack of common formats and open repositories hinders the possibility of distributing the workload among researchers within the research group, of re-processing the collected data with software developed by other research groups, and of sharing results with the rest of the cognitive processes research community. This paper describes the development of Hand Spy, a collaborative environment for managing experiments in the cognitive processes in writing. This environment was designed to cover all the stages of the experiment, from the definition of tasks to be performed by participants, to the synthesis of results. Collaboration in Hand Spy is enabled by a rich web interface. To decouple the environment from existing hardware devices for collecting written production, namely digitizing tablets and smart pens, Hand Spy is based on the InkML standard, an XML data format for representing digital ink. This design choice shaped many of the features in Hand Spy, such as the use of an XML database for managing application data and the use of XML transformations. XML transformations convert between persistent data representations used for storage and transient data representations required by the widgets on the user interface. Despite being a system independent from a specific collecting device, for the system validation, a framework for data collection was created. This framework has also been highlighted in the paper due to the important role it took in a data collection process, of a scientific project to study the cognitive processes involved in writing.
  • Item
    Managing Gamified Programming Courses with the FGPE Platform
    ( 2022) José Paulo Leal ; José Carlos Paiva ; Ricardo Queirós ; 5125 ; 6251 ; 5695
    E-learning tools are gaining increasing relevance as facilitators in the task of learning how to program. This is mainly a result of the pandemic situation and consequent lockdown in several countries, which forced distance learning. Instant and relevant feedback to students, particularly if coupled with gamification, plays a pivotal role in this process and has already been demonstrated as an effective solution in this regard. However, teachers still struggle with the lack of tools that can adequately support the creation and management of online gamified programming courses. Until now, there was no software platform that would be simultaneously open-source and general-purpose (i.e., not integrated with a specific course on a specific programming language) while featuring a meaningful selection of gamification components. Such a solution has been developed as a part of the Framework for Gamified Programming Education (FGPE) project. In this paper, we present its two front-end components: FGPE AuthorKit and FGPE PLE, explain how they can be used by teachers to prepare and manage gamified programming courses, and report the results of the usability evaluation by the teachers using the platform in their classes.