Learning Analytics Fellows

Fellows Projects

  • 2017

    Measuring the Impact of Information Literacy Instruction on Assignment-Level Learning Outcomes

    Andrew Asher (Library Academic Services)

    Using artifacts created by students while completing research-oriented course assignments, this study will evaluate the impact of library-provided information literacy instruction on students’ learning outcomes. In particular, this study will seek to understand the types of information literacy instruction that are most effective, the times during a student’s course of study that instruction is most impactful, and the types of students that most benefit from instructional interventions.  Additionally, this study will evaluate the cumulative and long-term effects of instructional interventions on students’ acquisition of critical thinking and information literacy skills, and will assist the IUB Libraries in allocating instructional resources and developing course-level and curriculum-level instructional interventions that are most effective, impactful and sustainable.


    Evaluating and Optimizing Homework and Quizzes to Increase Learning Outcomes in the Information Visualization MOOC

    Katy Borner (Intelligent Systems Engineering)

    The Information Visualization MOOC (IVMOOC) is a graduate level course taught each spring since 2013. It is offered for credit as part of the Information and Library Science and Data Science curriculums at IU(Z637), and offered for free to members of the public. As part of efforts to update course materials, instructors need to evaluate course activities to improve learning outcomes of students taking the course. The proposed work will analyze quizzes and homework submissions from 500 students that have taken or will take the Information Visualization course between 2016 and 2018. Student data from 2016 and 2017 will be analyzed to understand how students use and perform on course activities. Insights gained from this analysis will be used to optimize course materials for the 2018 version of the course. The revised course homework assignments and quizzes will be implemented for the 2018 course and used by two parallel cohorts. Student engagement and performance will be analyzed for 2018 and compared with the 2016 and 2017 data. Results will be written up in scholarly publications that empower other teachers to evaluate the impact of assessment quizzes and hands-on homework activities on student engagement and performance in order to improve instructional materials and learning outcomes.


    Using Analytics to Evaluate Influences on Student Learning Outcomes in a GenEdScience Course (G131, Oceans & Our Global Environment)

    Simon Brassell (Geological Sciences)

    This proposal seeks to utilize analytics on student demographics and grade records in combination with data on their performance in the GenEd NMS class G131 “Oceans and Our Global Environment” to assess how performance in the class and its range of assignments may be related to specific student characteristics.The results obtained from evaluation of the analytics data, including relationships examined using statistical methods, will be used to identify modifications to G131 that should be implemented (e.g., changes to help students disadvantaged by their prior science background, decrease withdrawal rates, enhance performance in subsequent classes) to improve the learning outcomes of students when the course is revised during 2017/18. The overall aim of the project is to use analytics to identify what characteristics serve as predictors for student performance in G131and determine how revisions to the course may help enhance student success in this class.


    Evaluating the Impact of the Intensive English Program on Student Success at IU

    Leslie Gabriele (Second Language Studies)

    This study will evaluate the impact of the Intensive English Program (IEP), an academic English language preparation program for pre-matriculated international students, on students’ future academic success using external measures such as University/College admission rates and student progress through their programs of study at Indiana University – GPAs, retention rates, and graduation rates. In the short term, the goal of the project is to identify factors that contribute to the variability of success among students, including, but not limited to student demographics and progression data through the IEP. In the long term, the results will be used for targeted curriculum improvement, and programmatic initiatives to maximize the impact of the IEP on admission rates, retention, and graduation at IU.


    Evaluating the Teaching Effectiveness of Principles of Microeconomics Instruction at the IU Bloomington Campus Relative to Other Institutions

    Paul Graf, Gerhard Glum (Economics)

    Enrollments in the College of Arts and Sciences at IU are continuing to shrink as, among other reasons, more and more incoming students choose to transfer into IU credits taken at other institutions, in particular from community colleges such as Ivy Tech. This transfer of credits applies to classes such as E201 Principles of Microeconomics. As community colleges and Research 1 Universities like IU employ different pedagogies, e.g. different class sizes and different teaching personnel, teaching effectiveness may well vary across these two institutions.


    The Factors of Differential Grading Standards across Academic Units 

    Michael Kaganovich (Economics)

    The proposed research is the second phase of the project Determinants of Students' Choices of Undergraduate Majors and Programs' Strategies started under this program’s aegis in Spring 2016.  The long-term agenda of this research is to identify factors contributing to IU undergraduate students' choices of their major concentrations, and especially to infer the actions and policies of major programs in effecting those student decisions. In the next phase of the project, we will employ the experience gained in organizing the data set and the initial results to obtain broader and more sophisticated understanding of the evolving grading standards across IU academic programs.


    The Factors to and Impact of K303 Success

    Jie Li (Business)

    Building on top of the research conducted by Kari Johnson in 2015, I would like to gather more data about K303, learn the data from different angles, use different analysis techniques, to further study the factors that might have certain predicting power of student performance in K303 and the impact of the course on students’ later academic performance, major selection and career development outcomes.


    Learning Analytics for Students Majoring in Healthcare Management and Policy

    Terri Renner (SPEA)

    Many students enrolled in SPEA’s Healthcare Management and Policy degree program (HMP) struggle in our quantitative courses in financial management and economics. SPEA has modified admissions criteria over the years in an effort to identify students who may have difficulty completing the degree, but the faculty is not satisfied that we have addressed this issue effectively.  My research question is as follows: What is the profile of the student who is successful (grade of “C” or better) in healthcare finance and economics courses? The data examined would include demographic characteristics and prior coursework that may be predictive of success in our Healthcare Management and Policy quantitative courses.  The ultimate goal is to use this information to identify students early on who may struggle in these courses and therefore offer early intervention options, such as appropriate prerequisite courses, a SPEA “math camp”, or online math “Modules” in conjunction with appropriate tutoring.  The anticipated result is improved student performance in these challenging courses, as well as increased retention and graduation rates.


    Human Expertise, Analytics, & Student Learning in Multi-Section General-Education Courses at Indiana University 

    Jennifer Robinson, John Arthros, and Jill Robinson (Anthropology, English, Chemistry)

    The HumAn Learning Project uses learning analytics to triangulate on strategies for fostering student success in multi-section, general education courses. Phase 3, proposed here, extends and further tests the premises and applications of this research by partnering faculty members who teach large, general education courses at IU.  The new work will (1) marshal institutional data on learning and student progress (e.g., demographics, college preparation, GPA, retention, choice of major, time to degree) in order to develop further multivariate modeling of student success and (2) design and implement classroom interventions based on our prior research that also reflect disciplinary ways of knowing.


    Exploring relationships Between the New Indiana Academic English Test (IAET) and External Measures

    Sun-Young Shin (Second Language Studies)

    The Department of Second Language Studies (SLS) has developed and administered the Indiana English Proficiency Exam (IEPE) to assess English proficiency of matriculated international students for the decisions made about their exemptions from or placements into the English Language Improvement Program (ELIP). However, the current IEPE exam has been used for a while without being revised and needs to be updated to enhance the degree of alignment between the test content and the ELIP curriculum. This proposed study will explore the relationships between students’ standardized language test scores (e.g., the TOEFL iBT and IELTS) and the IAET writing and listening scores to establish the concurrent validity of the test. It will also examine the consequential validity of the test by comparing the students’ GPA data across groups of students who are exempted from the test, tested out of the test, placed into different ELIP courses, or do not comply with placement recommendations based on test scores. This analysis will help us to see if proper placement and instructional intervention should result in measurable benefits to the students.


  • 2016

    Evaluating the Effect of Course-specific Library Instruction on Student Success

    Andrew Asher (Anthropology and IU Libraries)

    Using records of course-specific instruction provided by the IUB Libraries, this study will evaluate the impact of library instruction sessions on measures of students’ academic success and educational development. In particular, this study will seek to understand the types of library instruction that are most effective, the times during a student’s course of study that this instruction is most impactful, and what groups of students may especially benefit from additional instructional interventions. Additionally, this study will evaluate the potential cumulative and long-term effects of library instruction on students’ success and will create a model that will facilitate the assessment of the Libraries instructional programs that will assist the allocation of instructional resources and development of course-level and curriculum-level instructional interventions that are most effective, impactful, and sustainable.


    S&H Fulfillment Patterns and Their Effect on Student Retention

    Kalani Craig (History)

    Higher ed institutions must balance the need for undergraduate students to complete their degrees in a timely manner while simultaneously ensuring they graduate with the skills to make them effective in the workforce. To encourage that balance, our goal is to study patterns in student fulfillment of IUB GenEd S&H (Social & Historical Studies) requirements and attempt to understand how the variety of options available for S&H transfer credits affects student engagement, retention, and performance. Specifically, we will measure whether and how retention rates and academic performance differ for students who fulfill GenEd S&H requirements through dual-enrollment courses in history. We will compare these students to students who fulfilled S&H requirements via AP, transfers from 2-year institutions, and from IU-system institutions, with an eye to documenting and understanding the effect of dual-enrollment credit on student performance, retention and engagement. The data that results from this project will help inform ongoing changes to Indiana’s ACP curriculum in history, IUB history department curriculum, and to the literature on how dual-enrollment courses should and can function in disciplines where learning outcomes are primarily made in less easily measurable skills like critical thinking, argumentation and writing.


    Inflection Points of Economics Majors: A closer look at enrollments in Intermediate Microeconomics (E321)

    Paul Graf (Economics)

    In my previous SLAF study of Economics majors (see 2015 SLAF Grants), I was unable to determine more specific inflection points on when and possibly why students opted in or out of an Economics major. In this follow-up study, I plan to see if instructor selection, class pedagogy, and student preferences and expectations in Intermediate Microeconomics (E321) caused some of these changes. I will use the grade penalty measurement for E321 to see if this had any effect. In addition, I will examine transfer students who took equivalent economics courses at other institutions and compare them to the rest of the program’s students across several courses.


    The Impact of the “Becoming the Best Student” and “You@IU” Courses

    Anthony Guest-Scott (Student Academic Center)

    This study will analyze the impact of two of IUB’s “metaclasses”—classes about college classes, college academics, and college life more generally. These metaclasses support a broad range of students, including first-generation, transfer, international, students on probation, and students who are motivated to improve their college learning. The study will examine the impact of two metaclasses (EDUC-X150: “Becoming the Best Student”; EDUC-X159: “You@IU”), covering a twelve year period from 2004–2016 and nearly 7,500 students. For our analysis, we plan to use both quantitative institutional data (e.g., course grade, CGPA, Major CGPA, GPA rates of change, and retention and graduation rates) and qualitative data (student evaluations).


    Using Analytics to Compare Student Demographics For Different Delivery Methods (Face to Face, Hybrid, Online) of AMST-A 100 What is America?

    Vivian Nun Halloran (English and American Studies)

    The American Studies department is reconfiguring one of its introductory courses (AMST-A 100 What is America?). As a large lecture course, the department is considering both various methods of delivery (face to face, hybrid, and online) and enrollment size. New versions of the course are already underway, and student analytics data will allow an analysis of student success in current courses compared to past years’ courses.


    Analyzing the Transition from Developmental to Supplemental Education

    Daniel Hickey (Education – Counseling & Educational Psychology)

    Efforts to broaden access to higher education have drawn new attention to remedial education. Traditionally, schools assign under-prepared students to non-credit "developmental" courses, most commonly in reading and math. While such courses fit neatly into schools, they usually deliver paltry rates of completion, future enrollment, and graduation for students placed in them. As an alternative, many now argue for increased focus on more major-specific developmental courses and "supplemental" approaches that are embedded directly in introductory courses or offered alongside them. These include peer-assisted study sessions, centralized drop-in services like writing centers, and a range of other options. The proposed project will explore all manner of accessible IUB evidence regarding undergraduate developmental and supplemental instruction. The project will then search for ways to use this data to study and enhance the precision with which students are placed in or offered these services and the success of these services. Given the large investment by students, programs, and schools in remedial education, the proposed project has the potential to dramatically impact success among struggling undergraduates at Indiana University.


    Determinants of Students’ Choices of Undergraduate Majors and the Program Strategies

    Michael Kaganovich (Economics)

    The project focuses on (i) quantifying the factors that contribute to IU students' choices of majors as well as to the changes in those student decisions in the course of their studies, and (ii) inferring the actions and policies of major programs in effecting those student decisions, such as revealing the moves in curricular and grading standards aimed at attracting and retaining appropriately prepared students. We will also explore (iii) the statistical relationship between mid-career salaries associated with IU majors and curricular requirements and grading standards in them, after controlling for effects of student and class characteristics that influence students’ decision to major in a particular discipline and performance in particular classes. Finally, we will examine (iv) the patterns of students' "switching" between the programs, in the process of students' exploration of the best fit between their interests and preparation on the one hand, and the content and requirements of the major programs on the other.


    Role of Peer Networks in Student Choice of Academic Major at IUB

    Adam Maltese (Education – Curriculum & Instruction)

    Research on academic persistence and major choice for university students is most often based on secondary analysis of institutional or national datasets, which limits the conclusions one can draw. This study will combine secondary analysis of a large corpus of longitudinal data with conducting our own primary data collection to study student decision-making processes at a much more fine-grained level and as it occurs rather than in post-hoc analysis. More specifically, we are going to focus on trying to evaluate the role of peer networks on major choice and persistence. We will come at this from two angles: 1) we will analyze existing institutional data on course enrollments to discover networks based on co-enrollment; and 2) we will analyze data from our ongoing study on major choice to understand how peer networks develop and evolve, and what roles these networks may play in major choice and persistence.


    The HumAn Learning Project Phase II
    Jennifer Robinson (Anthropology)
    Phase I of the HumAn Learning Project (see 2015 SLAF Grants) uncovered important demographic trends in success among the 800-1000 students and 9-10 AIs in a multi-section, general education course. Phase II (1) refines our understanding of these trends with multivariate modeling and by comparing them to student success in other IU general education courses, (2) assesses variability of student success across sections of course, (3) analyzes student performance over time, and (4) begins to implement interventions based on Phase I findings. As with all big courses, the stakes are high in A122. The course represents significant investments by the sponsoring departments, their schools, the graduate student section leaders, and the undergraduate students who hope to lay a foundation for successful college and professional careers. This study has broad implications for teaching and learning at IU because it locates campus-wide trends in a single course where they can be actively disrupted for greater student learning and overall success.

    Ethical Innovations: Exploring How Moral Reflection Benefits Learning Analytics Development
    James Willis III, Joshua Quick (Education – Counseling & Educational Psychology)
    Using student data for the purposes of learning analytics incurs various ethical problems in the processes of curation, analysis, implementation, and modeling. In the use of learning analytics, ethics often remain an “after-the-fact” consideration in innovation (spurred by legal or financial concerns), and ethical theories brought to bear on innovative technology practices are done so at a theoretical level, which results in a persistent disconnect between two fields that may well symbiotically benefit. An answer to these critical problems is to place moral reflection and ethical practice in direct alignment with the development of learning analytics. This study will develop ethical approaches to using student data that is both timely and applicable for users of learning analytics.
              Proposal and Completion Report

  • 2015

    MOOC Visual Analytics Tools
    Katy Börner (Information and Library Science)
    This study will investigate student learning in massive open online courses (MOOCs), developing data mining and visualization tools that render data collected during the MOOCs into actionable insights. This project will empower teachers, students, MOOC platform designers, and researchers to continually improve the implementation of and experiences gained during these courses.
              Proposal and Completion Report

    An Inquiry Into Student Purpose And Motivation as Catalysts For Retention
    Molly Burke, Anthony Guest-Scott, and Andrew Koke (Student Academic Center)
    This study will investigate the efficacy of key aspects of the EDUC-X158 curriculum, examining a wealth of longitudinal data gathered in this course since 2009. EDUC-X158 is a retention course required for University Division students placed on Academic Probation. The researchers will particularly look at recent changes to the curriculum to determine to what degree a more robust examination of student motivation, major selection, and the multivalent purposes of higher education has influenced metrics of the course such as retention and grade point average.
              Proposal and Completion Report

    Do General or Specific Characteristics of E201 And E202 Affect the Number of Economics Majors?
    Paul Graf (Economics)
    This study will use analytical data to investigate the consistent upward trend in student demand for Economics courses. The IUB Economics undergraduate major concentration has grown by 257% over the last decade and is now the third largest undergraduate major in the College. Graff will examine class size, instructor selection, class pedagogy, and student preferences and expectations of two Economics courses (E201 & E202) to determine if enrollment and success in these courses results in a net increase in the number of Economics majors.
              Proposal and Completion Report

    Beyond Surveys and Data Mining: Searching for New and Potentially More Useful Indicators of Student Engagement
    Daniel Hickey (Counseling and Educational Psychology)
    Previous studies that measure student engagement in higher education have traditionally relied on self-report surveys of students’ activities and satisfaction, with little evidence that institutions or educators used this data to improve engagement or other outcomes. In contrast, this project will analyze unstructured text found within the artifacts that students generate while enrolled in online and hybrid courses in the School of Education. Hickey will analyze the fine-grained details of “productive disciplinary engagement” and the relationships of those artifacts to other indicators of engagement and success. Leading scholars outside of IU who are pursuing directly relevant research will be invited to give talks at IU and consult with the project.
              Proposal and Completion Report

    Finding the Keys to Success in Business X201
    Kari Johnson (Business)
    This study will use analytical data to better understand students enrolled in Business X201-Technology and Business Analytics. Johnson will analyze three student characteristics: incoming skills before the course, class activities and performance during the course, and graduation rate/job placement after the course. Johnson plans to use this data to better predict student success in the course, correlating those characteristics with other student learning outcomes.
              Proposal and Completion Report

    An Investigation of Factors Related to Student Choice of Academic Major at IUB
    Adam Maltese (Curriculum and Instruction)
    This study will investigate student academic persistence and choice of major using a multimodal strategy involving both existing longitudinal data and a collection of new primary data. Maltese plans to analyze key factors with both breadth (across a large volume of data) and depth (being able to follow-up with students actively making these decisions). The data will be collected at an individual class level in order to understand more broadly how student experiences and performance play a role in higher level decisions they make about major choice and persistence.
              Proposal and Completion Report

    Undergraduate Legal System Courses and Where They Fit in the Curriculum for Best Learning Outcomes
    Shannon Martin (Journalism)
    This study will investigate if student success in legal classes can be correlated to the time in their academic careers that they enroll in particular courses. Martin will use university student course sequencing data, looking for correlations of particular courses as a preparation or precursor to legal system courses and student success in those legal system courses. Recommendation to curriculum committees on the placement and timing of legal system undergraduate courses across programs is one possible outcome for this study. More specifically, as The Media School begins to revise and reimagine the curriculum, this research could be used immediately while at the same time the results may be useful to any of the units included in the data set who, like the Media School, are considering curriculum adjustments or re-invention.
              Proposal and Completion Report

    Learning Analytics in Recreation, Park, and Tourism Studies: The Impact of Two Courses on Student Performance, Major, Selection, and Degree Completion
    Rasul Mowatt, Sarah Young, Julia Knapp, and Jared Allsop (Recreation, Park, and Tourism Studies)
    This study will use analytical data collected from two core curriculum courses to gain better insights into the RPTS Curriculum and the type of decisions student make. The project will look for correlations between high student achievement in core curriculum courses, success in other RPTS courses, and the factors that lead students to choose RPTS as their major. Researchers will also study the affect student success in the core curriculum courses has upon timely graduation rates.
              Proposal and Completion ReportALS Teaching Institute 2016 Presentation

    The HumAn Learning Project (Humanities, Analytics, & Learning in a Multi-Section General-Education Course)
    Jennifer Robinson (Communication and Culture, Anthropology)
    This study will investigate the variability of student success across sections of a single large multi-sectional course, seeking patterns in demographics, teaching methods, and learning outcomes that can be used to gain greater student success in future iterations of the course. In the short term, the goal of the project is to identify factors that contribute to the variability of success among student cohorts, and begin to address points of opportunity that will improve the experience of both students and instructors. In addition, the project seeks to pilot an integrative model of analysis, one that combines quantitative and qualitative data to create a humanistically-informed portrait of teaching/learning that can be shared meaningfully with instructors in the humanities as well as others.
              Proposal and Completion Report

    Relevant Contributors to Student Success in a Non-Introductory course with a Highly Diverse Student Demographic
    Jeffrey Whitmer (Computer Science and Informatics)
    This study is concerned with effective teaching and learning in non-introductory courses with very diverse student populations, such as is often found in the School of Informatics and Computing. For example, the School of Informatics and Computing offers a variety of courses that can include Computer Science majors, Informatics majors, and other undergraduate majors from literally any program on the IUB campus. This study will examine whether eventual student success in these diversely-populated course can be attributed to one or more identifiable factors, including class rank, past student success in prerequisite courses, and past student success in other non-perquisite courses.