Yotam Sechayk

Instructors often rely on visual actions such as pointing, marking, and sketching to convey information in educational presentation videos. These subtle visual cues often lack verbal descriptions, forcing low-vision (LV) learners to search for visual indicators or rely solely on audio, which can lead to missed information and increased cognitive load. To address this challenge, we conducted a co-design study with three LV participants and developed VeasyGuide, a tool that uses motion detection to identify instructor actions and dynamically highlight and magnify them. VeasyGuide produces familiar visual highlights that convey spatial context and adapt to diverse learners and content through extensive personalization and real-time visual feedback. VeasyGuide reduces visual search effort by clarifying what to look for and where to look. In an evaluation with 8 LV participants, learners demonstrated a significant improvement in detecting instructor actions, with faster response times and significantly reduced cognitive load. A separate evaluation with 8 sighted participants showed that VeasyGuide also enhanced engagement and attentiveness, suggesting its potential as a universally beneficial tool.

Improvisation training for actors presents unique challenges, particularly in maintaining narrative coherence and managing cognitive load during performances. Previous research on AI in improvisation performance often predates advances in large language models (LLMs) and relies on human intervention. We introduce ImprovMate, which leverages LLMs as GPTs to automate the generation of narrative stimuli and cues, allowing actors to focus on creativity without keeping track of plot or character continuity. Based on insights from professional improvisers, ImprovMate incorporates exercises that mimic live training, such as abrupt story resolution and reactive thinking exercises, while maintaining coherence via reference tables. By balancing randomness and structured guidance, ImprovMate provides a groundbreaking tool for improv training. Our pilot study revealed that actors might embrace AI techniques if the latter mirrors traditional practices, and appreciate the fresh twist introduced by our approach with the AI-generated cues.

In the realm of e-learning, video-based content is increasingly prevalent but brings with it unique accessibility challenges. Our research, beginning with a formative study involving 53 participants, has pinpointed the primary accessibility barriers in video-based e-learning: mismatches in user pace, complex visual arrangements leading to unclear focus, and difficulties in navigating content. To tackle these barriers, we introduced SmartLearn (SL), an innovative tool designed to enhance the accessibility of video content. SL utilizes advanced video analysis techniques to address issues of focus, navigation, and pacing, enabling users to interact with video segments more effectively through a web interface. A subsequent evaluation demonstrated that SL significantly enhances user engagement, ease of access, and learnability over existing approaches. We conclude by presenting design guidelines derived from our study, aiming to promote future efforts in research and development towards a more inclusive digital education landscape.

Humans use multiple communication channels to interact with each other. For instance, body gestures or facial expressions are commonly used to convey an intent. The use of such non-verbal cues has motivated the development of prediction models. One such approach is predicting arousal and valence (AV) from facial expressions. However, making these models accurate for human-robot interaction (HRI) settings is challenging as it requires handling multiple subjects, challenging conditions, and a wide range of facial expressions. In this paper, we propose a data augmentation (DA) technique to improve the performance of AV predictors using 3D morphable models (3DMM). We then utilize this approach in an HRI setting with a mediator robot and a group of three humans. Our augmentation method creates synthetic sequences for underrepresented values in the AV space of the SEWA dataset, which is the most comprehensive dataset with continuous AV labels. Results show that using our DA method improves the accuracy and robustness of AV prediction in real-time applications. The accuracy of our models on the SEWA dataset is 0.793 for arousal and valence.

eラーニングは，教材への幅広いアクセスを可能にすることを目的としている．しかし，動画コンテンツを多用することは，アクセシビリティに大きな課題をもたらす．多様な参加者を対象とした予備的調査に基づき，e ラーニングのビデオコンテンツに存在するアクセシビリティの障壁を明らかにする．これらには，ユーザーの理解速度と動画速度の不一致，どこに注意を向けてよいかわからない視覚的複雑さ，ナビゲーションのしにくさなどが含まれる．また，我々の調査結果は，アクセシビリティの問題が，障害のある利用者とない利用者の両方に影響を及ぼすことを示している．さらに，既存のアクセシビリティツールには限界があり，ささらなる対応が必要であることを示している．そこで我々は，e ラーニングアクセシビリティツールである「Smart Replay」を提案する．私たちのツールは，学習ビデオのビジュアルとコンテンツに基づいた分析を行い，アクセシブルな再生オプションを生成する．視覚的分野と時間的分野の両方を強化したビデオセクションの復習を可能にする．