We developed a classifier to identify turn-grabbing intentions in video conferences, adopting positive-unlabeled (PU) learning. This method uses a small set of labeled positive samples while treating rest of the data as unlabeled, potentially containing both positive and negative instances.

In video conferencing, turn-grabbing intentions are inferred from participant behavior. When a participant becomes a new speaker, they likely have a turn-grabbing intention just before the previous speaker's utterance ends. We extract positive samples from video clips preceding these events while treating all other data as unlabeled.

Data Collection

We gathered 121 video recordings from YouTube and Vimeo, focusing on conversations involving 3 to 9 participants, predominantly in gallery view, and primarily in English. This dataset includes 112 recordings (135.8 hours) for training, 3 recordings (2.8 hours) for validation, and 6 recordings (4.2 hours) for testing.

Sample Extraction

To detect turn-taking events, we applied Active Speaker Detection (ASD) alongside OpenFace for facial feature extraction. A smoothing operation was applied to mitigate ASD glitches. A four-second interval, starting up to 10 seconds before speech onset, was randomly sampled as input data.

Model Training

We selected 19 features from OpenFace, focusing on 17 Facial Action Units and 2 Gaze directions, excluding other features to reduce overfitting observed in preliminary tests. Our lightweight network architecture consists of two 1D convolutional layers followed by an LSTM and a final fully connected layer for binary classification.

Using the model mentioned above, we achieved a Matthews Correlation Coefficient (MCC) of 0.175, an Area Under the Curve (AUC) of 0.602, an F-score of 0.392, an accuracy of 0.648, a precision of 0.315, and a recall of 0.519 on the test set.

Our system simulates the natural leaning-forward behavior to signal a desire to speak. When our predictive model detects this intention, the system manipulate the video to mimic a lean-forward motion. This helps balance conversations subtly and effortlessly.

FairTalk analyzes webcam video using OpenFace to extract facial expressions and gaze. It feeds this data into a machine learning model to predict turn-grabbing intention. Upon detecting an intention, FairTalk applies visual effects to the video with OBS. Predictions pause if the system detects ongoing speech.

This work is licensed under a CC BY-NC-SA: Creative Commons Attribution-Noncommercial-ShareAlike License.