Student Classroom Behavior Detection and Analysis Based on Spatio-Temporal Network and Multi-Model Fusion (2024)

III-A The SCB-ST-Dataset4 Production Process

Understanding student behavior is crucial for comprehending their learning process, personality, and psychological traits and is important in evaluating the quality of education. The hand-raising, reading, writing are important indicators of evaluating classroom quality. Based on our previous work, Based on the SCB-Dataset3-U dataset, we used the method shown in the Fig.2 to expand the image dataset into a spatio-temporal behavior dataset. This expansion method effectively reduces the required manpower and significantly increases the scale of video dataset creation. This expanded dataset is referred to as SCB-ST-Dataset4.

For the SCB-ST-Dataset4 (classroom dataset from kindergarten to high school), we collected over a thousand videos ranging from kindergarten to high school, each lasting approximately 40 minutes. We intentionally selected 3 to 15 frames with a specific time interval for each video to reduce imbalances in behavior classes and improve representativeness and authenticity. The videos were extracted from the “bjyhjy”, “1s1k”, “youke”. “qlteacher” and “youke-smile.shec” websites.

The production of SCB-ST-Dataset4 is divided into two parts. The first part entails the content on the left side of Fig.2 , which pertains to the production process of the image dataset. The right side of the figure outlines the procedure for expanding the image dataset into a spatio-temporal behavior dataset.

For the left part, firstly, we collected a series of suitable video frames from an online course website and carefully screened them. Next, we manually annotated these video frames to provide accurate label data for subsequent training. Then, we input the annotated data into the YOLOv7 model for training. By using this model, we can perform inference detection on the unannotated video frames, the model detects the target objects in the videos. We further fine-tune these detection results using artificial intelligence techniques to improve the accuracy and reliability of the detection while reducing human effort. Finally, we input the fine-tuned data back into the YOLOv7 model for training again to further optimize the model’s performance. Through iterative training, we continuously improve the accuracy and robustness of the model, resulting in more accurate detection results.

For the right part, each video in our extended spatio-temporal behavior dataset is set to a length of 5 seconds (30 frames per second). First, we select a video frame with existing labels as the frame for the 2nd second (61st frame). Then, a crucial step is taken: if we are able to track the position of this video frame in the video, we extract the preceding 60 frames and the subsequent 90 frames to form a continuous spatio-temporal behavior video. If we are unable to track the position of the video frame in the video, we replicate the video and make 60 copies in front and 90 copies at the back. Next, we generate labels for the spatio-temporal behavior dataset and input them into the spatio-temporal behavior detection network for training.

III-B The SCB-ST-Dataset4 Challenge

During the iterative training process on SCB-ST-Dataset4, we encountered an issue where there was an overlap between the bounding boxes of the “reading” and “writing” behaviors in the behavior detection (see Fig.3). Specifically, there were 8 instances where the bounding boxes overlapped between these two behaviors. This observation indirectly suggests that “reading” and “writing” behaviors are visually similar, making distinguishing between them more challenging for the YOLOv7 model. To improve the accuracy of the model, further exploration is needed to address this challenge.

And the classroom environment presents challenges for detecting behavior due to the crowded students and variation in positions, as shown in Fig.4 I and II. In addition to the high degree of similarity between reading and writing in certain scenarios, there is also a visual similarity between hand-raising and other classes of behavior. This further complicates the detection process, as shown in Fig.4 III.

The SCB-ST-Dataset4 was collected from different angles, including front, side, and back views (Fig.5 I). Additionally, the classroom environment and seating arrangement can vary, which adds complexity to the detection task (Fig.5 II). Student classroom behavior behaviors also differ across learning stages from kindergarten to high-school, creating challenges for detection (Fig.5 III).

III-C Statistics of The SCB-ST-Dataset4

Table I shows the one-year iteration process of our student behavior dataset, where SCB1 represents SCB-Dataset1, SCB2 represents SCB-Dataset2, SCB3-S represents SCB-Dataset3-S, SCB3-U represents SCB-Dataset3-U, and SCB4 represents SCB-ST-Dataset4 . The “class” row indicates the behaviors. The number 1 represents the hand-raising behavior, the number 3 represents the hand-raising, reading, and writing behaviors, and the number 6 represents the hand-raising, reading, writing, using phone, bowing the head, and leaning over the table. It can be seen from the table that our dataset has increased in both class and image quantity, especially notable is the rapid expansion of SCB-ST-Dataset4 in terms of image quantity.

Table II displays the number of different behavior classes in each dataset.

Some visual examples of the SCB-ST-Dataset4 are illustrated in Fig.6

IV-A Spatio-Temporal Network

SlowFast[6] is a dual-stream network structure. One stream processes the semantic information of the video’s spatial features at a low frame rate (Slow path, with frames input every $\tau$ frames, where $\tau=16$). The other stream processes the motion information at a high frequency (Fast path, with frames input every $\tau/\alpha$ frames, where $\alpha=8$). As shown on the left side of Fig. 7, the feature channels for the Slow path are represented by C, while the Fast path has $\beta C$ feature channels, where $\beta=1/8$, allowing the Fast path to run at a higher speed. In terms of time dimensions, the Slow path has a time dimension of T, while the Fast path has $\alpha$ T to better capture motion information. The paths share the same H and W dimensions to allow for lateral connections. SlowFast merges the features of the dual-stream branches via multiple lateral connections, and the merged information is then fed into a classifier for multi-label classification prediction.

IV-B Multi-Model Fusion

Our multi-model fusion system for student behavior analysis is shown in Fig.8. It consists of three parts: continuous behavior detection, non-continuous multi-model analysis, and the fusion of multi-model results.

To detect continuous student behavior, we sampled the video at 30 frames per second and used YOLOv7 with weights trained on the CrowdHuman dataset[34] to detect Person every 30 frames. The detection results were sent to Deep Sort and Spatio-Temporal Network(SlowFast). SlowFast detected continuous behaviors, We classified continuous behaviors into four major categories: student classroom behaviors (such as hand-raising, reading, writing, etc.), person pose (such as sit, stand, walk etc.), and person-person interaction (listen, talk, watch etc.). The student classroom behaviors are customized by us, and the remaining three categories come with SlowFast. Deep Sort is used to determine the ID of the person in the video.

For non-continuous multi-model analysis, the video was sampled at 1 frame per second. We used three models: YOLOv7 was employed to detect the identity of the student and the teacher, SynergyNet[35] was used for head pose estimation, and the Facial Expression model was implemented to recognize seven expressions: anger, disgust, fear, happiness, neutrality, sadness, and surprise.

Finally, we fuse the results of spatio-temporal behavior detection, Deep Sort, YOLOv7, SynergyNet, and the Facial Expression model to form a data table. The data table contains a row for each student/teacher, storing their multimodal result data.

We utilized YOLOv7 for student detection, but found it performed poorly in crowded classroom settings. Therefore, we adapted YOLOv7 to these settings by using weights trained on the CrowdHuman dataset.

IV-C Student Concentration Analysis

Fig.9 illustrates the process of analyzing student behavior concentration. We start with the data outputted by Student Behavior Analysis System with Multi-model Fusion, including student ID, time ID, and behavior ID. We then reorganize these behavioral data, as shown in step 1 of Fig.9, by grouping the behaviors of each student into raw behavioral sequences throughout the entire video. It should be noted that these raw behavioral sequences often contain a lot of repetition due to the continuity of behaviors. Next, we perform duplicate removal on the raw behavioral sequences, as shown in step 2 in Fig. 9, to obtain a behavioral sequence without duplicates.

In the bottom right corner of Fig.9, we present a set of concentrated sequences. These sequences are determined based on specific scenarios such as student self-study class, student interactive class, student experimental class, and student math class, where each scenario corresponds to a different concentrated sequence. These concentrated sequences can be proposed by experts in the field of education, and each sequence is assigned a corresponding weight score, w.

Next, in step 3 of Fig.9, we use the Longest Common Subsequence (LCS) algorithm to find the most similar sequence between the student behavioral sequences and the concentrated sequence group (It is also possible that a matching sequence may not be found), resulting in a similarity value. For example, in Fig.9, the similarity value between student s1 and concentrated sequence 1 is 0.75. Then, through traversal, we find the sequence values that have an intersection between the student behavioral sequences and the concentrated sequences (the matching length needs to be greater than or equal to 2), as shown in Fig.9, the intersected sequence of student s1 and concentrated sequence 1 is [2, 3, 4]. We have marked these intersected sequences with a green background.

Next is step 4 of Fig.9, where we map the intersected sequences back to the raw behavioral sequences. We employ the method of traversing through a subset sliding window to find the starting and ending points of the intersected sequences in the original sequences, achieving data mapping. In Fig.9, we have represented the mapped values with a green background.

Finally, in step 5 of Fig.9, we calculate the length of the concentrated sequences and the length of the unconcentrated sequences. Then, in step 6 of Fig.9, we compute the concentration score of the student behavioral sequences. The calculation formula is: concentrated * w / (concentrated + unconcentrated).

V-A Experimental Environment and Dataset

The experiment was compiled and tested using Python 3.8, the corresponding development tool was PyCharm, the main computer vision library was python-OpenCV 4.1.2, the deep learning framework used was Pytorch v1.11 with CUDA version 11.3 for model training, the operating system was Ubuntu 20.04.2, and the CPU: 12 vCPU Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz. The GPU: RTX 3080(10GB).

The dataset used in our experiments is SCB-Dataset3, SCB-ST-Dataset4 and StudentTeacher-Dataset, which we split into training, validation sets with a ratio of 4:1.

V-B Model Training

For SCB-Dataset3, the model was trained with an epoch set to 100, batch size set to 8, and image size set to 640x640. A pre-trained model was utilized for the training process.

As for SCB-ST-Dataset4, the model was trained with an epoch set to 43, batch size set to 2, learning rate set to 0.1, and image size set to 640x640. Similarly, a pre-trained model was used for the training.

V-C Evaluation Metric

When evaluating the results of an experiment, we use two main criteria: Precision and Recall. True Positive (TP) represents a correct identification, False Positive (FP) means an incorrect identification, and False Negative (FN) indicates that the target was missed.

To calculate Precision (Eq. 1), we divide the number of True Positives by the sum of True Positives and False Positives. For Recall (Eq. 2), we divide the number of True Positives by the sum of True Positives and False Negatives. Both Precision and Recall must be considered to properly assess the accuracy of the experiment.

To provide a more thorough evaluation of Precision and Recall, the metrics of Average Precision (AP, Eq. 3) and mean Average Precision (mAP, Eq. 4) have been introduced. These metrics calculate the average Precision over a range of Recall values, which provides a more comprehensive assessment of the model’s performance. AP is the average of Precision values at all Recall levels, and mAP is the mean AP value averaged over different classes.

As shown in Table III, mAP@50 refers to the mean average precision at IoU threshold 0.5, while mAP@50:95 represents the mean average precision at IoU thresholds ranging from 0.5 to 0.95.

To calculate the similarity of student behaviors, we introduce the BSI (Behavior Similarity Index), as shown in Eq. 5. In the equation, $S_{ij}$ represents the overlap count between behavior class i and behavior class j, $N_{i}$ represents the count of behavior class i, and $N_{j}$ represents the count of behavior class j. Therefore, BSI represents the overlap rate of behavior classes i and j, and BSI can to some extent reflect the similarity between behaviors.

V-D Analysis of spatio-temporal detect experimental results

For our training on SCB-Dataset3 and SCB-ST-Dataset4, we employed YOLOv5, YOLOv7 YOLOv8, SlowFast, The outcomes of our experiments are detailed in Table III. In the table,, v5s represents YOLOv5s, v5m represents YOLOv5m, v7 represents YOLOv7, v7x represents YOLOv7x, v8s represents YOLOv8s, v8m represents YOLOv8m, and SF represents SlowFast. From the table, it can be seen that SlowFast achieves the highest mAP@50, which is 82.3$\%$.

We compared the version of YOLO with the highest training accuracy in the YOLO series to SlowFast. As shown in Fig.10, SlowFast iterated a total of 43 times, while YOLOv7x iterated 100 times. From the graph, it can be seen that SlowFast quickly converges and achieves an accuracy of over 80$\%$ within the first 10 iterations. Whether it is from the convergence speed or the final accuracy, SlowFast outperforms YOLOv7.

In order to further analyze the detection accuracy of different networks for each behavior (with data imbalance between behaviors), using the weights from the last training weight, we compared the accuracy of YOLOv5m, YOLOv7x, YOLOv8m, and SlowFast models on three behaviors as shown in Table IV. It can be seen that compared to SlowFast, the YOLO series has better detection performance for the behavior ”writing” with less data. This is because during the training process, the YOLO series weights the imbalanced data, allowing the network to learn more from the minority samples, while SlowFast does not perform this operation.

Fig.11 demonstrates a comparison of the detection results between SlowFast and YOLOv7x on the same video. The results generated by SlowFast are enclosed within dashed lines.

In addition, we conducted experiments to validate the effectiveness of the BSI proposed by us. Through visual observation, we found that writing and reading are similar behaviors. Therefore, we proposed the BSI to quantify this similarity and recorded the experimental results in Table V, we used 0 to represent hand-raising, 1 to represent reading, and 2 to represent writing. Based on the results from Table V, we can observe that hand-raising behavior has almost no similarity with other behaviors while reading and writing exhibit a higher degree of similarity.

V-E Analysis of multi-model experimental results

YOLOv7+CrowdHumanWe utilized the YOLOv7x network (with a pre-trained model) and trained it for 200 epochs on the CrowdHuman dataset. The mAP@50 reached 85.6$\%$(Table VI). The comparison of model detection performance before and after training is shown in Fig.12. It can be seen that the performance of the YOLOv7x model trained on the CrowdHuman dataset has significantly improved in crowded classroom scenes.

Student Teacher Identification

First, we created a dataset with the identities of students and teachers (Student-Teacher Dataset), collecting a total of 785 videos. We sampled a frame every 25 seconds and labeled a total of 2785 video frames, with a total of 4078 labels. Detailed label information is shown in Table VII.

We trained the Student-Teacher Dataset on YOLOv7, achieving a mAP@50 of 94.5$\%$. Detailed information is shown in Table VIII. The detection results are shown in the second row of Fig.13.

The detection results for head pose and facial expression are depicted in the third and fourth rows of Fig.13, respectively.