Title

Improving User Attention of Terms and Policies: NLP-based Highlighting and a User Study

DGIST Authors

Junsu Lim ; Sunjun Kim ; Donghoon Shin

Advisor

김선준

Co-Advisor(s)

Donghoon Shin

Issued Date

2022

Awarded Date

2022/02

Citation

Junsu Lim. (2022). Improving User Attention of Terms and Policies: NLP-based Highlighting and a User Study. doi: 10.22677/thesis.200000594728

Type

Thesis

Subject

Human-Computer Interaction, Usable Privacy, Natural Language Processing, Crowdsourcing

Description

Human-Computer Interaction, Usable Privacy, Natural Language Processing, Crowdsourcing

Table Of Contents

I. Introduction 1
1.1 Research Problem and Approach 1
1.2 Abbreviation Definition 4
1.3 Contributions 4
1.4 Thesis Structure 5
II. Background 7
2.1 Terms of Service & Privacy Policy 7
2.2 Crowdsourcing 8
2.2.1 Amazon Mechanical Turk 9
2.3 Natural Language Processing 10
2.4 Evaluation Metrics 11
2.4.1 Confusion Matrix 11
2.4.2 ROC Curve 13
2.4.3 Precision Recall Curve 14
2.5 Human-Computer Interaction Security 15
III. Methodology 17
3.1 Qualitative Research 17
3.1.1 Grounded Theory 17
3.2 Quantitative Research 18
3.2.1 Likert Scale 18
3.2.2 Cronbach’s Alpha 19
3.2.3 Shapiro-Wilk Test 20
3.2.4 Paired Samples t-Test 20
3.2.5 Wilcoxon Signed-rank Test 21
IV. Research Design 23
4.1 Research Question 23
4.1.1 Solution 23
V. Implementation 24
5.1 Data Collection 24
5.1.1 Crawler 24
5.1.2 Data Preprocessing for Crowdsourcing 24
5.2 Crowdsourcing 27
5.2.1 Task Design 29
5.2.2 Data Preprocessing for Natural Language Processing 30
5.3 Natural Language Processing 35
5.3.1 Bidirectional Encoder Representations from Transformers 35
5.3.2 Classification Model Evaluation 39
5.3.3 Dataset for User-study 40
5.4 Test Webpage 41
VI. Pre-study 43
6.1 Participants and Experiment Design 43
6.2 Apparatus 44
6.2.1 Interview Design 46
6.3 Result Analysis 47
VII. User-study 52
7.1 Participants and Experiment Design 52
7.1.1 Procedure 53
7.1.2 Apparatus 54
7.1.3 Interview Design 55
7.1.4 Questionnaire Design 56
7.2 Result Analysis 57
7.2.1 Qualitative Result Analysis 58
7.2.2 Quantitative Result Analysis 60
VIII. Discussion and Conclusion 67
8.1 Discussion 67
8.2 Conclusion 69
IX. Appendix 71
9.1 Demographic Survey 71
9.2 Interview Questions 71
9.2.1 Pre-study Interview Questions 71
9.2.2 User-study Interview Questions 72
9.3 User-study Questionnaires 73
9.4 Github Link 75
9.5 Critical Values of the Wilcoxon Signed Ranks Test 75
References 77

URI

http://dgist.dcollection.net/common/orgView/200000594728
http://hdl.handle.net/20.500.11750/16291

DOI

10.22677/thesis.200000594728

Degree

Master

Department

Information and Communication Engineering

Publisher

DGIST

Show Full Item Record