{"work":{"id":"03df2169-4c1d-40f7-b5dc-edd5daaef02b","openalex_id":"https://openalex.org/W4400025328","doi":"10.48550/arxiv.2406.15513","arxiv_id":"2406.15513","raw_key":null,"title":"PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference","authors":null,"authors_text":"PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference , author=","year":2024,"venue":"cs.AI","abstract":"In this study, we introduce the safety human preference dataset, PKU-SafeRLHF, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs. Data is available at https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF.","external_url":"https://arxiv.org/abs/2406.15513","cited_by_count":2,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2406.15513","created_at":"2026-05-10T02:22:20.647006+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Pku-saferlhf: Towards multi-level safety alignment for llms with human preference","render_title":"Pku-saferlhf: Towards multi-level safety alignment for llms with human preference"},"hub":{"state":{"work_id":"03df2169-4c1d-40f7-b5dc-edd5daaef02b","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":24,"external_cited_by_count":2,"distinct_field_count":5,"first_pith_cited_at":"2024-12-07T08:07:24+00:00","last_pith_cited_at":"2026-07-08T20:42:46+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-21T20:39:44.911585+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":6},{"context_role":"dataset","n":1},{"context_role":"other","n":1}],"polarity_counts":[{"context_polarity":"background","n":6},{"context_polarity":"unclear","n":1},{"context_polarity":"use_dataset","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}