backend.pipelines module

backend.pipelines.prepare_generate_pipeline(generate_template: str, generation_llm: ~llama_index.core.llms.llm.LLM = OpenAI(callback_manager=<llama_index.core.callbacks.base.CallbackManager object>, system_prompt=None, messages_to_prompt=<function messages_to_prompt>, completion_to_prompt=<function default_completion_to_prompt>, output_parser=None, pydantic_program_mode=<PydanticProgramMode.DEFAULT: 'default'>, query_wrapper_prompt=None, model='gpt-3.5-turbo', temperature=0.1, max_tokens=None, logprobs=None, top_logprobs=0, additional_kwargs={}, max_retries=3, timeout=60.0, default_headers=None, reuse_client=True, api_key='sk-proj-...', api_base='https://api.openai.com/v1', api_version=''), token_counter: ~llama_index.core.callbacks.token_counting.TokenCountingHandler | None = None, verbose: bool = False) → QueryPipeline

Prepare a pipeline for (re)generating a section of a privacy policy

Parameters:

  • generate_template – a template for (re)generating a section

  • generation_llm – a language model for (re)generating a section

  • token_counter – a token counter for counting tokens

  • verbose – whether to show verbose output

Returns:

a pipeline for (re)generating a section of a privacy policy

backend.pipelines.prepare_region_selection_pipeline(region_selection_template: str = "\nAssistant: I can help determine which region a user input belongs to.\nThe regions are: United States, California, European Union, United Kingdom, Canada, Australia, India, Singapore, Japan, South Korea, Brazil, South Africa, Other. Make sure to include all options.\n\nExamples:\nUser: los angeles\nAssistant: California, United States\nUser: hong kong\nAssistant: Other\nUser: berlin\nAssistant: European Union\nUser: toronto\nAssistant: Canada\nNow, let's try with your input.\nUser: {user_input}\nAssistant: The regions that the user input belongs to are:\n", token_counter: TokenCountingHandler | None = None, verbose: bool = False) → QueryPipeline

Prepare a pipeline for selecting regions based on user input

Parameters:

  • region_selection_template – a template for selecting regions

  • token_counter – a token counter for counting tokens

  • verbose – whether to show verbose output

Returns:

a pipeline for selecting regions

backend.pipelines.prepare_regulation_query_engine(links_df: DataFrame) → SubQuestionQueryEngine

Prepare a query engine for regulations

Parameters:

links_df – a DataFrame with columns ‘regulations’ and ‘links’

Returns:

a query engine for regulations

backend.pipelines.prepare_regulation_syllabus_pipeline(format_syllabus_template: str = 'Given the privacy policy sections and key points:\n\n{query_str}\n\nPlease merge the sections and key points from different syllabuses into one syllabus.\n\nThen, please rearrange the sections and key points into these sections: Introduction, Data Controller Information, Data Collection and Usage, Data Subject Rights, Data Sharing and Transfers, Data Retention, Disclosure of Personal Information, Security Measures, Automated Decision-Making and Profiling, Cookies and Tracking Technologies, Changes to the Privacy Policy, Contact Information, and output with the following JSON format:', token_counter: TokenCountingHandler | None = None, verbose: bool = False) → QueryPipeline

Prepare a pipeline for generating a privacy policy syllabus compliant with regulations

Parameters:

  • format_syllabus_template – a template for formatting a syllabus

  • token_counter – a token counter for counting tokens

  • verbose – whether to show verbose output

Returns:

a pipeline for generating a syllabus for regulations

backend.pipelines.prepare_regulation_syllabus_pipeline_with_query_engine(regulation_query_engine: SubQuestionQueryEngine, ask_syllabus_template: str = 'I want to write a privacy policy, which should be {regulations} compliant.\nWhat sections should it include? And for each section, what key points should be covered?\nList them as bullet points.', format_syllabus_template: str = 'Given the privacy policy sections and key points:\n\n{query_str}\n\nDO NOT modify the key points!!!\nPlease rearrange the sections and key points into these sections: Introduction, Data Controller Information, Data Collection and Usage, Data Subject Rights, Data Sharing and Transfers, Data Retention, Disclosure of Personal Information, Security Measures, Automated Decision-Making and Profiling, Cookies and Tracking Technologies, Changes to the Privacy Policy, Contact Information, and output with the following JSON format:', token_counter: TokenCountingHandler | None = None, verbose: bool = False) → QueryPipeline

Prepare a pipeline for generating a privacy policy syllabus compliant with regulations

Parameters:

  • regulation_query_engine – a query engine for regulations

  • ask_syllabus_template – a template for asking a syllabus

  • format_syllabus_template – a template for formatting a syllabus

  • token_counter – a token counter for counting tokens

  • verbose – whether to show verbose output

Returns:

a pipeline for generating a syllabus for regulations

backend.pipelines.prepare_section_judge_pipeline(law_llm: LLM, judge_section_template: str = 'This {section_name} is a section of a privacy policy:\n\n{section_text}\n\nPlease judge whether this section is compliant with the {regulations} regulations. If not, please provide suggestions on how to improve it.\nNOTE THAT this is just one section, not the whole privacy policy. Only judge this section in isolation.\n', format_judge_template: str = "Given the comments from a legal expert:\n\n{query_str}\n\nPlease extract the legal expert's suggestions on how to improve the sections in a privacy policy.\n", token_counter: TokenCountingHandler | None = None, verbose: bool = False) → QueryPipeline

Prepare a pipeline for judging a section of a privacy policy

Parameters:

  • law_llm – a law LLM to judge a section

  • judge_section_template – a template for judging a section

  • format_judge_template – a template for formatting a judge

  • token_counter – a token counter for counting tokens

  • verbose – whether to show verbose output

Returns:

a pipeline for judging a section of a privacy policy