feat: Scrapping Documentations from website for building Knowledge Graphs

[debrupf2946]

Implement Python Script for Scraping Documentation into Llama-Index Document Objects

Description

This issue invites contributors to develop a Python script that scrapes documentation from websites

Objective

• Create a Python script to scrape documentation from a given website, including all sublinks.
• Store the content from each link in a separate Llama-Index document object.
• Each document object should have metadata that includes the source URL of the scraped data.
• Compile all document objects into a list to form the complete Llama-Index.

Requirements

1. Scraping the Documentation:

   • Utilize Python libraries like Beautiful Soup, Scrapy, or Requests-HTML to scrape content from the main documentation page and all associated sublinks.
   • Ensure accurate extraction of relevant content, including text, code snippets, and descriptions.

2. Llama-Index Document Object Creation:

   • Store the data scraped from each individual link in a separate Llama-Index document object.
   • Attach metadata to each document object that records the URL of the link from which the content was scraped.
   • Compile all individual document objects into a list, representing the complete Llama-Index.

3. Documentation:

   • Document the script clearly, providing instructions on how to use it.
   • note-book implementation for various strategies and responses mostly research
   • Later we can build a module out of it.

4. Error Handling:

   • Implement robust error handling to manage issues such as broken links, failed requests, or unexpected data formats.

Submission Guidelines

• Implementation: Develop the Python script ensuring it meets the outlined requirements.

• Documentation: Include comprehensive comments and docstrings that explain the functionality and usage of the script.

• Submit a Pull Request (PR):

  • Reference this issue in your PR.
  • Provide a description of your implementation, any challenges faced, and considerations made during development.

Resources

• Explore Python libraries such as Beautiful Soup, Scrapy, and Requests-HTML for web scraping.
• Refer to Llama-Index documentation for guidance on creating document objects and managing metadata.

Submission Checklist

• Implement the scraping script in Python.
• Create individual Llama-Index document objects with correct metadata.
• Compile document objects into a list representing the Llama-Index.
• note-book implimentation.
• Submit a PR referencing this issue.

We look forward to your valuable contributions that will enhance our capability to integrate website documentation into our knowledge systems!

[SarangShelke2304]

hi, is anyone working on this? can i do it?

[sriramsowmithri9807]

Hello everyone,

I am excited to take on the task of developing a Python script to scrape documentation from a specified website and create Llama-Index document objects. Below is a summary of my approach and the implementation details:

Implementation Overview

Web Scraping:

I will utilize the requests library to handle HTTP requests and BeautifulSoup for parsing the HTML content. This will allow us to extract relevant text, code snippets, and descriptions from the main documentation page and all associated sublinks.

Llama-Index Document Creation: For each scraped link, I will create a separate LlamaIndexDocument object that will store the content along with metadata, including the source URL.

Recursive Scraping:

The script will recursively follow sublinks within the same domain to ensure comprehensive coverage of the documentation.

Error Handling:

I will implement robust error handling to manage issues such as broken links, failed requests, or unexpected data formats, ensuring the script runs smoothly.

Documentation:

The script will be well-documented with comments and docstrings to provide clear instructions on how to use it.

Next Steps

I will begin by implementing the script and will keep you updated on my progress. Once completed, I will submit a Pull Request (PR) that references this issue, along with a description of the implementation and any challenges I encountered.

Thank you for the opportunity to contribute to this project! If you have any suggestions or specific requirements, please feel free to share.

Best regards,
Sowmithri Sriram
sriramsowmithri9807 --> github