Crawl4ai Installation

 Table of Content:

  • What is Crawl4ai?
  • Installing Crawl4ai

  • Running Crawl4ai commands
  • Conclusion


What is Crawl4ai?


Crawl4ai is an AI web crawler built for LLMs (Large Language Models). A web crawler, in simple terms, is a program that browses the Internet to search and collect data from webpages.

The data is collected by traversing through hyperlinks in the web pages and indexing them.
Search Engines use these programs to find relevant information to display in search results.


However, AI web crawlers have a different main purpose. They access information on the web to either:

  • help AI assistants provide information to users.
  • help train Large Language Models(e.g. GPT Series, Gemini, etc..)

Crawl4ai is a popular open source and flexible AI web crawler, boasting blazing-fast performance. It can be installed through its GitHub repository and a website that provides its documentation.

Links:
  • GitHub Repository: https://github.com/unclecode/crawl4ai
  • Documentation: https://docs.crawl4ai.com/core/cli/

Installing Crawl4ai

There are only a few commands to run to install Crawl4ai. This easy to install package is why it is one of the most popular web crawler on GitHub.

You can go for a:
  •  a basic python package installation for basic web crawling or scraping task.
  •  or development installation meaning for contributors who plan to modify the source code.


However before installing,

I would encourage users to create an environment so as to ensure packages are installed without conflicts with existing dependencies or system configurations. It also allows users to install packages without requiring administrative privileges.


Steps to deploy environment:
  1. Create a python virtual environment
      python3 -m venv venv 

  2. Activate the virtual environment 
     source venv/bin/activate 
To deactivate just type,
           deactivate


  • Basic Installation
For basic installation, after activating the environment in a new directory you created, run the commands given on the repository:


If you encounter any Playwright-related errors, you can manually install it using:


  • Development Installation
For development installation, run the following commands provided on the repository.
NOTE: You should clone the repository, cd to crawl4ai then create the environment and run the last command:


There are also optional features that may be downloaded, for development purposes and they can be installed by:




Running Crawl4ai commands



Now after installing crawl4ai, you can run a basic snippet provided on the repository but beforehand you need to open a python interpreter on your system. In this case we can use 

        python

to open the default python interpreter then use the following commands provided below:


The result is:




NOTE : The screenshot above are only small sections of the result displayed on the terminal as it is quite long. Most of the information displayed has an associating hyperlink alongside.


We can also use the new command-line interface to directly use command like:



However, the only command I can use is the first one in the above picture which gives the same result as the code snippet used in python interpreter.

The second command is either not implemented or I unfortunately missed something. It gives error:


and the third command, we need to use an API key which I do not still understand.


Conclusion


This is a basic guide to the installation of crawl4ai. Crawl4ai is a good gateway to introduce yourself to the world of AI web crawlers and these bots have both good and bad repercussions.







Comments

Post a Comment

Popular posts from this blog

This blog will be dedicated to my journey related to programming, Linux, open source projects and much more.  It will show my progress and various projects.
Read more

Connecting to a server using OpenSSL

Image
  What is OpenSSL? OpenSSL is an open source cryptographic tool used to secure communication over computer networks. Using the terminal we are going to connect to cyberstorm.mu which is acting as the server here, using OpenSSL from our VM which is the client. The command to be used : openssl s_client cyberstorm.mu:443 Note: s_client means we are in client mode and 443 is the port through which we are connecting to cyberstorm.mu Result from the command But what we are most interested in all of the output given is the ' Server Temp Key ' field. This field will help us identify whether this connection is using a post-quantum secure connection. As we are only focusing on the ' Server Temp Key ' field, we are going to use ' grep ' command with a pipe so we can search that field from the given output. The command to be used :  openssl s_client cyberstorm.mu:443 | grep 'Server Temp Key' Result from above command How to know if a connection is using a post-quant...
Read more

Understanding GITHUB

Image
Understanding GITHUB What is GitHub? GitHub is made up of two main aspects.  It is a cloud storage for your repositories and it is built upon on and integrated with Git so it is a fantastic tool for Version Control. It is a web based platform built for collaboration with others, whether it is a group of colleagues in a small team in a company or a group of strangers spread across the world. We will focus mainly on GitHub in this blog and keep Git as a future topic to be discussed. CREATING A REPOSITORY Creating a repository named Test_template Here I am creating a new repository on GitHub with the option of Public sharing meaning anyone can try to collaborate with me on this repo whereas private option is usually for personal work not meant to  be shared or a project developed by a small team in a company.  You see a 'Add a README file' option turned on. What is a README file? It is basically the first thing that pops up when someone clicks on the repo. It is a markdown (...
Read more