ivxol Bot

ivxol is the collective name for the web crawlers used by ivxol Search to discover, analyze, and index publicly accessible content across the internet.

ivxol currently operates two crawler types:

• ivxol Web Crawler — A general-purpose crawler that discovers and indexes web pages, documents, and other text-based content.
• ivxol Media Crawler — A specialized crawler that discovers publicly accessible images and media resources.

You can identify the crawler subtype by examining the HTTP User-Agent header included with each request. While the crawlers may perform different functions, both respect the same robots.txt directives and website owner preferences.

How ivxol Accesses Your Website

ivxol discovers new URLs through links found on previously crawled pages, XML sitemaps, and other publicly available discovery mechanisms.

For most websites, ivxol is designed to crawl responsibly and efficiently, minimizing server impact while maintaining a comprehensive and current search index. Crawl frequency may vary depending on factors such as website size, update frequency, server responsiveness, and webmaster directives.

Before accessing content, ivxol checks the website’s robots.txt file and follows applicable instructions. Website owners retain full control over crawler access through standard web protocols.

Public Content Only

ivxol is designed to index publicly accessible information only. Our crawlers avoid intentionally accessing private, restricted, or protected content and respect website owner directives regarding crawl permissions.

Blocking ivxol From Your Website

Website owners can control ivxol crawling through robots.txt.

To prevent ivxol from accessing your website:

User-agent: ivxol
Disallow: /

To allow ivxol to access your website:

User-agent: ivxol
Allow: /

Specific directories may also be controlled individually:

User-agent: ivxol
Disallow: /private/
Disallow: /admin/

Crawling and Indexing

Crawling and indexing are separate processes.

Blocking ivxol from crawling content prevents our crawlers from accessing that content. If content should remain accessible to visitors but not appear in search results, website owners should use appropriate indexing directives or access controls.

For content that should not be accessible to either users or crawlers, authentication and access restrictions should be used.

Our Commitment

ivxol was built on the principles of transparency, privacy, fairness, and respect for the open web.

Our crawlers:

• Respect robots.txt directives.
• Identify themselves clearly.
• Focus on publicly available content.
• Minimize unnecessary server load.
• Operate without collecting user browsing profiles.
• Support an open and accessible web ecosystem.

ivxol’s goal is to build a high-quality, privacy-focused index of the public web while giving website owners clear and effective control over how their content is accessed.