For industrial automation professionals, access to technical documentation, software resources, and industry information is part of day-to-day work. When a website displays a bot-check page instead of its content, the cause may be an anti-scraping system such as Anubis. Its approach uses Anubis proof of work to make automated, high-volume access more expensive while keeping the additional effort for an individual visit relatively small.
What Anubis is designed to do
Anubis is a protective layer intended to reduce aggressive automated scraping. Large-scale scraping can place enough load on a website to contribute to service disruption, potentially affecting access for other visitors. Rather than treating every request as harmless, Anubis adds a computational challenge before allowing a visitor through.
The system is described as a compromise rather than a complete solution. Its proof-of-work approach raises the cost of repeated automated requests, but it can also require a challenge page to be shown to people who are trying to access the site legitimately.
How Anubis proof of work functions
Anubis uses a proof-of-work scheme similar in concept to Hashcash, a proposal developed to help reduce email spam. The basic idea is to require a small amount of computational work as evidence that a request has completed a challenge. The website can then use that proof when deciding whether to serve the requested content.
The cost is intended to be modest at the scale of an individual visit. At the scale of a scraper generating many requests, however, the work accumulates. Requiring each request to carry this additional cost can make high-volume scraping less attractive without imposing the same total burden on a typical visitor.
This mechanism does not mean that a visitor has necessarily done anything wrong. A challenge indicates that the websiteās protective system is asking for proof before proceeding; it is not, by itself, evidence that a person is a bot.
Why legitimate visitors may see a challenge
The challenge page is a way to apply the proof-of-work check before the site has confidently identified a visitor as likely legitimate. Anubis describes this as a temporary measure while work continues on methods for distinguishing automated browsers from ordinary ones.
One area identified for improvement is browser fingerprinting: looking at characteristics of how a browser behaves or renders content. Font rendering is given as an example of a feature that may help identify headless browsers. If a system can more reliably recognize visitors who are likely to be legitimate, it may be able to avoid showing them a proof-of-work page.
For organizations that rely on web-based technical resources, the practical point is that an unexpected bot check may come from a siteās traffic-protection policy rather than a problem with the resource itself. If the challenge does not complete, browser settings or extensions may be relevant.
JavaScript compatibility and browser extensions
Anubis requires modern JavaScript features to run its challenge. The siteās notice warns that privacy or security extensions such as JShelter may disable features the challenge depends on. When that happens, the proof-of-work step may not function as intended, and the page may remain inaccessible.
If a challenge page does not load or complete, a measured troubleshooting sequence can help identify whether browser compatibility is involved:
- Check whether JavaScript is enabled for the site.
- Review extensions that block or alter JavaScript features.
- If appropriate for your organizationās security policies, temporarily adjust the relevant extension setting for that domain.
- Reload the page and see whether the challenge proceeds.
These steps should be considered in the context of local security requirements. Disabling a protective extension broadly is not necessary based on the site notice; the compatibility issue is specific to the features required by the challenge.
Balancing access and protection
Anubis illustrates a familiar tradeoff in web infrastructure: a control intended to protect service availability can add friction for legitimate users. Proof of work distributes some cost to the request-making side, with the aim of making mass scraping more expensive. Browser identification may eventually reduce the need to show the challenge to visitors who are more likely to be human or otherwise legitimate.
For engineers, integrators, and operations teams, understanding this distinction can make troubleshooting more efficient. A bot-check page may reflect a deliberate anti-scraping measure, while an incomplete challenge may point to browser support or JavaScript-blocking settings. Recognizing both possibilities helps teams restore access without mistaking a protective mechanism for a failure in the underlying content or industrial system.
Access Official Technical PDF