Content & Access - JustCharity

For developers and hostmasters

JustCharity data is collected from public APIs and feeds where available, and in some cases additional meta data may be collected via web applications directly, in these cases the following user-agent details apply:

The JustCharity application user-agent is in the following format, where [SHA] is replaced with a release version.


        JustCharity/rev[SHA] (+http://justcharity.org)

Example


        JustCharity/rev61178ed36ce1e41e71d3634b57387a4f5880aa9f (+http://justcharity.org)

Robots.txt

JustCharity collection conforms to applied robots.txt, this is a manually reviewed process and robots.txt changes may take 7-14 days to propagate. If the JustCharity bot does not seem to be confirming to defined rules, please Contact Us with access logs and we can take a look.

The Crawl Delay directive is a useful rule to apply if you are seeing higher than expected load, if this is undefined this will be automatically set at between 5-60 seconds.

Example


      # Applies to all bots/spiders (e.g. Google, Yahoo, Baidu, etc), sets a 60 second delay

      User-agent: *

      Crawl-delay: 60



      # Applies to JustCharity only

      User-agent: JustCharity/rev* (+http://justcharity.org)

      Crawl-delay: 60