Skip to Content
Last modified by Eleni Cojocariu on 2026/10/21 14:26
Show last authors
1 A wiki open on the Internet is visited by search robots that follow every URL they find, including the ones that export a page, run a query or open an editor, and that costs the instance far more than serving a page: see [[Performance>>doc:documentation.xs.admin.performance.WebHome]]. A ##robots.txt## keeps them to what is worth indexing.
2
3 1. Write the rules below into a ##robots.txt##. The patterns use ##*## rather than ##/bin/## so that they also match a path-based subwiki such as ##/xwiki/wiki/myalias/view/Space/Page##.(((
4 {{code language="none"}}
5 User-agent: *
6 # Prevent bots from executing all actions except "view" since:
7 # 1) we don't want bots to execute stuff in the wiki!
8 # 2) we don't want bots to consume CPU and memory
9 # (for example to perform exports)
10 # Attached images are allowed below, before /download/ is disallowed.
11 Allow: /xwiki/*/download/*.png$
12 Allow: /xwiki/*/download/*.jpg$
13 Allow: /xwiki/*/download/*.jpeg$
14 Allow: /xwiki/*/download/*.gif$
15 Disallow: /xwiki/*/viewattachrev/
16 Disallow: /xwiki/*/viewrev/
17 Disallow: /xwiki/*/pdf/
18 Disallow: /xwiki/*/tex/
19 Disallow: /xwiki/*/edit/
20 Disallow: /xwiki/*/create/
21 Disallow: /xwiki/*/inline/
22 Disallow: /xwiki/*/preview/
23 Disallow: /xwiki/*/save/
24 Disallow: /xwiki/*/saveandcontinue/
25 Disallow: /xwiki/*/rollback/
26 Disallow: /xwiki/*/deleteversions/
27 Disallow: /xwiki/*/cancel/
28 Disallow: /xwiki/*/delete/
29 Disallow: /xwiki/*/deletespace/
30 Disallow: /xwiki/*/undelete/
31 Disallow: /xwiki/*/reset/
32 Disallow: /xwiki/*/register/
33 Disallow: /xwiki/*/propupdate/
34 Disallow: /xwiki/*/propadd/
35 Disallow: /xwiki/*/propdisable/
36 Disallow: /xwiki/*/propenable/
37 Disallow: /xwiki/*/propdelete/
38 Disallow: /xwiki/*/objectadd/
39 Disallow: /xwiki/*/commentadd/
40 Disallow: /xwiki/*/commentsave/
41 Disallow: /xwiki/*/objectsync/
42 Disallow: /xwiki/*/objectremove/
43 Disallow: /xwiki/*/attach/
44 Disallow: /xwiki/*/upload/
45 Disallow: /xwiki/*/download/
46 Disallow: /xwiki/*/temp/
47 Disallow: /xwiki/*/downloadrev/
48 Disallow: /xwiki/*/dot/
49 Disallow: /xwiki/*/svg/
50 Disallow: /xwiki/*/delattachment/
51 Disallow: /xwiki/*/login/
52 Disallow: /xwiki/*/loginsubmit/
53 Disallow: /xwiki/*/loginerror/
54 Disallow: /xwiki/*/logout/
55 Disallow: /xwiki/*/charting/
56 Disallow: /xwiki/*/lock/
57 Disallow: /xwiki/*/redirect/
58 Disallow: /xwiki/*/admin/
59 Disallow: /xwiki/*/export/
60 Disallow: /xwiki/*/import/
61 Disallow: /xwiki/*/get/
62 Disallow: /xwiki/*/distribution/
63 Disallow: /xwiki/*/imagecaptcha/
64 Disallow: /xwiki/*/unknown/
65 # Don't index sample content.
66 Disallow: /xwiki/*/view/Sandbox/
67 # The Admin space is protected by rights anyway; this is a safety net.
68 Disallow: /xwiki/*/view/Admin/
69 # Statistics pages are not useful and are CPU intensive.
70 Disallow: /xwiki/*/view/Stats/
71 # Panels are UI, not content.
72 Disallow: /xwiki/*/view/Panels/
73 # The search page.
74 Disallow: /xwiki/*/Main/Search
75 # The REST API.
76 Disallow: /xwiki/rest/
77 # UI elements that send crawlers into infinite loops (XWIKI-16915).
78 Disallow: /xwiki/*?*xpage=*
79 {{/code}}
80 )))
81 1. Serve the file from the **root of the host**, which is the only place a crawler reads it: the document root of the web server in front of XWiki, or the ##ROOT## web application of the Servlet Container when it answers directly. A ##robots.txt## inside ##/xwiki/## is never read.
82 1. Give each host its own file. A wiki farm answering on several domains, and a subwiki on a domain of its own, each serve their own ##robots.txt##.
83 1. Request the file and check that the rules are the ones the crawler will see.(((
84 {{code language="bash"}}
85 curl https://wiki.example.com/robots.txt
86 {{/code}}
87 )))

About

About

Support

Platform

User Guide

Admin Guide

Developer Guide

Projects

XWiki

Extensions

Other

Contribute

Status

Practices

Under the Hood

Get Involved

Get Connected