robots.txt, sitemap.xml and llms.txt

These files guide how your site is crawled and protect crawl budget, complementing the performance measured by Core Web Vitals.

Updated

robots.txt

Controls crawling (not indexing). Google Search Central. Google does not support the Crawl-delay field.

User-agent: *
Disallow: /admin/
Disallow: /tmp/

Sitemap: https://www.web-perf.com/sitemap.xml

sitemap.xml and hreflang

List URLs to index with a reliable lastmod (Google ignores priority and changefreq). For a bilingual site, declare language alternates; submit the file to Search Console and Bing Webmaster Tools.

<url>
  <loc>https://www.web-perf.com/</loc>
  <xhtml:link rel="alternate" hreflang="fr" href="https://www.web-perf.com/"/>
  <xhtml:link rel="alternate" hreflang="en" href="https://www.web-perf.com/en/"/>
  <xhtml:link rel="alternate" hreflang="x-default" href="https://www.web-perf.com/en/"/>
  <lastmod>2026-10-26</lastmod>
</url>

llms.txt

A 2024 proposal: a Markdown file (title, summary, links to key pages) aimed at AI assistants. It is not a permissions file and real-world usage remains unconfirmed: treat it as optional. To block AI crawlers, use robots.txt with their user-agents.

# Web-perf.com
> Web performance guides: Core Web Vitals, images, cache, server, loading.

## Guides
- [LCP](https://www.web-perf.com/en/lcp/): Largest Contentful Paint