Thin pages, sitemaps and AI crawlers on a university directory

A new directory never has the same number of entries everywhere. Universities DB lists 198 universities in 65 countries, and 50 of those countries have three each. The United States has eight, the United Kingdom and India have five each, and eight countries have fewer than three. Kenya and Luxembourg have one each, and Angola, Kuwait, Mongolia, Myanmar, Panama and South Korea have two.

Those eight are where a directory is most likely to get into trouble with search engines. A country page with one listing on it is close to empty, and a site with many pages like that gives search engines a lot of low-value URLs to crawl and judge. Here is how Universities DB deals with them, along with a few other decisions about crawling that we checked in October 2026.

Noindex until a page has enough on it

The eight thin country pages carry a robots meta tag set to "noindex, follow". Search engines are asked to leave the page out of results but still follow its links to the universities listed on it, so the university pages themselves are still found and indexed. City pages appear to follow the same rule. The Bangkok page, with two universities, is set to noindex, while the Singapore city page, with three, is indexed.

The thin pages are not left blank for visitors either. The Angola page, for example, has a box headed "We're still adding universities in Angola", a link for universities to add themselves for free, and links to nearby countries in Africa (Egypt, Ethiopia, Ghana, Morocco, Nigeria and South Africa, three universities each). A student who lands there from a link has somewhere to go next.

If the rule is tied to the listing count in the database, nobody has to remember to change it. When a third university is added to Kenya, the page becomes indexable without anyone editing it by hand. That is how we would set it up for a client.

A sitemap split by page type

The sitemap at /sitemap.xml is an index pointing to four files: one for general pages (7 URLs), one for countries and cities (70), one for universities (198) and one for articles (4). Each entry in the index has a lastmod date.

The thin pages are missing from the locations file. Angola is not in it, and of the 65 countries, 57 are listed, which is the number you get once the eight noindexed countries are taken out. A sitemap should only list pages the owner wants indexed. Putting a noindexed page in the sitemap sends search engines two opposite instructions about the same URL.

Splitting the sitemap by type also helps with diagnosis later. Google Search Console reports indexing per sitemap file, so if university pages stop being indexed, the owner sees it in the universities file without wading through countries and articles.

AI crawlers allowed by name

The robots.txt file has the usual block for all user agents, which allows the site and keeps crawlers out of the admin area, the claim pages, the API and the upgrade pages. Below that is a second block that names eleven AI crawlers one by one: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and CCBot. They get the same rules.

Naming them changes nothing technically, since the general block already lets them in. It does make the owner's choice visible to anyone who reads the file, and it means a later change to the general rules will not shut out AI crawlers by accident. Some of these agents fetch pages when a user asks a question in a chatbot, some build search indexes for AI answers, and some, such as Google-Extended and Applebot-Extended, control whether content can be used for model training. A directory owner should decide on each group separately before copying a robots.txt from somewhere else.

The site also publishes /llms.txt, with links to every country and university page, and /llms-full.txt, which runs to about 175 KB and gives each of the 198 universities a short block of plain text: type, city, founding year, teaching language, degree levels, programmes, official website, whether the university has verified the listing, and the date it was last updated.

Saying where the data comes from

The methodology page explains the sources in order. Official university websites come first for contact details, programmes, tuition and deadlines. Founding dates and locations are checked against open databases such as Wikidata. Rankings are quoted from the publisher with the year. AI tools are used to find and structure public information, and the page says that their output is treated as a draft and checked against the university's own website, with unconfirmed data left blank.

The same page says that basic listings are free, that paid placements are labelled and do not change the factual data, and that the site does not sell rankings. A directory that uses AI in its research and says so, with the checking steps written down, gives journalists and students something to judge it by.

Where the enquiries go

Every listing ends with a "Request information" form. Past the name and email, it asks what the enquiry is about (admissions, entry requirements, fees, scholarships and so on), the level of study, the subject, the preferred intake term and year, a budget band in US dollars, and the student's country of residence and nationality. On the Chulalongkorn page a checkbox offers to send the same enquiry to Thammasat and Mahidol. The form states that enquiries are kept for up to 24 months and deleted on request.

For the university receiving it, an enquiry that already says "Master's, business, August 2027 intake, budget USD 5,000 to 15,000" can go straight to the right office.