The Microsoft Bing search engine has quietly stopped accepting XML Sitemap pings over the holiday break. XML Sitemaps are a structured data format that contains a list of a website’s primary pages. Websites can proactively ping search engines when they publish changes to request that the search engine come and index their new pages.
Sometime between December 14th and 22nd, Bing stopped accepting new ping submissions to its XML Sitemap Ping service. The API endpoint at bing.com/ping now returns an HTTP 410 Gone error message. The error message suggests deliberation and permanence (as opposed to an intermittent problem (HTTP 500) or an HTTP 404 Not Found).
Update (): Bing today published a blog post announcing the deprecation of the bing.com/ping API. The announcement says it’s deprecated “starting today”. However, readers of my blog will know it stopped working five months ago.
Bing’s XML Sitemap documentation for webmasters still suggests pinging Bing using the removed API endpoint. It hasn’t communicated the removal in Bing Webmaster Tools, through their blog, nor on Twitter.
Bing has a second and undocumented API endpoint for sitemaps. The secondary API endpoint is only referenced in a few blog posts from Bing published over a decade ago. The alternate bing.com/webmaster/ping.aspx endpoint still works. However, this API endpoint is only intended for submitting new sitemaps to Bing, and not to notify it about changes to existing sitemaps.
Baidu removed their XML Sitemap Ping API in 2017. The same year, Yandex also removed any mention of their API from their webmaster documentation. Yandex’s API endpoint at webmaster.yandex.com/ping still works, though. Google’s XML Sitemap API at google.com/ping is documented and also continues to work.
XML Sitemap submissions require websites to produce up-to-date and fully qualified sitemaps. The format is well-documented, but it can pose a technical challenge to implement well. An XML Sitemap can be just a list of URLs, and it can help search engines discover new pages.
However, search engines also want to know when a website’s existing pages have changed significantly. This requires websites to include the optional lastmod (last-modified) element with the correct date for each page. Without this information, search engines waste indexing capacity on revisiting pages needlessly.
The creation of a sitemap and the inclusion of the last modification dates can be automated, assuming all the URLs and last-modification time of a website are known. Depending on your publishing system, this information may or may not be available.
Cloudflare research suggests that sitemap files can improve search engine indexing efficiency and lower the carbon footprint of websites. Detailed XML Sitemaps are still valuable and will continue to aid search engines for years to come. However, XML Sitemaps can be huge.
What if you only want to submit one new or modified URL? What if you didn’t have to submit links to each search engine individually? Bing and Yandex decided to work together on a new protocol to improve indexing for everyone.
The IndexNow initiative is a new API from Bing and Yandex that launched in November 2021. IndexNow lets any website submit one or more new or changed URLs to all participating search engines with one API call. Submit your URLs to any IndexNow-compatible search engine (currently only Bing and Yandex), and they have agreed to share the submissions with each other.
IndexNow is a more efficient alternative to creating and maintaining a sitemap file listing every single URL of a website. Bing, Baidu, and Yandex have previously experimented with similar direct-submission APIs. However, these APIs have required websites to invest time and effort to support each search engine. IndexNow makes it easier for websites to adopt as it’s just one standard and one API request to submit to multiple search engines in one go.
IndexNow enables websites to request immediate indexing of new and modified content. It’s up to the search engines to decide whether to act on the requests or not. Neither Bing nor Yandex has published any data on how they evaluate the quality of IndexNow submissions; except to say that websites have a daily submission quota. I’m guessing that they also use a domain quality score to decide whether or not to act on indexing requests.
You’re required to create a key file and host it on the domain you’re submitting URLs from to the IndexNow API. The key lets search engines verify submissions and must be included with every IndexNow API call. You don’t need to register anywhere to get an API key; you generate it yourself using a rather clever scheme.
Google said to Search Engine Land that they’re evaluating IndexNow, but hasn’t committed to supporting it. I’m guessing that they’ll adopt the standard. However, I don’t believe that Google will agree to share indexing submission data with its competitors. Websites might end up having to make two IndexNow API calls: one to Google, and one to any of its competitors.
Bing did not respond to my request for comments. To be fair, it’s the holiday season, so there’s probably no one around. I’ll update this story if I hear back from Bing.