Multilingual applications: how to manage sitemaps and meta-tags for better indexing

If you're using Rails then managing the SEO component of a web application is as simple as it is important to implement. You can easily find guides on how to do so online.

Once you've done that, sitemaps can be generated at virtually no cost by using gems such as sitemap_generator.

Difficulties emerge, however, as soon as you try start managing multilingual applications. As such applications may, for example, contain sections that are only available in a subset of the languages being used.

Gems such as route_translator — which extends Rails DSL to the level of routing using a particular localized directive — come to our aid to facilitate the process of internationalizating URLs:

# config/routes.rb
localized do
  get "/about" to: 'static#about', as: :about
  resources :posts, only: [:show]
end

The route translations are specified in local files:

# config/locales/fr.yml
fr:
  routes:
    posts: nouvelles
    about: a-propos-de-nous

# config/locales/en.yml
en:
  routes:
    posts: posts
    about: about

# config/locales/it.yml
it:
  routes:
    posts: articoli
    about: chi-siamo

For every route indicated within the block, a route helper specified for each language will be generated, as will a helper capable of dynamically selecting the most correct route for the active language:

about_it_path    # => /chi-siamo
about_en_path    # => /en/about
about_fr_path    # => /fr/a-propos-de-nous
about_path       # => dipendente dalla lingua corrente

These helpers make it very simple to generate a language switch:

/ app/views/static/about.html.slim
- content_for(:switch_locale) do
  ul.switch_locale
    - I18n.available_locales.each do |locale|
      li= I18n.with_locale(locale) do
        = link_to "Switch to #{locale}", about_url

Now, let's imagine a site with twenty routes to be translated. Will we have to replicate a similar snipped for each view? That would hardly be a very DRY approach. Why not just use the url_for method?

/ app/views/layout/application.html.slim
ul.switch_locale
  - I18n.available_locales.each do |locale|
    li= I18n.with_locale(locale) do
      = link_to "Switch to #{locale}", url_for(locale: locale)

This trick works so long as we're working with models that don't also require the slugs to be translated (for example, by copying globalize + friendly_id):

/ /app/views/posts/show.html.slim

= I18n.with_locale(:it) { post.slug } # => "il-mio-primo-post"
= I18n.with_locale(:en) { post.slug } # => "my-first-post"
= I18n.with_locale(:fr) { post.slug } # => "mon-premier-nouvell"

= params.inspect
/ => {"controller" => "posts", "action" => "show", "id" => "my-first-post"}

- %i(it en fr).each do |locale|
  = I18n.with_locale(locale) { url_for(locale: locale) }

/ => /articoli/my-first-post
/ => /en/posts/my-first-post
/ => /fr/nouvelles/my-first-post

What's happened here? Well, the url_for method constructs the URL based on the keys (in our case, local) that it has received and the keys of the global hash params — (in our case, action and id).

Unfortunately for us, the id key contains the value of the slug relative to the current page ("my-first-post"). The post_path route generated by route_translator is able to translate the part of the URL related to controller and action, but it can't do much with the id parameter that it receives as input, and that isn't recognizable as the Post model that it originates from.

We can get around this inconvenience by making the local switcher slightly more configurable:

/ app/views/layout/application.html.slim
ul.switch_locale
  - I18n.available_locales.each do |locale|
    li= I18n.with_locale(locale) do
      - url = yield(:current_page_url) || url_for(locale: locale)
      = link_to "Switch to #{locale}", url

/ app/views/posts/show.html.slim
- content_for(:current_page_url) { post_url(@post) }

Doing it this way we are free, in the case of the routes with translated slugs, to specify a particular method to use for URL generation:

<li><a href="/articoli/il-mio-primo-post">Switch to it</a></li>
<li><a href="/en/posts/my-first-post">Switch to en</a></li>
<li><a href="/fr/nouvelles/mon-premier-nouvell">Switch to fr</a></li>

Once the generation of the translate URLs has been set up, it's also easy to generate alternative links inside our <head> by using the same logic:

/ app/views/layouts/application.html.slim

- I18n.available_locales.each do |locale|
  - I18n.with_locale(locale) do
    - url = yield(:current_page_url) || url_for(locale: locale)

    - if locale == I18n.default_locale
      link rel="alternate" hreflang="x-default" href=url

    link rel="alternate" hreflang=locale href=url

We now have all of the information necessary to generate exhaustive sitemaps.

Let's hypothesize a scenario in which we have a site with two different domains: myblog.it and myblog.com. The first domain is written exclusively in Italian, while the second contains content translated into English and French.

My searching through blogs to find solutions for generating sitemaps hasn't yielded satisfactory results: most of the articles are limited to copying what has already been written based on Google's guidelines or the SeoMoz cheatsheet.

Following on from numerous discussions with our resident SEO expert, the solution we have ultimately adopted has been to generate two distinct sitemaps — one for each domain — specifying the alternative URLs within the XML <url> tag. These results can be reproduced using these few lines of code:

# app/controller/sitemap_controller.rb
class SitemapController < ApplicationController
  def index
    @available_locales = @domain == "myblog.it" ? [:it] : [:en, :fr]
  end
end

# app/views/sitemap/index.xml.builder
xml.urlset( "xmlns" => "http://www.sitemaps.org/schemas/sitemap/0.9", "xmlns:xhtml" => "http://www.w3.org/1999/xhtml") do
  @available_domain_locales.each do |locale|
    I18n.with_locale(locale) do
      xml.url
      xml.loc posts_url
      xml.priority "1.0"
      xml.changefreq "monthly"
      xml.lastmod "2015-01-01"
      I18n.available_locales.each do |other_locale|
        I18n.with_locale(other_locale) do
          xml.tag! "xhtml:link", rel: 'alternate', hreflang: other_locale.to_s, href: posts_url
        end
      end
    end
  end
end

The result is the following:

# http://myblog.it/sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">
  <url>
    <loc>http://myblog.it/articoli/il-mio-primo-post</loc>
    <priority>1.0</priority>
    <lastmod>2015-01-01</lastmod>
    <changefreq>monthly</changefreq>
    <xhtml:link rel="alternate" hreflang="it" href="http://myblog.it/articoli/il-mio-primo-post" />
    <xhtml:link rel="alternate" hreflang="en" href="http://myblog.com/posts/my-first-post" />
    <xhtml:link rel="alternate" hreflang="fr" href="http://myblog.com/fr/nouvelles/mon-premier-nouvell" />
  </url>
  <url>
    <loc>http://myblog.it/chi-siamo</loc>
    <priority>1.0</priority>
    <lastmod>2015-01-01</lastmod>
    <changefreq>monthly</changefreq>
    <xhtml:link rel="alternate" hreflang="it" href="http://myblog.it/chi-siamo" />
    <xhtml:link rel="alternate" hreflang="en" href="http://myblog.com/about-us" />
    <xhtml:link rel="alternate" hreflang="fr" href="http://myblog.com/fr/a-propos-de-nous" />
  </url>
</urlset>

# http://myblog.com/sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">
  <url>
    <loc>http://myblog.it/posts/my-first-post</loc>
    <priority>1.0</priority>
    <lastmod>2015-01-01</lastmod>
    <changefreq>monthly</changefreq>
    <xhtml:link rel="alternate" hreflang="it" href="http://myblog.it/articoli/il-mio-primo-post" />
    <xhtml:link rel="alternate" hreflang="en" href="http://myblog.com/posts/my-first-post" />
    <xhtml:link rel="alternate" hreflang="fr" href="http://myblog.com/fr/nouvelles/mon-premier-nouvell" />
  </url>
  <url>
    <loc>http://myblog.it/fr/nouvelles/mon-premier-nouvell</loc>
    <priority>1.0</priority>
    <lastmod>2015-01-01</lastmod>
    <changefreq>monthly</changefreq>
    <xhtml:link rel="alternate" hreflang="it" href="http://myblog.it/articoli/il-mio-primo-post" />
    <xhtml:link rel="alternate" hreflang="en" href="http://myblog.com/posts/my-first-post" />
    <xhtml:link rel="alternate" hreflang="fr" href="http://myblog.com/fr/nouvelles/mon-premier-nouvell" />
  </url>
  <url>
    <loc>http://myblog.com/about-us</loc>
    <priority>1.0</priority>
    <lastmod>2015-01-01</lastmod>
    <changefreq>monthly</changefreq>
    <xhtml:link rel="alternate" hreflang="it" href="http://myblog.it/chi-siamo" />
    <xhtml:link rel="alternate" hreflang="en" href="http://myblog.com/about-us" />
    <xhtml:link rel="alternate" hreflang="fr" href="http://myblog.com/fr/a-propos-de-nous" />
  </url>
  <url>
    <loc>http://myblog.com/fr/a-propos-de-nous</loc>
    <priority>1.0</priority>
    <lastmod>2015-01-01</lastmod>
    <changefreq>monthly</changefreq>
    <xhtml:link rel="alternate" hreflang="it" href="http://myblog.it/chi-siamo" />
    <xhtml:link rel="alternate" hreflang="en" href="http://myblog.com/about-us" />
    <xhtml:link rel="alternate" hreflang="fr" href="http://myblog.com/fr/a-propos-de-nous" />
  </url>
</urlset>

By using this mechanism, each domain provides a complete list of the URLs that it manages and, for each URL provided, a full list of any cross-domain alternatives, in such a way that search engines will be able to get a full mapping of the URLs instantly.