Global Search and Googlebot
When most people think of search, they think of Google. With a fast, intuitive, and highly accurate search engine, Google has become the primary tool for navigating the global web. To deliver such a powerful search experience, Google relies on Googlebot.
Googlebot explores billions of web pages each day, following each link it finds and indexing every page’s content by scanning the HTML and extracting specific elements. While Googlebot is programmed to understand and record standard HTML elements (such as <title>, <h1>, <h2>, <body>, and so on) Googlebot also understands <meta> tag elements. Google created these <meta> tags to let site owners provide additional information about a page that may not otherwise be clear from standard tags. As Googlebot scans a page, both <meta> and standard elements are filed in a consistent structure that is optimized for Google’s search engine algorithm, allowing Google to draw upon a richer body of material when delivering results for a particular query.
Internal Search and Swiftbot
When it comes to internal website search, only Swiftype offers a system of <meta> tags to index pages at a similar speed and depth as Googlebot. Most site search providers require some sort of data feed via .xml, .csv or an API to update results—all of which are cumbersome to set up or difficult to maintain. To solve this problem, Swiftype has worked for over two years to build Swiftbot—an extremely powerful web crawler that passes through individual websites and quickly indexes their content. Swiftbot, like Googlebot, understands standard HTML tags, and can also read custom <meta> tags, which allows site owners to create their own system of <meta> tags to completely reconfigure what information they want visitors to discover through search.
For instance, let’s say an ecommerce site wants to let visitors search and sort products by price, color and average review rating. All they need to do is create a <meta> tag for each of these features in the page header:
<head>
<title>page title | website name</title>
<meta class="swiftype" name="name" data-type="string" content="page title" />
<meta class="swiftype" name="description" data-type="text" content="this is the description" />
<meta class="swiftype" name="price" data-type="float" content="3.99" />
<meta class="swiftype" name="color" data-type="string" content="green" />
<meta class="swiftype" name="review_avg" data-type="integer" content="4.5" />
</head>
With this model, site owners can create as many features as they want Swiftbot to index. Because Swiftbot is programmed to understand and file these <meta> tags when it passes over a page, these elements become central pieces of the site’s data structure and allow site owners to customize search results with faceting and functional boosts.
By offering these custom <meta> tags, Swiftype empowers website owners to tightly control their user search experience and help visitors quickly find the information they need. Website owners can even see all their extracted metadata from the Swiftype dashboard. For more information on how to format and install Swiftype Meta Tags please follow our documentation. Also, check out how you can use <meta> tags to add multiple values for location attributes.
Click here to get started with Swiftype and create a powerful search engine for your site today.