Why use the Swiftype Web Crawler?
Easily index your website content for search
The Swiftype web crawler, Swiftbot, is a quick and easy way to index your website’s content for search. While you can also index your content using the Swiftype API, utilizing the web crawler requires less technical know-how and generally enables you to implement your site search faster. If you’re trying to decide whether to use the crawler or the API, check out this blog post written by Swiftype cofounder Quin Hoxie.
Swiftype’s crawler gathers and indexes data from your specific website in a similar manner to the way Google crawls data for its internet-wide search engine. To get started with the Swiftype crawler, you simply input your website’s URL and Swiftype will begin indexing your content. Once your website has been crawled, you can use the Swiftype dashboard to review your indexed content, customize your search results, and view analytics.
Getting started with the Swiftype Web Crawler
Here’s a quick overview of how to get started with the Swiftype web crawler. For additional information, check out our Crawler Quick Start Documentation.
Create your search engine
To use the Swiftype crawler, you will first need to create a new search engine in your Swiftype dashboard and then enter your site’s URL. Still need to create a Swiftype account? You can sign up here.
Preview your search engine before implementation
After your URL has been verified, Swiftype will begin indexing your website’s content. If you have a sitemap, Swiftype will use it as a starting point for crawling your site. When all of your content has been indexed and your search engine is ready, Swiftype will notify you with an email. If you would like to test out your search engine before implementing it on your site, you can preview your search experience directly from the Swiftype dashboard.
Installing a search box on your website
When you’re ready to install a search bar on your website, you can copy and paste the code given to you in the Swiftype dashboard.
Customizing the look and feel of your search
Since it’s likely that you’ll want your search UI to match the design of the rest of your website, Swiftype makes it easy to customize the appearance of your search bar and results container. To customize your search installation, simply click the “Edit Install Setup” or “Edit Appearance” cards under the “Customize Your Installation” portion of the Swiftype onboarding flow.
As you experiment with the look and feel of your search, you can preview your changes in the Swiftype dashboard (before pushing your changes live). For more information on configuring the look and feel of your search engine, check out this video tutorial.
Indexing new website content
After Swiftype has indexed your content and you’ve implemented your search engine on your website, Swiftype will continue reindexing your content according to your plan level. If you make changes to your website and would like to manually trigger a recrawl, you can do so from the Swiftype dashboard.
Customizing your crawl
Provide direction to the Swiftype web crawler
By default, Swiftype will index your entire website (up to page limit of your plan). In order to optimize the search experience on your website, it’s likely that you will need to customize how the Swiftype crawler indexes the content on your site. For example, you may want to exclude certain pages or portions of pages on your website from being indexed by your search engine.
How to customize your crawl
When it comes to customizing how Swiftype indexes your website content, you have a few options.
Excluding/Including portions of your site:
- Domain path exclusions – Use the Swiftype dashboard to whitelist and/or blacklist portions of your website
- Robots.txt rules – Swiftype will adhere to the rules set in your robots.txt file
- Robots meta tag – Swiftype will adhere to the rules set in your robots meta tag
Excluding/including portions of a page (such as header and footer):
- Content inclusion/exclusion – Use a special html tag to exclude/include portions of your web pages
Use meta tags to explicitly set what data is extracted from a page (override default crawler settings):
- Meta tags – Deliver structured information to the Swiftype web crawler and determine what content on your pages will be made searchable
- Body-embedded data attribute tags – If you’d prefer to set what data will be indexed inside the body of your pages (rather than using meta tags), you can add special html tags to your existing source code
