The Swiftype Blog

On the Search:
St. Mary’s University Implements Swiftype

st-marys-university

On the Search: Elaine Shannon Web Developer & User Interface Specialist

What kind of challenges with search did you face before Swiftype?

One of the biggest challenges we hoped to solve with a new search tool was the ability to search across multiple domains. Over the years we have created many websites, both internal and public-facing, and there was no good way to search across them. It seems like an easy problem to solve, but on our main platform, we didn’t have a tool that could handle results from more than one website.

We also had a hard time making sure search results were relevant. As soon as we installed Swiftype, we started spending time customizing results. Users don’t need to sift through the thousands of pages and documents we’ve indexed across all these sites, they just need the top handful of results.

What were your “must-haves” for search during your evaluation?

The most important factor for us was building a search engine that spanned multiple websites on multiple domains. We also wanted to be able to have better control over search results. We can’t always change our content to match searchers’ intentions — for example, we don’t call our residence halls “dorms” — so being able to set up synonyms and custom result sets at an affordable price point sold us on Swiftype.

In what creative ways have you used Swiftype to improve your search?

Students don’t always know what major they want to pursue, so it’s not always easy for them to browse through a university website and find a program they want to commit to. We thought it might make it easier to decide if we set up a list of interests and linked those to related programs. So, we built a dedicated search tool. If you’re interested in art, you can type that in and be presented with five different program options that match that interest.

Our program page used to be a long list of everything we offered, and we found that site visitors didn’t stay on that page very long. Now, we offer a combination of options — you can search by interest, or you can filter through programs by degree or by school, and we’ve found that 34% of visitors are now using the Search by Interest tool and immediately finding what they’re looking for.

How are you making ongoing adjustments to your relevancy model?

I check on the dashboard weekly, even if it is just a quick glance. If there is anything glaring, like a popular search that’s not returning any results, or a search result set that’s not getting any clicks, we’ll quickly update those results. Otherwise I try to make a focused effort once a quarter to go through our data more deeply and customize our results to provide a better visitor experience.

We also build out content as needed for empty search results. A fun example of this is we didn’t have anything on our mascot for a while, so that was an easy one to add.

What kind of success have you seen since improving your search with Swiftype?

Search is one of the most popular ways to navigate our website. We’ve heard a lot of positive feedback on both our main engine (which searches cross-domain, so you can find the main site, the law school, the bookstore, and the intranet all in one engine) and our Search by Interest engine.

For us, the gold standard of conversion is when a visitor applies for admission, but prospective students are a long sales cycle and visit our sites multiple times before they take that step. We know that registering for a campus visit is a high indicator of a user’s likelihood to apply, and we can track those metrics and how site search has impacted them to date.

Like any tool, there was a bit of a learning curve when we first set up Swiftype. We had to learn which factors impacted the search results most — whether it be a headline, title, or page content. Now that we’ve had more than a year to tweak results, we’ve found a good balance and are serving our visitors more relevant results, which keeps them on the site longer.

Questions about your Swiftype installation? Our Support Team has answers.

Congrats on your Swiftype implementation! You’ve already started delivering a powerful site search experience for your visitors. Trust me, they’ll thank you for it.

We’ve been listening to our community and we’re here to answer 4 of the most commonly asked questions from Swiftype customers.

1. Why do my results all look the same?

When Swiftype’s crawler, lovingly called “Swiftbot”, crawls a domain, there’s potential for repetitive template elements to be indexed with the page body. Most notably the navigation/header, sidebars, and/or footer content. With all the great, meaningful content encased in template noise, the quality of the customer’s search experience can be negatively affected.

For example, Apple.com has these elements on their website:

support_blog_img1

Since these elements appear on all pages, when their site it indexed, in the Swiftype dashboard, it looks like this:

support_blog_img2

You can easily clean this up by using our Content Inclusion/Exclusion tag recognition. By adding Swiftype specific meta tags (data-swiftype-index=’true’) to the HTML container(s) that holds the primary page content, it’s possible to instruct Swiftbot to index only those sections of the page body.

The best practice is to set the main content container to true. If you want to further refine what’s indexed from that section, you can add additional tags with a value of false to containers nested within.

Example

<body>
    <nav>Blah Blah blah</nav>
    <div id="main_content" data-swiftype-index='true'>
         <p>All of my sweet, sweet 5/7 content is going to go in here.</p>
             <div id="ad_widget" data-swiftype-index='false'>This bit really isn't as important which is why I'm going to add a 'false' exclusion parameter.</div>
         <p>This bit will be indexed though, because it’s still within the ‘main_content’ div that’s set to 'true'. Everything outside of the ‘main_content’ div container will be ignored, yo.</p>
             </div>
    <footer>Copyright Attempting to Sound Official© 2016</footer>
          </body>
        

2. How do I prevent Swiftype from indexing certain pages of my site?

For crawler based engines there are three approaches you can take to determine what pages are indexed from your domains: URL path rules, a customized robots.txt file, and robots meta tags.

Path Rules:

From the Manage > Domains section of the Swiftype customer dashboard, there’s an option for each domain that will allow you to ‘Manage Rules’ for that domain. From here, you can define specific paths to include (Whitelist) or exclude (Blacklist) when crawling your site.

Examples of common cases are the exclusion of /category/ paths for ecommerce sites, so the focus is exclusively on crawling product pages. For other CMS based sites, you’d likely see paths to login or administrative pages excluded, as well as dynamically generated content, like pages based on tags or categories. More examples and tips on using this feature can be found here.

Robots.txt Files:

A robots.txt file is a plaintext document that you can upload to the root directory of your website’s domain. With the robots.txt file, you can define URL path exclusion rules for all or only specific web crawlers to follow. Many websites will commonly have a robots.txt file already in place, and it’s presence is one of the first things Swiftbot will look for when starting a crawl process.

Check out our Robots.txt documentation to learn how you can leverage this with Swiftype.

Robots meta tags:

If you need to exclude content in a more precise way (page by page or page template basis), we recommend and fully support robots meta tags. We adhere to the robots tag standard that’s a companion to the aforementioned robots.txt file.

This means that we’ll pass over any page we attempt to crawl that contains a meta tag:

<meta name="robots" content="noindex">

You can also configure these tags so they only apply to Swiftype’s web crawler:

<meta name="st:robots" content="noindex">

Similar to the robots.txt file, meta tags are configured and managed outside of the Swiftype dashboard via their webhost/CMS.

You guessed it. We’ve got documentation of robots meta tags support here.

3. “Why are pages missing from my search engine?”

Here are 3 reasons why Swiftbot, our crawler, may not be able to locate and index pages on your site:

A. We’re unable to find the content because it’s not linked to from other pages.

When spidering a domain, the Crawler will examine all links within a page to discover URLs that are part of the domain submitted to the engine and also adhere to any configured path rules (see question 2 above). If content exists within a site that is not linked to from another known page, within the site’s navigation menu, or listed on a domain’s sitemap, chances are Swiftbot will not be able to locate it.

One of the best ways to ensure our Crawler is able to index all desired content is to include a current sitemap. Sitemaps are files that are typically stored at the root of a URL domain and contain a list of links to pages on their site(s) that are available for crawling. Our documentation for sitemap support and installation notes can be found here.

B. Improperly configured canonical URL elements/tags.

A canonical link element is often used like a meta tag, in order to prevent duplicate content indexing issues by pointing web crawlers to the preferred (or canonical) URL version of a web page. A scenario that can occur is that a customer’s site content is configured with static canonical tags that all point to the root domain URL.

Due to this misconfiguration, even if the crawler finds links to all the content on the site, it is being given instructions that all pages on the site are another version of the home page. With that directive in place, only the home page will be indexed. If the customer re-configures or removes those elements, a recrawl will be able to index their content successfully.

For best practices on canonical elements, you can refer to Google’s documentation here.

C. The content is being excluded by one of the methods noted in question 2.

Just as with misconfigured canonical link elements, misconfiguration or conflicts in a customer’s path or robots rules can cause pages to be skipped over.

4. My site is password protected / behind a firewall / hosted on our company’s intranet.

It is possible for Swiftbot to crawl secured content, but you’ll first need to make minor configuration changes to your web or intranet site’s host server.

All Swiftype accounts have an account specific User-Agent ID string. By whitelisting this identification string with your server, you can allow, or disallow, crawlers access to your site’s content.

Swiftype has a unique security feature where we encode our crawler’s User-Agent with a secure key that is uniquely tied to your Swiftype account. This approach enables you to limit access solely to Swiftype’s crawler, and is an extra level of security many customers enjoy.

If you’re interested in using the Swiftbot web crawler to access your secured content, please contact our support team and we’ll be happy to supply you with your account specific User-Agent ID string.

Hopefully these answers to commonly asked questions will point you in the right direction. If you ever have questions, suggestions, or feedback, you can always email support@swiftype.com to reach our team. We’re happy to help!

On the Search:
Roanoke College Implements Swiftype

roanoke-college

On the Search: Michael Santoroski, Director of Web and Software Development

Roanoke College was ranked 2nd on the 2014 U.S. News and World Report list of Up-and-Coming National Liberal Arts Colleges. As the Director of Web and Software Development, Michael Santoroski was tasked with improving the college’s site functionality, after a site redesign. The initial search solution was not returning the desired results and with so much on the line after a complete site overhaul, search was critical to help visitors find their way around a newly designed website.

Today we discussed the ease of implementing Swiftype and the biggest impact it has had on both their internal and public-facing sites.

What kind of challenges with search did you face before Swiftype?

Roanoke is a small college with about 2,000 students.  Like many colleges, we don’t have the resources to have the latest technology, but we needed to improve our on-site search experience because our site serves such a wide range of audiences. From prospective students, current students, faculty, and alumni – search is often the fastest way for them to find answers to their questions.

We were using Google’s Custom Search Engine, but it was no longer meeting our needs. The tool wasn’t flexible enough for our site and wasn’t producing the results we expected.

What were your “must-haves” for search during your evaluation?

Two things were really critical for us. Weighted search and being able to rank search results was key for us. Secondly, the type-ahead feature was really important to me. It just made searching so much better when Google first implemented it for the web.

With a limited IT team, I was also looking for a solution that would be easy to implement. I like that it’s a crawler-based solution. I just had to add our domain and set up probably took less than an hour to get it to a functioning state. Swiftype has really made installation as unobtrusive as possible.

What has surprised you the most with your new deployment?

The number of emails to our web support has been reduced to almost none. We were constantly getting emails through our contact page from users who couldn’t find content they were looking for, whether that be an upcoming course schedule or an event. As soon as we implemented Swiftype, the emails stopped…virtually overnight.

What new things did you find readers searching?

We had one instance where a faculty member told me she couldn’t find the academic catalog. I was in a meeting, so I went in and switched the search results around a little bit, but then she emailed me two days later and said “Oh, I found my problem, I was spelling it like British catalogue – with “gue”. So I just added that as a synonym and it’s fixed either way now.

That kind of on the fly customization really helps us keep our site search results relevant.

What kind of success have you seen since launching the improved search with Swiftype?

For higher education, it’s a little harder to quantify our success metrics in relation to site-search. It is all about ease of use. Swiftype does have an engagement module for that, but I haven’t really gotten to play with those. What we end up seeing in the admissions process is that a student comes back to the site over multiple sessions, and when they come to the site, each time they have a different objective.

On the flip side, you also have alumni who receive emails, come to the site looking for alumni events and rely on search. Ultimately those are visitors you want making donations to the college, so each type of visitor has different success metrics.

Higher Ed websites are difficult. The administration wants all kinds of content on the site, but students want something else, so there is a balance in terms of what information to show.

I think in a way, search has solved that problem. It’s one of those things that works great and I don’t worry about it too much. It just does exactly what I need it to do and doesn’t make me spend a lot of time fighting with it. When I’m in a meeting and someone says “I tried to search for this thing and I didn’t find it”, I can just go in there and easily make it happen. Make that be the top search result and it’s super easy to do, even I can’t screw it up.

Swiftype Partner Spotlight:
The Web Development Group

wdg_logo_black

Swiftype is not just a solution for end businesses, but also a partner to many other companies who navigate site search solutions on behalf of their clients. Today we’re introducing The Web Development Group (WDG™), a full-service digital agency based out of Washington D.C., who has been able to leverage Swiftype technology to effectively and efficiently deploy search solutions for their clients. I had the privilege of sitting down with Ab Emam, their Agency Director to discuss the challenges they face as a business and how they’ve been able to make great strides for their clients when it comes to optimizing on-site search.

The Web Development Group

The Web Development Group works in a wide range of verticals, from associations, foundations, and nonprofits to government entities, higher education, and Fortune 1000 businesses.  When clients sign on with WDG for a website redesign, it becomes critical to determine a robust site search plan at the beginning of a project due to the amount of content the typical WDG client normally houses on their site.

The Web Development Group offers both Web Design & Development services as well as Digital Marketing Consulting.  Partnering with Swiftype has allowed them to offer a robust site search solution that is not only easy for their developers to implement but has SEO and usability benefits as well. Initially, there was concern that developers would prefer to develop their own site search solution, but WDG has learned developers have become some of the software’s biggest fans for its ease of installation. WDG has since dedicated several developers who are completely knowledgeable with the implementation and functionality of Swiftype’s solution.

When a company is considering a site redesign it becomes an opportune time to evaluate their existing site search functionality. Many site platforms have built-in solutions or basic add-ons like Google Custom Search. But with such limited customizations available, WDG has chosen to work strategically with Swiftype on behalf of their clients to introduce the advantages a customizable solution offers within the initial exploratory phase of a project. In turn, Swiftype is able to provide comprehensive support to answer technical questions and help WDG implement successful site search functionality.

Customer Success

The City of Alexandria recently relied on the expertise of WDG for a site redesign, which included a new site search solution. As a government entity, the City’s site is home to a significant amount of content that needed to be easily accessible to end-users. While navigation elements remain a prominent feature of the site, an easy-to-use and efficient search feature greatly improved the new site’s usability. The site features a prominent search bar within its redesign as the site needed to deliver content efficiently to a wide range of users – tourists, residents, and local businesses. All these audiences have unique needs and by implementing a prominent search bar as a way of navigating through the site, AlexandriaVA.Gov is able to quickly serve up relevant content.

WTOP.com has also worked with The Web Development Group for a site redesign and, as one of Washington DC’s top news sources, a robust site search was critical for them due to the high volume of content published on a daily basis. Swiftype’s solution is a seamless integration for publisher sites where website managers can adjust search results to feature the newest or breaking content, as well as prioritize sections of their site based on selected keywords. The Swiftype product dashboard also provides publishers with insights into the keywords their audience is looking for which can help to shape news stories that haven’t even been written yet.

These partnerships are just that – true partnerships that allow for clients to benefit from the mutual efforts of expert digital teams. By leveraging the talents from both Swiftype and The Web Development Group, our clients stand to benefit from robust search capabilities with the potential to see exponential returns.