The Swiftype Blog

Building an Asynchronous API to Improve Performance

One of the challenges we’ve had to deal with at Swiftype is that we have had customers pushing a lot of search and indexing traffic from very early on. When a customer is pushing hundreds of index updates per second, it’s important to respond quickly so we don’t start dropping requests.

In order to do that, we’ve built bulk create/update API endpoints to reduce the number of HTTP requests required to index a batch of documents and moved most processing out of the web request. We’ve also invested in front-end routing technology to limit the impact customers have on each other.

However, we were not satisfied. Sometimes when a large customer was indexing a huge number of documents, our front-end queues would still back up.  In the pursuit of even better response times for our customers, we’ve built an asynchronous indexing API. Our goals in creating the new API were high throughput, supporting bulk requests for all interactions, and excellent developer ergonomics. We wanted an API that was fast and easy to use.

Here’s how it works.

async_bulk_API_vertical_2.28.39_PM

First, customers submit a batch of documents to create or update. The request for this looks just like our pre-existing bulk create or update API, but goes to a new endpoint.

When our server receives the response, it performs a quick sanity check on the input, without hitting the database. If all the input parameters are present and validly formatted, we create two records in our database for each document that was submitted: a document creation journal, and a document receipt.

For performance, we insert these rows using activerecord-import. This is a great library that uses a single INSERT statement with multiple rows. This results in a massive speed improvement compared to standard ActiveRecord when saving a large number of records. We also generate the IDs ahead of time using BSON. By generating the IDs ahead of time, we don’t need to get them from the database after inserting, and using BSON lets us encode a timestamp in the ID at the cost of a larger ID column.

Once created, we enqueue a message for each document creation journal onto a queue that is read by a pool of loops workers. Loops is a dead-simple background processing library written by our Technical Operations Lead, Oleksiy Kovyrin. It makes it easy to write code that does one thing forever, in this case, reading messages off the queue and creating the associated document in the database.

The response to the API request includes a way to check the status of all the document receipts. To make the API easy to use, we’re including URLs to the created resources. Though we’re not following all its precepts, this approach is inspired by the idea of the hypermedia API. These URLs make it easy for both humans and computers to find the resource.

Since the API is asynchronous, users must poll the document receipts API to check for the status of the document creation. We’ve built an abstraction in our Ruby client library that allows developers to simulate a synchronous request, although we recommend that only for development.

By pushing all work except for JSON parsing and the most minimal input validation to the backend, we’re able to respond to these API requests very quickly. On the backend, the loops workers read messages off the queue and create documents. When a loops worker attempts to create a document, it updates the document receipt (either with the status of “complete” and a link to the created/updated document, or with the status “failed” and a list of error messages) and deletes the document creation journal.

This brings us to one final aspect of the asynchronous API: how we make sure it keeps working. If our loops workers started failing, the document creation journals would back up without being processed, and no documents would be created/updated. To guard against this, we have built a monitoring framework that alerts us when the oldest record in the table is older than a certain threshold.

This solution has been successful for us in beta tests with our largest API users, and we have now rolled it out to everyone.

We hope this helps you build out your next high-throughput API. If this is the kind of thing you’re interested in, we’re hiring engineers for our core product and infrastructure teams.

Swiftype Announces $13 million Series B Funding Round

Today we’re excited to announce that we’ve raised a $13 million Series B financing led by our investors at NEA. We’ve actually worked with NEA since our company was only a few weeks old — first as a part of the Seed round we raised during Y Combinator and again when they led our Series A in September of 2013 — and we’re delighted to further our partnership with them today. We’re also excited to officially welcome NEA Partner Chetan Puttagunta to the Swiftype board.

Swiftype-Office

Swiftype has come a long way since graduating from Y Combinator in March of 2012. Quin and I started the company alone in an apartment in Mountain View, CA, adding Luke Francl and Oleksiy Kovyrin as quickly as we could convince them, and then the team stayed that size for a while. But today we’re a team of over 25 incredibly talented individuals, and our business has grown similarly. Today we power billions of queries every month across hundreds of thousands of websites, mobile applications, and SaaS applications. Along the way we’ve had the privilege of working with many fantastic companies, including Qualcomm, CloudFlare, Shopify, and Twitch. It has taken a tremendous amount of dedication — and a lot of hard work — from our entire team to get to where we are today, and I couldn’t be more proud of the progress we’ve made so far.

Looking ahead, these new funds will fuel our continued growth as we look to bring on more search engineers, expand our sales and marketing efforts, and establish a stronger position of leadership within the search industry.

If you’re interested in joining a quickly growing, fast-paced organization, we encourage you to check out our jobs page. We’re always excited to meet new people who are interested in joining the Swiftype team, and we’d love to hear how you can contribute.

Swiftype for Zendesk

Zendesk_logo_RGB

As we’ve written before, search is a critical tool for self-service support centers. Users who enter the documentation or support section of a website generally have a clearly defined issue in mind, and a powerful search bar provides these users with a clear path to the content they are looking for.

To help more support centers bring great search to their users, Swiftype is excited to announce our integration with Zendesk Help Centers, making it easy for knowledge bases built with Zendesk to deliver the search experience that customers expect. With Swiftype powering search on your Zendesk Help Center, site owners have the power to customize results for individual queries, ensuring that users always find the most helpful content. Furthermore, Swiftype provides detailed analytics on what your users are searching for, providing key insight into user issues and identifying content gaps by populating a list of top queries that return no results.

With these tools and more, site owners can redefine their Help Center user experience by providing powerful and customizable search without the need to consult engineering. Get started by creating account today.

New feature: Live Index Preview tool

Today we are excited to announce a new dashboard feature for site owners with crawler based engines: the Live Index Preview tool. Some of you may have noticed this feature since we rolled it out a month ago, but we announce it today in conjunction with our new meta tags tutorial, which walks site owners through the steps required to implement Swiftype meta tags on your website template.

Now when you log into your Swiftype account and visit the Content tab, you have the ability to compare the live state of your site’s content with what Swiftype presently has indexed for your search engine. To do this, click on the Details button for a page document from your main Content tab view, or look up a specific URL from the search box.

On the Details page, switch from the indexed page overview to the live view by clicking on the bolt icon. Once you switch to live view, you can see what Swiftype currently has indexed in the left column, side by side with your live site offerings in the right column.

Compare your search engine index directly with your live site content with the Live Index Preview tool.

Why is this tool useful? This tool allows site owners to make changes to their site (modifying content or adding custom meta tags) and immediately see how their index will change during the next recrawl. Furthermore, from the Options drop down it’s possible to trigger instant recrawls of specific pages.

The Live Index Preview enables a much greater degree of flexibility when making changes to your website, allowing you to immediately see the impact of those changes in your search engine index without having to wait for a Swiftype recrawl to take place. To view our new meta tags tutorial, click on the link below.

Improve your site search today with these quick tips

On the modern web, search bars are everywhere. Google, the modern address bar doubling as a search bar, and the prevalence of search tools throughout the web have trained users to expect a search option when they have a clear idea of what they are looking for. This clear idea—intent—makes visitors who use search one of the most valuable online audiences.

The power of Google’s web search and of major internet players’ such as Amazon’s site search often lead to an assumption that all search solutions offer the same experience that your visitors expect. However, if you’ve used many search functions, you know this is not true. While a comprehensive, simple solution to robust site search is the best way to ensure your site doesn’t disappoint your visitors, we’ve put together a list of steps that site owners can take to improve their search experience right away.

  1. Use your search analytics. A search bar is an extremely unfiltered form of customer feedback—a free text box that asks the searcher to type in exactly what they want. This unfiltered feedback should be valued for what it is: a goldmine of analytics on your visitors. Be sure to listen to this feedback, checking for popular searches, searches that return no results, autocomplete selections where applicable, and any other information about what your visitors are looking for. You should then pipe these learnings into your marketing and product strategy. For example, an ecommerce company who sells shoes might notice that their visitors are searching for a brand they do not carry, suggesting that they should consider adding the brand.
  2. Ensure your search is ready for mobile. The share of searches of occurring on mobile websites and in mobile apps is steadily increasing. As users begin to expect their experience to be seamless across their devices, a well-designed and powerful mobile search feature is becoming as important as a desktop search solution. Good mobile search generally includes a prominent, easily accessible search bar, as navigational elements such as breadcrumbs are often difficult to display clearly for mobile visitors. Also, be sure to optimize your results page, autocomplete dropdown, and product thumbnails for the smaller screen space available to mobile users.
  3. Feature your search box prominently and intuitively. Visitors who start with search are your most valuable visitors—as much as 70% more likely to convert than those who avoid search. A search indicates a visitor came to the site for a reason, and a strong search experience moves that visitor to their desired destination quickly and painlessly. Featuring search also encourages your visitors to tell you what they are looking for, critical to an effective marketing and product strategy. To encourage searches from your visitors, make sure your search bar is prominent and intuitively located so that your visitors naturally use the feature. For inspiration, we’ve highlighted a few of our favorite search-driven site designs.
  4. Offer autocomplete results. For each additional page a visitor sees on a website, there is an associated visitor drop-off. Offering autocomplete results in a dropdown reduces the time a searcher needs to spend waiting for a results page to load. This streamlined experience will delight your visitors and reduce friction in their experience.
  5. Don’t ignore misspellings. We’ve all made typos when searching for something, either inadvertently or because we don’t know how to spell a tough word. The success of companies such as Google and Amazon in typo tolerance leads the average internet user to expect search tools to protect them from their mistakes. However, most do not come nearly close enough to Google or Amazon in typo protection, causing situations like this:
    Misspellings can cause frustrating no result pages.
    As the screenshot shows, a missed space between words—the most common typo—in a query that should have results can return none. As modern searchers expect typo protection, not accounting for misspellings can lead visitors to believe their intended query has no results, driving them away from your site. Be sure to pay attention to the most common misspellings, and take steps to ensure your visitors do not mistakenly see a no results page.

Many of these issues are best addressed by using a third-party tool that has developed easy tools for managing issues like results reordering, spellcheck, synonyms, and autocomplete. However, each of these issues represent common problems with native search tools, and each can be rapidly improved simply by paying attention to them. We’ll be featuring more tips for improving your search in the future, so be sure to check the blog often. Finally, if you are looking for a solution to each of these issues that you can get up and running quickly, sign up for a Swiftype account and create your first engine.

Swiftype for Desk.com

desk_logo
Swiftype search is an invaluable tool for self-service support centers. That’s why we’re excited today to announce our integration with Desk.com, making it easy for support centers built on Desk.com to bring powerful search to their users. With Swiftype powering your Desk.com support center search engine, site owners get

  • Detailed analytics. See what users are searching for most to identify common user issues, and see what queries are returning no results to identify gaps in your knowledge base. From there, site owners can create content to address these questions, or use Swiftype’s custom result ranking tool to add in search results for any query they want.
  • Custom result ranking. Drag and drop to rearrange search results for individual queries through Swiftype’s user friendly dashboard, add in results that don’t appear by default, or eliminate individual results entirely. With this power, site owners can tightly customize search results for high volume queries, quickly guiding users to the right support content.
  • Complete front end customization. Once Swiftype is installed, site owners can customize the look and feel of their search bar, autocomplete menu, and search results page to match the aesthetic of their help centers. For even more control, it’s possible to work with Swiftype meta tags to specify exactly what information our web crawler indexes as well as add custom fields and data.

And much more. For more information about Swiftype for Desk.com, check out our listing on the Desk.com app hub, or, for more information about Swiftype for self-service support centers and knowledge bases, check out our case studies.

The importance of robust Shopify site search

Swiftype for Shopify

When a shopper visits your store, many have a clear idea of what they want to purchase. Because of their extensive experience with websites like Google and Amazon, these shoppers are naturally drawn to the search bar when they have a specific product in mind. Data bears this out – visitors who begin their visit with search are 70% more likely to convert than those who do not, making them some of the most high value visitors on your site. A strong search engine for your store is key to ensuring that your most valuable customers have a clear and simple path from search to checkout.

Problems

Shopify, along with most other ecommerce platforms, comes up short in providing the type of search experience your customers expect. The reason behind this shortcoming lies in the fact that implementing a powerful ecommerce search engine is a very technically complex problem. While Shopify ensures that you have a search option on all their stores, they have (correctly) allocated their engineering resources towards building a reliable, fast, and user-friendly ecommerce hosting product. This challenge is complex enough without adding the complexity of robust ecommerce search with the features users expect, such as autocomplete, spellcheck, typo protection, and an advanced algorithm that handles phrases and single word searches seamlessly, not to mention optimizing them for each and every one of their users.

The Risks of Weak Shopify Search

Shopify stores with underwhelming search engines miss out on numerous valuable opportunities, not only in maximizing their conversion and customer value, but in customer intelligence. There’s a reason Amazon features search so prominently in its user experience, beyond the conversion benefits: the invaluable user data they collect about their customers’ purchasing habits and click-through rates. Search queries are a direct signal of user intent – a free text box asking simply “What are you looking for?” Store owners should be paying extremely close attention to visitor trends such as their top searches, the most popular products in search results, and the searches that most commonly yield no results, and incorporating this data into both their product offerings and marketing strategy.

Fix it Yourself or Hire a Professional?

If you employ a large development team and have the resources and capabilities to build, maintain, and scale a robust ecommerce search engine, building your own search solution might be the right approach (after all, Amazon does this). However, if you are like the vast majority of Shopify store owners, you chose Shopify as a way to keep your development costs down and aren’t interested in hiring a team of developers who only work on maintaining an excellent search experience. In this case, using a third party provider is likely your best solution, as you’ll have a team of search professionals working full time to ensure you have the best search available. At Swiftype, search is the core of our business, and we have developed a Shopify search product that provides a beautiful and intuitive search experience, a powerful search algorithm addressing the problems of out-of-the-box search, and a user-friendly dashboard that allows non-technical stakeholders nearly infinite customization and control over their store’s search experience (and we’re releasing more dashboard features regularly). We even power search for Shopify. For more information on Swiftype for ecommerce stores, visit our solutions page today, or to install search immediately, follow the link below.

Browsing vs. buying: UX design considerations for mobile shoppers

In 2015, it is not surprising to online retailers that mobile users comprise a major portion of online shopping. Phones are becoming bigger, faster, and more user friendly; kids who have grown up using smartphones and tablets are entering the consumer market in a major way; and ecommerce websites are quickly adapting by building responsive, mobile optimized websites and apps. All of this has contributed to the significant rise in mobile traffic and sales over the past several years, with nearly half of all online traffic and almost a quarter of sales in the 2014 holiday season taking place on smartphones or tablets, according to a recent study by IBM. The takeaway is clear: today’s consumers expect seamless mobile shopping experiences, and online retailers need to adapt quickly to keep pace with industry leaders.

Mobile traffic is growing, but desktop still dominates in sales.

However, a closer look at this data suggests that the division between mobile and desktop shopping is not so black and white. After splitting mobile traffic and sales data between tablets and smartphones, it seems clear that many users have different preferences for what device they browse with vs. what device they actually complete a purchase on. For instance, IBM’s data reveals that although smartphone traffic more than doubled that of tablets, sales on tablets were over four percent higher. This contrast, coupled with the relative swell in desktop sales vs. desktop traffic, suggests that smartphones are primarily used for research and browsing, while tablets and desktops are favored for completing transactions.

What can site owners learn from this behavioral trend, and how can they optimize their mobile shopping experience(s) to satisfy these browsers? The answer to this question revolves around two main topics of discussion:

  1. What elements of the user experience are most important for researching and browsing?
  2. Should site owners focus on building apps or creating mobile optimized web browsing experiences?

Let’s explore these two questions independently before returning to a discussion of where mobile shopping UX design might go in the future.

User Experience: how to build for browsers

Compared to desktops, mobile phones do not offer nearly as much space on the page to place navigational elements, such as product categories. These elements could be placed in a dropdown menu activated by a single “menu” link, but these can be difficult to design and clunky for smartphone users. A better alternative is to feature a prominent search bar that persists across all subdomains, allowing shoppers to search, browse, and re-search without having to go back at any point in the process. Amazon is a leader in this respect, with a large search bar that provides autocomplete suggestions for users as they type. Statistics clearly demonstrate the importance of search for mobile browsers. A recent survey by Harris and IAB of over 2,000 smartphone users revealed that search is the primary portal through which users find new content on their mobile devices.

Furthermore, product thumbnails should be enlarged to compensate for the reduced screen size—basic responsive design is not enough if your search results are still displayed in a 4×4 product grid on a small phone screen. Additionally, users should be able to zoom in on high resolution product images so that they can look closely before adding them to their cart. These UX changes are relatively simple but important principles that should inform both mobile optimized and application UX design.

Prioritization: creating apps or optimizing for smartphones

In an ideal world, site owners would not need to make a decision on this issue, instead allocating engineering resources to simultaneously build an app and mobile optimized browsing experience independently. However, most websites lack the resources to move so quickly on this front, and are forced to adopt a strategy for developing one before the other. Internet Retailer’s 2015 Mobile 500 report presents a key statistic in this debate, revealing that 80% of online shopping takes place within an app, and that users within an app are 30% more likely to convert. This means that more traffic and sales flow through apps, making a seemingly clear case for focusing on app development.

Still, mobile browsing experiences should not be overlooked. As we touched upon earlier, many online shoppers begin by browsing on their phone even though they often won’t ultimately complete a purchase through this channel, making this mobile browsing experience a critical first impression. Furthermore, Internet Retailer admits that the 80% of traffic in apps statistic is, “surely skewed by large numbers of loyal users of apps from big players such as Amazon.com and eBay.” Taking these caveats into consideration, mobile browsing remains an important UX priority.

Looking forward

Though desktop remains the dominant platform for shopping research and sales, its share of online sales and traffic has been on a steady decline for years. Today, a failure to address mobile design translates to lost opportunities, especially when mobile browsing is considered as the “first impression” that modern shoppers have of an online retailer. For details on how Swiftype helps websites deliver superior mobile browsing and in-app search experiences, check out our mobile solutions page today.

Laser Focused: B2B ecommerce white paper

Today we are excited to announce our first white paper: “Laser Focused: Why B2B Buyers Demand Powerful Search”. This report synthesizes a number of ecommerce and B2B reports from 2014, but places a special focus on buyer behavior and the importance of site search for the B2B consumer.

Download Laser Focused: Why B2B Ecommerce Buyers Demand Powerful Site Search.

Internet Retailer’s industry-wide October survey of B2B professionals revealed that site search is the most important feature that buyers look for on a supplier website, making it clear that powerful site search is a clear competitive advantage in the quickly changing space of B2B ecommerce. Laser Focused discusses the underlying reasons behind this trend, and provides concrete recommendations about what site owners can do to improve search on their store in 2015. Along with this discussion, the white paper addresses questions such as:

  • How has the growth of online shopping transformed the traditional model of B2B commerce, and how have consumer expectations shifted over time?

  • What are the key differences between B2B and B2C buyers? What are their similarities?

  • Why is a basic search algorithm insufficient for B2B buyer behavior? What advantages do more robust, complex algorithms provide?

  • What does a powerful search experience look like? What are the options for implementing powerful search on ecommerce websites?

To download Laser Focused, click on the link below. To get in touch with a team member today to talk about your search, contact [email protected].

Year in review: our favorite implementations from 2014

To kick off 2015, we wanted to highlight some of our favorite 2014 Swiftype implementations across a range of different websites to demonstrate the results that some of our users have been able to achieve. In all, Swiftype started powering search on thousands of websites this year, and served billions of queries. Nearly three years after launching, we are excited that so many people have found Swiftype to be such an integral part of their website, and we want to share these stories to inspire users in the future.

Qualcomm

Qualcomm is a global semiconductor company that designs and markets wireless telecommunications products and services, serving millions of customers worldwide, with more than 25,000 employees across 150 offices. When Qualcomm approached a leading web development agency to help them rehaul their existing website, search was a top priority. Because Qualcomm offers such a wide variety of products and services, they wanted to create an easy browsing experience to let customers hone in on exactly what they are looking for quickly.

Qualcomm made search a central pillar of their website user experience.
Qualcomm’s search implementation is great for several key reasons. To begin with, their search tool is prominently displayed on the left side of the page and remains visible on every page of the website. This makes it easy for users to search, browse, and re-search without needing to scroll to the top of the page. Another aspect we like is their autocomplete and faceted search options. The combination of these two features cuts down the time users need to spend searching by directing them straight to content they are looking for and letting them refine their results without moving to a separate results page. Lastly, we are happy to see the great results that this web development agency produced with Swiftype—providing their clients with a powerful search engine without the need for back end development work.

HubSpot

HubSpot is a major inbound marketing and sales platform that came to Swiftype for help powering their knowledge base search engine. While HubSpot’s support team had spent countless hours developing helpful content to answer user questions, their previous search solution did not allow enough control over their search experience. By switching to Swiftype, HubSpot was able to completely customize their search engine and create the user experience they hoped for.

Hubspot is a great example of powerful search over a large knowledge base.

HubSpot’s search is simple yet effective. For starters, we appreciate how they prominently displayed the search bar on their support page—a clear prompt for users to begin their support questions with a search. Furthermore, the placeholder text in the search bar clearly tells users how to interact with the search box, reassuring them that they should feel free to “type your question here.” From there, the fast autocomplete drop-down is color coded to differentiate between “quick answers” and “user guides,” letting users choose results based on the article type. Similar refinement options are available on their results page, where users can select the types of articles they’d like to see displayed.

Qualified Hardware

Qualified Hardware is a B2B retailer that sells high quality door and lock hardware. Before switching to Swiftype, Qualified Hardware was using a home-grown search solution that was costly to maintain and often failed to return relevant results for the highly specific queries that their customers were performing (such as item or part number searches). Swiftype’s powerful search algorithm helped return better results immediately, and to bring their search to the next level, Qualified Hardware took advantage of Swiftype’s custom meta tags to create a highly refined relevance model, as we’ll discuss below.

A quick search will immediately reveal the great work that their team put into styling their autocomplete with suggested brands, categories, and specific products. The results page also offers the option to display products in a grid or list format. These elements give users a great front end experience, but what makes this implementation truly outstanding is the skill with which Qualified Hardware leveraged Swiftype meta tags to deliver highly relevant results based on a wide range of product attributes.

Qualifed Hardware distinguished itself by creating a highly refined relevance model optimized to drive conversions.

Swiftype meta tags are a unique tool that any site owner can leverage to pass specific information to our web crawler as it indexes your site’s content. This allows site owners to fine-tune their relevance algorithm and deliver highly relevant results (for specific details about using Swiftype meta tags, see our tutorial). Qualified Hardware used this tool to its fullest extent, listing detailed information for each product such as popularity, whether or not the item is in stock or needs to be special ordered, the product SKU, and more. A quick peek at their source code on a product page will demonstrate just how extensively they used this tool. Once this information was indexed, Qualified Hardware was able to tweak their search algorithm to feature popular products, move special order products to a lower position for general queries, and overall optimize their results for conversions.

Modern Healthcare

Modern Healthcare is a leading source of healthcare business news, research, and data, with thousands of articles and publications. Before Swiftype, Modern Healthcare used their CMS’s default search function, which was slow, produced poor results, and was difficult to customize. Swiftype gave them improved speed and relevancy, with the ability for non-technical team members to customize search results.

Modern Healthcare's search bar is easy to find across their whole website.

Like Qualcomm, Modern Healthcare made their search bar easy to find by floating it in a static header element that follows users as they scroll down the page. This makes it easy for users to always have access to search and begin a new browsing process from any page on the site. Once users begin searching, the autocomplete quickly suggests articles, while displaying thumbnails of images associated with each article. Finally, on the results page, users have the option to refine results by publication date— ensuring that results are from the specific timeframe they have in mind.

These implementations demonstrate some of the incredible results our customers have been able to achieve with Swiftype. If you’re thinking about taking your search experience to the next level, we’re here to help make it happen. To see how Swiftype can work for you, contact [email protected] to request a demo.

Subscribe to our blog