From Zero to One in AI - How Relentless Exploration Led Us to an Unfair Advantage 💪🔥

Mon, 05 Jun 2023 00:00:00 GMT

#Introduction

Over the past year, we have witnessed a remarkable transformation in the industry, as artificial intelligence (AI) has transitioned from being perceived as niche and technical to being embraced by businesses worldwide. Its adoption has become widespread across various industries.

At SaaS Labs, we embarked on an AI journey just a year ago, armed with enthusiasm and a determination to succeed. With no prior experience in the AI space, we relied on relentless exploration and a commitment to learning from industry leaders. Today, we are proud to share our progress and the framework that has empowered us to develop and deploy over 10 AI models within a year, which now form an integral part of our product suite, including Justcall iQ, Justcall, & Helpwise giving us an unfair advantage in the industry.

#The Power of Exploration

When starting from scratch, it can be overwhelming to navigate the complex world of AI. We recognized the need to immerse ourselves in the latest industry trends, insights, and best practices. Our team actively sought out and devoured numerous articles, research papers, and case studies to gain a deeper understanding of AI fundamentals. We also engaged in meaningful conversations with established experts who graciously shared their knowledge and experiences. This relentless exploration became the bedrock of our success.

#Building the Framework

Armed with newfound knowledge, we set out to create a robust framework that would enable us to develop and deploy AI models quickly and efficiently. Our aim was to establish a process that would support scalability while maintaining high standards of quality and reliability.

#1. Data Collection and Preparation

In data collection, we gathered relevant data from public sources and modified it to suit our specific use case. Our team played a vital role in refining and preparing the data for training our AI models. This meticulous process ensured the quality and accuracy of our models.

#2. Development & testing.

During the development and testing phase, we utilized tools such as Jupyter Notebook and Google Colab to build our models. Our approach was flexible, and we explored various solutions to find the optimal fit for our needs. In some cases, we employed virtual machines (VMs) for training our models, and we also explored the capabilities of Vertex AI. While we continue to search for the ideal solution that can fulfill all our use cases, our focus remains on finding the perfect fit to meet our specific needs and drive optimal results.

#3. Deployment and Scalability.

In deploying our AI models and ensuring scalability, DevOps team played a crucial role. We leveraged technologies such as Docker, Google Kubernetes Engine (GKE), Jenkins, and Bitbucket. Docker enabled easy deployment, while GKE provided a scalable environment. Jenkins automated our CI/CD processes, and Bitbucket facilitated efficient source code management. With the help of our DevOps team and these tools, we successfully deployed and scaled our AI models

#Realizing Our Unfair Advantage

The fruits of our labor quickly became evident as we witnessed the successful deployment of over 10 AI models across our products within a year. Our framework streamlined the development process, reducing time-to-market and increasing efficiency. With each new model, our knowledge base expanded, allowing us to tackle more complex challenges and deliver even greater value to our customers.

Demystifying Search

Sat, 22 Oct 2022 00:00:00 GMT

Search is a very normal use case while building communication applications. As customers send and receive messages using our products, they’re generating lots of information that they also want to access using text queries. Let’s take the example of Helpwise, which needs to find the correct email from millions of emails. This post discusses how we find the correct message for a search query in Helpwise & how we make our search engine able to handle gazillions of messages.

#Search Query

Let’s look at how this search works for your use cases. Suppose we have 100 emails to search; then, in general, we need to check all of those documents 1 by 1 and see which document matches our query. In SQL, this can be written as:

SELECT * FROM emails WHERE body LIKE '%<query here>%'

If you look carefully, this query has a complexity of O(n) as the underlying Database that goes through each and every record to match the search query. This means if we have millions of emails, then the time taken would increase, which is pretty bad as the number of messages is virtually infinite.

This can be further optimized by restricting the database to search into the records of the particular customer by having a secondary index on something like the account id. Below is the syntax for the same:

SELECT * FROM emails WHERE body LIKE '%<query here>%' AND account_id=<account id>

Still, we don’t know how many emails can be in a single account, and it doesn’t allow us to offer performant searches.

#ElasticSearch to the rescue

Elastic Search is the rescue to the basic search query & it helps to accelerate the search results. According to elastic,

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.

#How it works?

It stores data in a data structure called an inverted index that allows faster text retrieval.

#Inverted Index

Let’s say there are 3 messages with the following text:

DocumentId	Text
1	quick brown
2	kangaroo jumps
3	quick brown fox jumps

Here, Inverted Index breaks the text and uses it as the Index key.

Text	DocumentIds
quick	1, 3
brown	1, 3
fox	1, 3
kangaroo	2
jumps	2, 3

Now, if we need to find messages with the keyword “jumps”, an elastic search plays the role of getting the results in a constant time.

#Distributed?

The above inverted index example was a very simple one. Now, imagine the same with millions of email/text messages. It can create a virtually infinitely large inverted index. Saving this in a single storage device limits the ability to scale it in terms of computation and storage. So, Elastic Search breaks data into almost equal-sized shards and distributes it among multiple nodes, making horizontal scaling possible.

More or less, Elastic Search abstracts the complexity of saving and searching the inverted indices metadata.

#Reasons to choose ElasticSearch

Simple Restful APIs
Uses JSON to represents documents.
Scalable
Good Industry support

Pitayan's RSS Feed