Choosing the best language to build your AI chatbot

No, this is not about whether you want your virtual agent to understand English slang, the subjunctive tense in Spanish or even the dozens of ways to say “I” in Japanese. In fact, the programming language you build your bot with is as important as the human language it understands.

But how do you differentiate between them? Facebook, Slack and Telegram all support the most popular languages, while API platforms such as Dialogflow, LUIS and wit.ai offer SDKs for the majority.

Of course, the caveat should always be to veer toward the language you are most comfortable with, but for those dipping their toe into the programming pond for the first time, a clear winner starts to emerge. Python is the language of choice.

Why Python and not the others: natural language processing

Python is essentially the Swiss Army Knife of coding thanks to its versatility. It also is one of the easier languages for a beginner to pick up with its consistent syntax and language that mirrors humans.

This meant that when Python was first released it was applied to more diverse cases than other languages such as Ruby, which was restricted to web design and development. Meanwhile, Python expanded in scientific computing, which encouraged the creation of a wide range of open-source libraries that have benefited from years of R&D.

With regards to natural language processing (NLP), the grandfather of NLP integration was written in Python. Natural Language Toolkit’s (NLTK) initial release was in 2001 — five years ahead of its Java-based competitor Stanford Library NLP — serving as a wide-ranging resource to help your chatbot utilize the best functions of NLP.

Stanford NLP and Apache Open NLP offer an interesting alternative for Java users, as both can adequately support chatbot development either through tooling or can be explicitly used when calls are made via APIs. But NLTK is superior thanks to its additional support for other languages, multiple versions and interfaces for other NLP tools and even the capability to install some Stanford NLP packages and third-party Java projects.

While critics argue that NLTK’s inefficiency and steep learning curve make it more of an academic’s theme park than the solution to chatbots, TextBlob solves this problem by using it as a springboard to provide a more intuitive interface and a gentler learning curve for users.

An interesting rival to NLTK and TextBlob has emerged in Python (and Cython) in the form of spaCy. It does have some advantages. Namely, that it implements a single stemmer rather than the nine stemming libraries on offer with NLTK. This is a problem when deciding which one is most effective for your chatbot. As seen here, spaCy is also lightning fast at tokenizing and parsing compared to other systems in other languages. Its main weaknesses are its limited community for support and the fact that it is only available in English. However, if your chatbot is for a smaller company that does not require multiple languages, it offers a compelling choice.

NLTK is not only a good bet for fairly simple chatbots, but also if you are looking for something more advanced. From here a whole world of other Python libraries is opened up to you, including many that specialize in machine learning.

Machine learning

On the subject of machine learning, what better approach than to look at some hard data to see which language the experts prefer? In a recent survey of more than 2,000 data scientists and machine learning developers, more than 57 percent of them used Python, while 33 percent prioritized it for development.

Why is this? Similar to NLP, Python boasts a wide array of open-source libraries for chatbots, including scikit-learn and TensorFlow. Scikit-learn is one of the most advanced out there, with every machine learning algorithm for Python, while TensorFlow is more low-level — the LEGO blocks of machine learning algorithms, if you like. This versatility is why Python shines.

Many of the other languages that allow chatbot building pale in comparison. PHP, for one, has little to offer in terms of machine learning and, in any case, is a server-side scripting language more suited to website development. C++ is one of the fastest languages out there and is supported by such libraries as TensorFlow and Torch, but still lacks the resources of Python.

Java and JavaScript both have certain capabilities when it comes to machine learning. JavaScript contains a number of libraries, as outlined here for demonstration purposes, while Java lovers can rely on ML packages such as Weka. Where Weka struggles compared to its Python-based rivals is in its lack of support and its status as more of a plug and play machine learning solution. This is great for small data sets and more simple analyses, but Python’s libraries are much more practical.

Where does Python struggle?

Python’s biggest failing lies in its documentation, which pales in comparison to other established languages such as PHP, Java and C++. Searching for answers within Python is akin to finding a specific passage in a book you have never read. In addition, the language is severely lacking in useful and simple examples. Clarity is also an issue, which is incredibly important when building a chatbot, as even the slightest ambiguity within one of the steps could cause it to fail.

If speed is your main concern with chatbot building you will also be found wanting with Python in comparison to Java and C++. However, the question is when does the code execution time actually matter? Of more importance is the end-user experience, and picking a faster but more limited language for chatbot-building such as C++ is self-defeating. For this reason, sacrificing development time and scope for a bot that might function a few milliseconds more quickly does not make sense.

Natural language processing implemented with Python

Let’s take a look at one aspect of NLP to see how useful Python can be when it comes to making your chatbot smart.

Sentiment analysis in its most basic form involves working out whether the user is having a good experience or not. If a chatbot is able to recognize this, it will know when to offer to pass the conversation over to a human agent, which products users are more excited about or which opening line works best.

We could use sentiment analysis to determine if an interaction is negative or positive. Have a look at this sentence for example:

“Brilliant, my card is not working.”

Of course, the sentiment here is negative, but that might be difficult for a bot to detect given the word “brilliant” is used. How do we equip our bot with robust sentiment analysis? Note: Examples of the actual functions that have been described below can be found here and here.

While it is arguably much simpler to use spaCy and TextBlob, understanding how NLTK works provides a solid grounding in order to help grasp the concept of sentiment analysis. Using NLTK, we can train a bot to recognize sentiment by first examining a set of manually annotated data. We create this by taking three lists: one of positive comments, another of negative comments and a test list that contains a mixture. The more examples we have on each list the more reliable the sentiment analysis will be. The manually annotated data will test the exactitude of our classifier.

Following this, we need to extract the most relevant words in each of the sentences (in the example given above it would be “brilliant,” “not” and “working”) and rank them based on their frequency of appearance within the data. To do this we can get rid of any words with fewer than three letters. Once completed, we use a feature extractor to create a dictionary of the remaining relevant words to create our finished training set, which is passed to the classifier.

The classifier is based on the Naive Bayes Classifier, which can look at the feature set of a comment to calculate how likely a certain sentiment is by analyzing prior probability and the frequency of words. From here, a measurement of how likely a sentiment is can be given.

While it is factually correct to argue that “language is just a tool” to equip your chatbot with AI, using Python and its wider variety of libraries and off-the-shelf algorithms means it is a much more straightforward option than other languages.

Like choosing the best tires on your racing car, the language you choose for your chatbot depends on a number of conditions. What kind of bot are you hoping to create? With which language are you most comfortable? Which is robust enough to handle your specific project as it continues to grow?

But if you are starting out fresh and are wondering which language is worth investigating first to give your chatbot a voice, following the data science crowd and looking at Python is a good start.