Context
Xayn is a Berlin-based AI company; its core product is a News Reader that offers a continuous, personalized news feed without advertising or data storage. The company started as a research project at The University of Oxford and Imperial College London in 2017. To this day, they remain research-focused, with a workforce of around 30% PhDs.
Recommendation systems, like personalized content feeds, have existed for quite some time. Generally, these recommendation engines send user data to the cloud and use it to fetch relevant content. But with the slew of data leaks and questionable data handling practices of some tech companies, there has been a growing demand for privacy-first alternatives.
Problem
An obvious solution to this problem is to run the recommendation systems on the end user’s device, but it isn’t that easy. Edge devices have limited processing power and cannot run heavy AI models. So there is an inherent trade-off between privacy and the quality of recommendations a recommendation system can offer.
In our journey to create a privacy-centric personalized news feed, we realized that even though our users value privacy, they aren’t willing to compromise on the quality of news content recommended to them. Over five years and many iterations, we have developed a suite of multilingual NLP models that are efficient enough to run on the minimal computational power of mobile phones and browsers. These models offer state-of-the-art quality recommendations without sending user data to cloud servers.
The NLP models take as input the previously read articles (and for our version 3.0, time spent on the articles), they craft news queries and fetch the most relevant articles. And for a smooth user experience, we need all this to happen in real-time.
To make the best of these NLP models, we need a reliable news provider with excellent multi-language coverage of the areas and publications we are interested in. Besides that, the news provider needs to be able to handle a lot of requests without an increase in response latency.
Solution
We checked out around 30 to 40 different news providers. We ruled out some because they don’t fit our business model. Many news providers and APIs, like PR companies, focus on customers who want analysis. Their business model is not concentrated on scaling but on making a few fetch requests per month, which does not match our needs/use case.
Secondly, we decided on the NewsCather News API because of their transparency in how they do things. A lot of these other news data companies are super secretive. Our entire code base, from our infrastructure, our AI, our engine, everything is open source. So someone can steal everything; they don't even have to steal it because it’s already there.
NewsCatcher at least transparently describes what they do in their documentation. You can understand how things work and quickly understand if something can be adapted to suit your needs better. Other news providers are more protective of their methods, and often when you request changes or additions, you’re told no because they’re using something that restricts them from doing it.
We integrated NewsCatcher's News API into our pipeline in the March of 2022. Our News Reader app uses it to search for the most relevant news articles for thousands of users. Based on the end-user interactions, the app generates queries. These queries are sent to a proxy (that strips them of information like IP address, etc.), which then communicates with the NewsCatcher News API. The response from the API passes through a CDN network, and the JSON file is finally sent to the end device. Where finally, the user interacts with the stack of relevant news content.
The NewsCatcher News API works well for Xayn because it allows us to run a large number of unique requests per day for all our users. Artem, Maksym, and the rest of the NewsCatcher team have been incredibly helpful in collaborating with us to optimize the infrastructure and pipeline to provide the best possible user experience for our users.
“The most important question for us was the working relationship. So simply put, whenever we have something, the NewsCatcher team replies, faster than expected. They also solve things very fast. They’re very reactive. So it’s an excellent working relationship.”
Results
Using NewsCatcher's News API, the Xayn team is:
- actively pulling real-time coverage from 10+ countries in multiple languages
- creating the industry standard for privacy-first personalized news feeds
- rapidly iterating and adding new features to their recommendation engine
Get in touch with us for more details.