We express gratitude for preparation of article to Artem Zhukovets — CTO NovaPress Publisher. Winner project of the tender "Heroes of the Russian Startups"
Hi everyone! Today we will tell about architecture of the NovaPress Publisher service and experience of its transfer in Microsoft Azure cloud.
NovaPress Publisher – web service which facilitates to the companies work with social networks. It allows to plan content on social networks for days and the months ahead. And also automatically to take content in process of emergence on the website of the company and to post in all social networks.
At the moment this service is already used in work more, than 9500 companies, including the known brands — logs, newspapers, TV channels, the commercial and state organizations.
As everything began
Our service appeared in 2010, but to work with Azure we came not at once. In the first years of life of service we used for work virtual computers on different hostings.
We were faced by a task – without delays to process and post in social networks on several thousands of records a minute in the mode 24/7.
And at the same time periodic interruptions in operation of virtual computers were this problem. They led to the fact that in queue the huge number of unpublished records accumulated, or content created by users became temporarily unavailable.
Besides, in rush hours load of service was too big, and there were delays when sending records.
At that time service had to send up to 6000 records a minute to social networks and synchronize up to 1500 RSS feeds a minute. Now these values it is significantly more.
Loading in morning and evening hours, especially 00 minutes and 30 minutes was highest (for example, 17:00, 17:30, 18:00 and so on).
Our solution at that time was such:
The robots which are carrying out the scheduled tasks (publishing content in social networks) were duplicated on several servers, also, as well as content created by users. With growth of number of users we connected additional servers.
However over time scaling was given harder and harder, new bottlenecks opened. Besides, periodic failures at a hosting providers created to us vital issues. And instead of implementing new functions in our service, we spent a lot of time for coping with the increased loading and to provide despite everything high availability of service and the publication of records without delays.
Therefore we began to look for a new solution which could protect us from failures and provide flexible scaling depending on loading.
And as a result, in 2013 we stopped the choice on a cloud of Azure in which scaling options, replications and balancing of loading were available at once, from a box.
Transition to Azure
The scheme of work of our service after transition to Azure cloud became such:
Users work with service through the website (Azure Website, ASP.NET MVC), data are stored in SQL database of Azure and Azure Storage storage. For quick access to data Redis cache in a cloud is used.
All automatic equipment (placement of records in social networks, copying of records from the websites or other social networks) execute Windows services, placed on a large number of Azure VM virtual computers. These services work in several flows, carrying out tasks of a measure of their receipt in queue of ServiceBus. If tasks become more, and loading raises, we increase the number of virtual computers.
Besides, peak minutes (18:00, 18:30, etc.) service is reconfigured (automatically reduces number of the robots which are carrying out minor tasks and increases number of the robots publishing in social networks) that as soon as possible and without delays to publish content in social networks.
We carried out transition to Azure in several steps, since the most important:
- Transfer of the database. Before transition to a cloud we used the SQL Server base. Migration of the SQL Server base in cloudy SQL Azure took place quite simply, by means of a special application. After transition we got a stable job of the database without failures. (level of availability of 99.99%).
- Transfer of the user content. Before transition to a cloud ourselves duplicated the user content (a photo, the text) on several servers to provide availability in case of failure. But it demanded many resources and disk space. Upon transition to a cloud we transferred all content to Azure Storage with automatic georeplication (all content is automatically duplicated in 3kh Azure data-centers). Availability level: 99.99%
- Transfer of queue of tasks in ServiceBus. Earlier queue of tasks (for example, records which need to be published) was tied based on data. After transition to Azure service sends all tasks which need to be executed, to queue of ServiceBus. It unloaded part the database and significantly increased the potential of further scaling of service.
- Transfer of the website in a cloud. The website was transferred to Azure Website and included automatic scaling that the website was always available and better held loading.
- Cash Redis in a cloud. We use Redis to provide the fastest placement of records on social networks. Content prepared for the publication and settings of its publication are stored in a cache that the necessary minute, as soon as possible to go to social networks. Redis in a cloud already from a box has automatic replication and guarantees availability of at least 99.99% of time.
In December, 2015 we let out a new beta in which improved a service stuffing:
- The service website represents AngularJS SPA-application now. All HTML forms, scripts and styles of a minifitsirovana are also loaded at start that further switching of forms was instant.
- Work with data goes through a separate web application of ASP.NET Web API. You challenges of Web API are made so that as soon as possible to receive result of operation. The caching by means of Redis is actively used.
In further plans at us the following improvements:
- Work with service in real time thanks to WebSocket (SignalR). As several employees of the company at the same time work with service often, it would be desirable that the changes made by each user instantly were displayed at other employees. Also the user will at once see as soon as service carries out this or that task (for example, will publish records on social networks, or will load new records from the website).
- Monitoring and analytics on social networks. Will show, how effectively the company worked with social media and will prompt how to improve these indicators. In this plan we look towards Stream Analytics and Machine Learning.
The received results
As a result of transition to Azure we:
- provided high availability of our service to clients. Everything works like clock-work and it cannot but be pleasant to clients.
- received performance stock for the further growth of service. Now service posts in social networks 23000 records a minute, but it is not a limit.
- released our time for other important tasks, having shifted ensuring high availability and scaling to Azure. So we could concentrate on improvement of service and adding of new functions completely.
About the author
Artem Zhukovets is the Technical director of the company of NovaPress Publisher
This article is a translation of the original post at habrahabr.ru/post/273039/
If you have any questions regarding the material covered in the article above, please, contact the original author of the post.
If you have any complaints about this article or you want this article to be deleted, please, drop an email here: email@example.com.
We believe that the knowledge, which is available at the most popular Russian IT blog habrahabr.ru, should be accessed by everyone, even though it is poorly translated.
Shared knowledge makes the world better.