Developers Club geek daily blog

Spark local mode: processing of big files on the normal notebook

1 year, 2 months ago
image
All hi.
On January 4 there was a new version of introduction before experience of use in projects. Spark works at the majority of operating systems and it can be started in the local mode even on the normal notebook. Using simplicity of the Spark setup in this case a sin not to use the main to functions. In this article we will look as on the notebook quickly to configure processing of the big file (more random access memory of the computer) by means of normal SQL queries. It will allow to make requests even to the unprepared user. Additional connection of iPython (Jupyter) notebook will allow to make full reports. In article the simple example of processing of the file is sorted, other examples on Python are here.

Read more »


Reefs of use of Excel Power Query and MySQL for automation of the reporting

1 year, 2 months ago
image
All hi.
There came new 2016, so it is time to update tools for simplification of boring mechanical work. Departments of analytics, marketing, sales often face the following difficulties when updating the reporting:
1. Data should be aggregated from several sources.
2. Reports are formed in Excel that imposes considerable restrictions for volume of the processed data.
3. Modification of the unloadings which are in advance configured by developers business as a rule not the faster.

If reports need to be updated weekly or even daily, then this procedure becomes very napryazhny even for the most patient. By means of a superstructure of Excel Power Query and data record in MySQL it is possible to reduce updating of the majority of reports before simple clicking of the Update button:
1. Data from any quantity of sources are imported through SQL queries to the normal tables Excel.
2. Even from big base it is possible to write in Excel only small part of data (for example, total sums for the necessary range of dates with grouping only on the necessary columns).
3. Changes can be made to the report just having changed the SQL query. Further we create the necessary report standard means of Excel.

I will show how to configure and autocomplete the simple MySQL databases (on the example of unloading of statistics of all key word from Metrics Yandex) in this article, and then one button to update reports in Excel, using Power Query superstructure. Power Query has very strange features of work by drawing up SQL queries (especially dynamic) which we will sort in the second part of article.

Read more »


Audio and video in the Tox messenger

1 year, 3 months ago
While one people think of regulation of messengers, other people develop the distributed messengers. In the previous publication use of API of a kernel of the Tox messenger on the example of creation of a simple echo-bot was considered. Development of Tox does not stand still and on November 3rd the kernel of Tox was enriched with a new subsystem of audio and video calls — ToxAV about which I also would like to tell in this publication.

image

Read more »


Non-standard approach to standard development of addition (Add-In'a) on C#

1 year, 3 months ago
By means of Add-In'a it is possible to implement the additional functionality which is seriously facilitating daily use of the application. Standard approach is simple – we study API applications for development of addition and we implement it in the form of separate library. At choice several languages are usually offered, С# most likely will be in this list because of its prevalence and popularity for applications programming under Windows on the .Net platform.

However often happens it is necessary that addition constantly displayed some visual information, necessary when using the application. At a simple solution of this problem — creation of the user window and its display or over all windows, or its movement of the foreground — we receive some inconveniences with the organization of a user space. With them it is possible to be reconciled somehow, but what to do if such windows has to be a little?

As a rule in the application environment everything is organized as it is necessary — the necessary actions are implemented in the form of additional windows which arrangement and the size the user both wants, and configures for itself. It is possible to assume that API has to contain this functionality of integration of the user windows on Wednesday, but strangely enough, most likely such functionality will be absent. But, if it is very necessary, it can be implemented.

Read more »


AeroState — monitoring and the forecast of quality of air in Moscow (and not only)

1 year, 3 months ago
Hi, Habr!

This post — about quality of air which we breathe. It is considered to be that in general, air of big cities is unhealthy. It also is clear — here to you both a traffic and plants and you never know what else. Generally, all this keeps residents of the megalopolis in permanent concern about "an adverse ecological situation".

image
Picture from here

However, however, slightly more difficult.

Read more »


MultiCAD.NET API: a block insert with attributes

1 year, 4 months ago
In today's publication it will be a question of an insert of blocks with use of MultiCAD.NET API, it is one of most frequently asked questions in the programmer section of our forum.

So, there is a file of the drawing which already contains blocks templates for schematic symbols of elements of the drawing, for example, of the scheme of water supply. Let's say blocks for representation of different types of cranes contain two attributes:
  • NAME – the name of an element (for example, "Sharovy"),
  • LABEL – marking of an element (for example, "11b27p (M)"),
also have the following graphical representation:

image

Read more »


JS-DOS API: we start DOS in the browser

1 year, 4 months ago
More than 5 years there is an emscripten project. During existence of the project huge work was done. It became much more difficult to surprise the experienced reader. We already saw DOOM, Dune 2, TTD, С&C; and a lot of things in the browser. However, it is still difficult to start the DOS program in the browser: it is necessary to understand not badly C/C ++ and emscripten. Experiments with Dosbox developed into the em-dosbox project, the majority of DOS programs became available to the browser. To put the end, it was necessary only to create open API for start of DOS programs in the browser.

Read more »


Find the corrupt official. Data analysis of officials from projects of Clerical one hundred (with examples on R)

1 year, 4 months ago
How to define officials, the most suspicious from the point of view of corruption? Simplest — having compared their income and standard of living.

I want to show possibilities of the websites with open information on officials in this article, to look at how these officials live and to try to define those who are most suspicious from the point of view of corruption.

Why open information on the income of officials is important? Because it allows to control them.

image
Photo from an Instagram of the daughter of the former head of GAI of Ukraine Alexander Yershov. On a photo the daughter of Yershov in Cannes near Paris Hilton. As a result of scandal because of mismatch of the declared income and a way of life of a family Yershov resigned.

Read more »


Creation of a plug-in for Intellij IDEA. Notes and small councils

1 year, 4 months ago
Half a year ago or about that I lit up ideas to write the plug-in for Intellij IDEA. According to an idea, he had to consider, what is the time the developer spent for the project (everything, in a day, for a session) and to display result on the chart. Any magic, but such function fine would help me to calculate operating time.


Read more »


GDG DevFest Nizhny Novgorod 2015

1 year, 4 months ago
Hi, Habrakhabr! The DevFest conference takes place several years in a row in the different cities of Russia and collects a large number of participants: developers, IT specialists, students of IT faculties, teachers, all to whom technologies of Google are interesting.



This year we already carried out DevFest in Voronezh (the report will be published soon), and the November meeting of fans of technologies of Google will take place in Nizhny Novgorod. For the IT sphere of this city of DevFest will become a unique event. First, we gather in Nizhny Novgorod for the first time, and secondly, among speakers not only local speakers, but also authoritative experts from other cities.

Read more »