Developers Club geek daily blog

Python Meetup 25.09.2015: monitoring of performance and use of BDD

3 years ago
Hi everyone!
With a small delay we publish video from September Python Meetup.
This time, within reports we discussed:
  • Monitoring of performance of a web application on Python with use of pinba + graphite + tessera
  • ispolzovany BDD in development on Python

Also participants of community discussed several actual Python'ovskikh of subjects within Lightning Talk.
You will find records of all performances under a cat.


Read more »


PyTest

3 years ago

Preface


On historical calling I SQL щик. However the destiny brought me on BigData and after that the curve incurred — I mastered both Java, and Python, and functional programming (studying of Scala is in the list). Actually on one of pieces of the project there was a need of testing of a code for Python. Children from QA advised for these purposes of PyTest, but even they found it difficult to answer plainly with what this animal is good. Unfortunately, in a Russian-speaking segment of information on the matter not so there is a lot of: as it is used in Yandex and all in an amicable way. At the same time described in it to article travel on this path looks rather difficult for the person of a beginner. Not to mention official documentation — it purchased for me sense only after I understood with the module on other sources. I do not argue, interesting things, but, unfortunately, at all not for start are written there.

Python unit testing


That and for what to tell sense I do not see it — Wikipedia all the same knows more. Concerning the existing modules for Python it is well described on Habré.

Input on necessary knowledge


For the described moment of knowledge of Python at me were rather superficial — I wrote some simple modules and knew standard things. But at collision with PyTest I had to fill up knowledge base with decorators here and here and construction of yield.

Benefits and shortcomings of PyTest


1) Independence of API (no boilerplate). As the code looks in the same unittest:

Code
import unittest

class TestUtilDate(unittest.TestCase):
    def setUp(self):
        #init_something()
        pass
        
    def tearDown(self):
        #teardown_something()
        pass
        
    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')
        
    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        
    def test_failed_upper(self):
        self.assertEqual('foo'.upper(), 'FOo')
        
if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestUtilDate)
    unittest.TextTestRunner(verbosity=2).run(suite)


The same in PyTest:

Code
import pytest

def setup_module(module):
    #init_something()
    pass

def teardown_module(module):
    #teardown_something()
    pass

def test_upper():
    assert 'foo'.upper() == 'FOO'
    
def test_isupper():
    assert 'FOO'.isupper()
    
def test_failed_upper():
    assert 'foo'.upper() == 'FOo'


2) Detailed report. Including unloading in JUnitXML (for integration with Jenkins). The type of the report can change (including colors) additional modules (about them will be later separately). Well and in general the color report in the console looks more convenient — red FAILED are visible at once.

image

3) Convenient asset (standard of Python). It is not necessary to hold all heap of different assert'ov in the head.

4) Dynamic fikstura of all levels which can be caused as automatically, and for specific tests.

5) Additional opportunities fikstur (a returned value, finalizers, area of visibility, object of request, autouse, the enclosed fikstura)

6) Parametrization of tests, that is start of the same test with different sets of parameters. In general it belongs to the point 5 "Additional Opportunities Fikstur", but an opportunity is so good that it is worthy separate point.

7) Tags (marks) allowing to pass any test, to mark the test as falling (and this his expected behavior that is useful when developing) or just to name test set that it was possible to start only it by name.

8) Plug-ins. This module has rather big list of additional modules which can be installed separately.

9) A possibility of start of the tests written on unittest and nose, that is complete backward compatibility with them.

About shortcomings, let and there is not a lot of them, I can tell the following:

1) Lack of the additional nesting level: For modules, classes, methods, functions in tests there is an appropriate level. But the logic demands presence of the additional testcase level when the same one function can have several testcase'ov (for example, check of returned values and errors). It is partially compensated by the additional module (plug-in) pytest-describe, but there is a problem of lack of an appropriate level of a fikstura (scope = "describe"). With it of course it is possible to live, but in some situations can break the main principle of PyTest — "all for simplicity and convenience".

2) Need of separate installation of the module, including in the prodakshena. After all unittest and doctest enter into the basic Python tools and do not demand additional gestures.

3) Use of PyTest requires a little more knowledge of Python, than for the same unittest (see. "Input on necessary knowledge").

The detailed description of the module and its opportunities under a cat.

Read more »


10 applications for studying of Python on Android-devices

3 years ago
In the course of studying of language we usually use the PC for work with the corresponding applications, Wednesdays, programs, we read books, we use courses mass online. Now, when smartphones with us always and everywhere, sin not to use their opportunities for assimilation of knowledge of programming or their improvement.
image
In the course of studying of programming language, in particular Python, I consider that is necessary to plunge into it completely. Slogan: "Day without code!" I supplement, well if to kodit then to esteem about it. Not to leave rhythm in conditions when it is necessary to make a lot of things in life, but there is no time, and Python would like to learn, mobile applications will help not to spend in vain time in transport, during expectations, etc. I offer the list of applications from Google Play for studying of Python on Android-devices which will help not only to gain knowledge, but also to check the level on Python.Tri from below the listed applications I surely advise students when studying Python: Learn Python, Quiz&Learn; Python and Python Challenge. Well now

Read more »


Celery: the best practicians

3 years ago
If you work with Django, at some stage to you background processing of long running tasks can be necessary for development. It is possible that for such tasks you use any tool for management of queues of tasks. Celery — one of the most popular projects for solution of similar tasks in the world of python and Django at the moment, but is also other projects for this purpose.

While I worked on some projects using Celery for management of queues of tasks some best practicians which I have decided to document have come to light. However it is big words for this purpose that I think of the correct approach to solution of similar tasks, and also of some not enough used opportunities which are offered by community of the Celery project.

Read more »


Creation of .exe of the file for authorization and interaction with VK API by means of Python

3 years ago
In this article the method will be considered to create .exe the file allowing to pass authorization of VK and to execute certain requests to  VK API. The Python installation is not required.

System requirements


Read more »


PYCON RUSSIA-2015: video and presentations of reports

3 years ago
On September 18-19 in Yekaterinburg there has taken place the third international python-developers PyCon Russia conference. 21 reports, two master classes, Lightning Talks — all this PyCon-2015. Under cat — the report on last conference, is a lot of video and presentation of reports.

0_1217f0_23d019c7_orig.jpg
Pythonists have gathered from 25 cities, among which: San Francisco, Paris, Prague, Bonn, Tallinn, Minsk, Kiev, Moscow, St. Petersburg, Kazan, Ufa

Read more »


Service for the system administrator. Part 1

3 years ago
image

1. Preface


Tasks of the system administrator — are various. Work in the console — creation of users, testing, installation and configuration of packets on servers, viewing of logs and traffic, setup of network and tunnels. Work with iron — installation of the equipment and documentation, writing of plans of works, the description of work of services.

System administrators have large number of scripts for automation. They are stored, usually, in their house folders (and it is good if not only there, but also in CVS), apdeytitsya periodically on new functionality. As a rule, only the same people who write them can use such scripts (and, otherwise, effects can be the most destructive).

Thus — tasks of automation often need simple GUI for convenient management, simplification. For example — collecting traffic. Or opportunity to roll away bekapy / releases by the button (even if someone slides packets through SCM system). Or to change Mysql master without peeping the console (some quantity of errors arises because of incorrectly hammered into teams, not on that server).

Read more »


Big data, Beeline and kokoko

3 years ago
Couple of days ago, having accidentally come on Habr without adblok, I have seen banner:" Beeline, be the man — solve the task shaitan". Chelendzh sounded interestingly, to determine age by set of such parameters, as the region, tariff plan, etc.


Read more »


Remote execution of system level commands on demand through sockets on Python 3. Part 2. Data transfer protocol

3 years ago
In the previous article I have told as to create the server and the client on Python 3, using the built-in sockets. But this application had many shortcomings which I will try to correct in this and the subsequent articles.

So what shortcomings our application possesses?
  • The unique packet which length cannot exceed the preset border in 1 Kb is sent.
  • Application without check transfers the argument accepted from network to cover (URL).
  • Lack of functionality. We cannot download, for example, all images from Habr, or download separate hub.

Today I will tell as to solve the first problem, and at the same time and it is a little about TCP.

Description of the protocol

We used the "naked" TCP protocol for data transmission between the server and the client. TCP — is the stream protocol, he transfers data consecutive set of bytes. Transferring team with arguments on network in the first version of our application we read only 1024 bytes of data from the accepted packet. But what to do if data are not located in 1024 bytes? Output only one — to break data into some packets on one host and "to stick together" them in one piece when receiving on other host. But how to learn when one team comes to an end (with her arguments) and another begins? For this purpose it is necessary for us it is necessary to know, what length of all transferred message.

As in advance to learn message length from us not to turn out, it is necessary to transfer it in one of packets. Certainly, it is better to do it at the very beginning of the first packet. Having selected only 4 bytes under storage of length of the message, we will be able to transfer the message over 4 billion characters long! Message length — is information on him, that is, part of heading, heading of our protocol. What protocol you ask? If to trust Wikipedia,
The data transfer protocol — set of agreements of the interface of the logic level which define data exchange between different programs.

We have agreed that we will transfer data in several packets on TCP, and at the beginning of data of the first packet length of all message will be stored in bytes. Thus we have developed our simple protocol! It is necessary to remember that our protocol is based on TCP, so, the last possesses the same features, as.

Read more »


Remote execution of system level commands on demand through sockets on Python 3 or as I downloaded the sites

3 years ago
The project has been written rather in the educational purposes (to learn network programming in Python), than in the practical. Bears the same role and to be become, after all now hardly someone will download the sites to read couple of articles (except for some cases when similar really can it is useful).

Not so long ago quality of the mobile Internet in my city began to worsen gradually because of the loading increasing on network of operators and some sites demanding large number of connections (dependent files of the page) began to be loaded well OCHEN slowly. In the evenings speed falls on so many that some sites can be loaded completely during several tens seconds.

There are some ways of solution of this problem, but I have decided to select way, a little unusual to our time. I have decided to download the sites. Of course, data the way is not suitable for the large sites, like Habr, it is more reasonable to use the parser here, but it is possible to download also separate hub, the list of users, or only the publications by means of HTTrack Website Copier, having used filters. For example, to download Python hub from Habr it is necessary to use the filter "+habrahabr.ru/hub/python / *".

This way can be used in several purposes. For example, to download the site, or its part before you appear without Internet connection, for example, in the airplane. Or to download the sites blocked on the territory of the Russian Federation if to download them through Tor that will be very slowly, or via the computer in other country where data the site is not forbidden, and then to tell it to the computer which is in the Russian Federation that will be much quicker for the multipage sites. Thus we can download, for example, xHamster Wikipedia via the server in Germany or the Netherlands and to receive the site in summary form on SFTP, FTP, HTTP or another, convenient for you, to the protocol. If, of course, the place is enough, for such big site :)

Well, we will begin!?

Read more »