Developers Club geek daily blog

Use of the LSM engine from SQLite 4 as separate NoSQL of the database with use of Python

1 year ago
image

Today two of my most favourite subjects — SQLite and key-value of the database. And this time I write about both at once: this post is devoted to a Python-wrapper for the storage used in SQLite 4 key-value on the basis of LSM.

I not too attentively trace SQLite releases, but version 3.8.11 drew my attention as in its description serious increase in performance in comparison with 3.8.0 is declared. In accompanying information I came across mentioning of new experimental expansion for full-text search (about which wrote once) and therefore it became interesting to me what develops a situation with SQLite 4.

Having studied available information, I paid attention that one of tasks of developers was to provide in new versions the interface for the connected engines of databases. At the time of writing of this post in SQLite 4 there were already two built-in backends, one of which — key-value storage on the basis of LSM. Few months I happened to be played the last with Cython while I wrote a Python-wrapper for the built-in k-v of UnQLite and Vedis storages. And I thought that it would be quite good to apply Cython to creation of the interface of the DB engine on the basis of LSM used in SQLite 4.

Having dealt with the source code SQLite 4 and the tiny heading LSM file, I wrote python-lsm-db (documentation).

Read more »


We invite to Tarantool meetup on January 28

1 year ago


On January 28, 2016 in the Moscow office of Mail.Ru Group there will take place the second meeting of Tarantool meetup. If someone else does not know: Tarantool is the NoSQL In-Memory DBMS open source which is created for ensuring the greatest possible performance. On the second mitap we will consider the main benefits and features of Tarantool, we will tell about the experience of use of this product and plans for the future. First of all this meeting will be interesting to the developers, Unix-system administrators and other specialists anyway working with databases. You watch the program of a meeting under a cat.

Read more »


We study the graph - the oriented Neo4j DBMS on the example of the lexical Wordnet base

1 year, 1 month ago
Neo4j DBMS is NoSQL the database oriented to storage of graphs. A highlight of a product is declarative language of requests Cypher.

Cypher borrowed a key word like WHERE, ORDER BY from SQL; syntax from such different languages as Python, Haskell, SPARQL; and as a result there was a language allowing to make requests to graphs in a visual form like ASCII art. For example, I would present heading of this article in the form of the graph (Neo4j) — [we study]-> (Wordnet). And it is almost ready database request!


Read more »


Postgres NoSQL is better, than MongoDB?

1 year, 1 month ago
In general, relational database management systems were conceived as "one - the size - is suitable - all a solution for storage and data acquisition" for decades. But the growing need for scalability and new requirements of applications created new problems for traditional management systems of RSUBD, including some dissatisfaction with approach "one - the size - approaches - all" in a number of scalable applications.

The new generation of the lightweight, high-performance databases created to throw down a challenge to domination of relational databases was the answer to it.

Read more »


Tarantool as application server

1 year, 1 month ago
Hi, %khabrayuzer of %. The command of the Tarantula continues to share insayta and examination for effective work with data in the high-loaded projects. Today we will try to understand why Tarantool is "two in one": not only database, but also application server. Probably, some heard about the Tarantula as about superfast persistent in-memory storage with support of replication and hranimok on Lua. Provide that we take Redis pieces, we add the frozen Node.js, on top we fill Go then we cook, slowly mixing, within five minutes after boiling. It would seem at what here Application Server?


Read more »


Under Redis cowl: Hash table (part 2) and List

1 year, 1 month ago
In the first part I told that a hash the table is a few LIST, SET and SORTED SET. You judge — LIST consists of ziplist/linkedlist, SET consists of dict/intset, and SORTED SET is ziplist/skiplist. We already considered the dictionary (dict), and in the second part of article we will consider structure of ziplist — the second most often applicable structure under Redis cowl. Let's look at LIST — the second part of its "kitchen" is simple implementation of the chained list. It is useful to us attentively to consider often mentioned council about optimization a hash of tables through their replacement by lists. Let's consider how many it is required to memory on overhead costs when using these structures what price you pay for economy of memory. Let's sum up the results during the work about a hash as tables, when using the coding in ziplist.

Last time we finished that ziplist of 1,000,000 keys saved with use occupied 16 MB of random access memory whereas in dict the same data demanded 104 MB (ziplist 6 times less!). Let's understand what price:

Read more »


Under Redis cowl: Hash table (part 1)

1 year, 1 month ago
If you know why after execution of 'hset mySey foo bar' we will spend not less than 296 bytes of random access memory why engineers of instagramm do not use line keys why it is always worth changing hash-max-ziplist-entries/hash-max-ziplist-val and why the data type which is the cornerstone of hash it and part of list, sorted set, set — do not read. For the others I will try to tell about it. The understanding of the device and work a hash of tables in Redis is crucial when writing systems where the economy of memory is important.

About what this article — what expenses incurs Redis on storages of the key that such ziplist and dict when and for what they are used how many borrow in memory. When hash is stored in ziplist when in dicth and that it gives us. What councils from fashionable articles about optimization of Redis you should not perceive seriously and why.

Read more »


Under Redis cowl: Lines

1 year, 2 months ago
If you know why the simple line 'strings' in Redis will borrow in random access memory of 56 bytes — to you, I think, article will not be interesting. I will try to tell all rest what is lines in Redis and why it is important to the developer using this database to understand how they are arranged and work. This knowledge is especially important if you try to calculate the actual consumption of memory your application or are going to build highly loaded systems of statistics or accounting of data. Or, as often happens, you try to understand urgently why suddenly your copy redis began to consume unexpectedly a lot of memory.

Read more »


Asynchronous Python 3.5 and Mongodb

1 year, 2 months ago

It is rather loose translation of article about the main innovations of the asynchronous driver for mongodb used in tornado. The main motive which served for writing of this transfer — the innovations which appeared in this version such as support asyncio, async, await and Python 3.5. Article not, how many transfer of innovations, how many laconic examples of asynchronous work with MongoDB.


Introduction
asyncio
aggregate
Python 3.5
async and await

Read more »


"Stored procedures" in Redis

1 year, 2 months ago
image

Many know about an opportunity to store procedures in sql databases, about it many chubby manuals and articles are written. However very few people know that similar opportunities are available also in Redis, starting with version 2.6.0. But as Redis is not a relational DB, and the principles of the description of stored procedures rather strongly differ. Stored procedures in Redis — almost full-fledged Lua scripts (at the time of writing of article as the interpreter Lua 5.1 is used).

Read more »