Developers Club geek daily blog

2 years, 10 months ago
Hi! I with colleagues optimize the available channels. We are constantly confused to optical sealants, to engineers-cablemen, in general, to loaders.

Stories from practice of optimization of a network

And our work is an analysis of a stack of protocols and its complete rearrangement under features of the channel, setup of the optimum sizes of a frame, collecting of several packets for the channel with big latency in one, a deduplication, normal compression, analysis of SSL and reassembly with the same certificate. It is solved in the simplest case installation of special iron on the accepting and transmitting end. As it is necessary to reach each point, we also work as exit engineers. And, as at any exit engineers, stories at us the sea. Below I will tell a little, I will only change a number of minor circumstances that it was impossible to recognize the customer.

For example, there are night works in very large shop. The administrator of the customer and our engineer came per hour nights into server, work. The engineer left in a toilet, returned. In a couple of minutes – knock at a door. Open — and at once GBR with automatic machines, at once legs for the body, a muzzle in a floor and in handcuffs flies.

Then approach police and main on shop. Main assesses a situation and with authority declares:
— I know this, this is our administrator. And I do not know this. Take away.
Came to an end, fortunately, well. The administrator from a floor hisses:
— We wrote you!
Main gets phone, reads:
— There will be night works … I and such … Aaaa, that — turns to police and with explicit pity continues:
— It is necessary to release.

Did not notice how it became good

Mobile optimizers on bank machines in one branch thrust. It is necessary to tell that the Internet was such "good" there that we as a result delivered an image of the virtual computer there on a disk. And so, in this image there was a software agent who works with pieces of iron in this case, in TsODe. Connected, distributed trialny licenses – and test implementation rushed.

There is no back coupling: well, we think, all right, so the optimizer is not necessary to them there. Perhaps the channel was corrected. But! In three months call, want complete. It turned out that the trial ended, and they suddenly noticed how it became bad. What is interesting – as it became good, did not even peep, words. And as it became back bad – users filled up with complaints the manual.

Telephone hooligan

Did test implementation of diagnostic systems of a network. Unrolled a complex in a data-center, it is slowly trained. And it is necessary to tell that the local infrastructure grew in result of gradual merge of the companies therefore not really a gomogenna. In sense, many systems are used, and everyone shows a situation at the level, and on a traffic of applications anybody has no whole picture.

At this moment they just started the new application which well proved to be in a test segment, but in an industrial network under loading did not see it yet. Implemented, but because of difficult structure not all IT specialists knew about it. In 3-4 days after inclusion of service problems with IP telephony began. And the call center was on it. And calls lost in all directions. Client. Tried to solve a problem the means in the afternoon, but could not catch the reason in any way – a bug floating, and is reproduced, is not present.

Administrators knocked to us to ask whether it is possible to use our diagnostic system for the detective. There is "Cascade", he is just able to sort a traffic on the fly. Entered a debug mode and began to look behind network anomalies.

It became clear that new service used the same ports, as at telephony. The new traffic with the same ports began to get to a priority and to force out a voice. After that changed settings of classification of a traffic of telephony.

The similar situation was in one large call center of the operator. The subscriber calls, he hears the operator, and the operator they are is not present. More than a month local network sales managers looked for a solution the means and by forces — it is necessary to tell that so long because the bug was floating, and to detect it was rather nontrivial. As a result business reached the Moscow command, more precisely, to their top. That knocked with something on a table and put a deadline. Addressed us. We brought the same diagnostic system of Riverbed. Collected a traffic from different segments — compared and in a realtayma began to watch quality of a voice on different segments. Within 5-6 hours found the faulty switch at which queues flew. There was a strange problem with iron, settings flew from time to time. The command dealt with a body of the hooligan already – we just changed the switch, and it completely closed a question.

Never be updated on Friday after a lunch

With updating still history was excellent. Too bank, but already another. The administrator updated firmwares of balancers about 12 nights. Approximately in two one of them began to flee silently a traffic which something was not pleasant to it. Not all. Accidental packets. Found it still in a few minutes when the bezopasnik from Khabarovsk suspected something strange with transaction. In 6 minutes classical alarm, "X command, on departure". And the preliminary diagnosis – machine failure in the kernel switch. As usual, the elevator, storekeepers with the new switch, on machines – and there below wait. Fortunately, while went, my colleague who was picked up far off, understood and rolled away firmwares of balancers.

Progress force

Put the optimizer in the training mode on object in bank. The local administrator had to throw it in the fighting mode as he "will accustom" with local flows. Usually from day to one week on profiling because there are still features with bezopasnik. There small reswitching with a vigil therefore, probably, in two months nothing occurred in respect of switching was necessary. The piece of iron received the copy of a traffic and patiently studied. The bank was in procurement process in several stages therefore the chief administrator used this license for other object. Like, here and so well so far, is closer to peak we will put.

And then at them fell (and it is very serious) the land channel (and, maybe, it and did not fall, and the administrator knew that he should be changed – and therefore the optimizer when it is necessary to live without reserve did not cut). They jumped on the satellite. The teleport was directly on a roof, already repeatedly used it therefore, logically, problems should not be. It was supposed to wait on it two weeks until bring a new land line. But during the time which passed from the last long sitting on the satellite, a butt it was a little updated and became more exacting to a band.

In other words, the software began to work badly.

And in a day clients came. Service went slowly, the people were confused who behind whom stands, looped back queues and not bad so became virtualized, occupying at once a little from them. Virtual visitors in bank stood under three hundred, physical – in four it is less time. Approximately time at all this system of queues happened at three o'clock kernel panic, and business threatened to end in a fight.

Urgently requested one more license for sale from us. So we also learned all this history. Further technical progress helped people to solve the problems.

Thanks, guys

Replication of base between Moscow and very far city of Russia. A band – the satellite. The traffic chases compressed base, breakthroughs, non-optimal packets, is often lost. We offered optimizers. With them replications went quicker, queues of requests were not saved, the band was (as usual) utilized much more. But was still where to grow. The main point – very much it was not pleasant to us that the compressed base was replicated. Compressed to sort and accelerate – sense read percent. It was much better to drive crude base that it was sorted by pieces of iron.

The crude base could not be driven for any reason, it did not pass further a firewall. We picked in settings, and in couple of days the crude fighting base began to fly normally. Here standard opportunities of their iron were connected, the processor time (leaving on compression) was released – from 27 hours on replication they began to get in 23. While we with optimizers did 12-14. And here we are told that everything, thanks, men, the task was in SLA to keep within. Already we keep within. Optimizers your abrupt, but now so far will be enough.

So ourselves to ourselves at the runtime the order cancelled this order.

The server for a bottle

We do pilot implementation of the riverbedovsky optimizer of a traffic on one especially large industrial facility. The device is big, a full-fledged rack. Server on object on the second floor, and on the first – directly workshop, the armature sticks out, severe dirty men go. Under server some administrative block, there accounting and control.

And so, there are we through this workshop, and here because of one of machines two directly pronounced representatives of working class taxi. As it is necessary – on the person traces whether paints, whether fuel oil in a form of points, on the heads the scratched helmets. Speak:
— Listen, and it will be possible not to remove you a piece of iron yet?
— In sense? We just behind it.
One hesitated:
— And it how many it costs?
The question, of course, very much guarded. We try to win time:
— We have no right to discuss terms of the contract with you.
The second worker passes to business at once:
— Well approximately? We want to buy it.
Here we slightly okoset. Men not absolutely truly interpreted our silence, and decided that everything, it is possible to begin to bargain:
— Well … for the BOX of vodka you will give?

We somehow beat off them, explained that the optimizer costs not one ten thousand dollars and very quickly reached server. There also began to find out that it was.

It turned out that it is not just hard workers, and operators automated workplaces who are responsible, in particular, for the reporting. And they have such vital periods when they come to this administrative block to work. There costs them to ERP where it is necessary to drive in data. DSL channel, and from those that not really far left from ISDN. Thin, noisy as the empty goods train, gives heritage of the 90th, generally – factory classics. And the application modern, "talkative", drives a heap of the encoded traffic. Therefore logs between click on the drop-down list and its demonstration – seconds 10-20. At best. And to drive in there oho-go how much. The situation is a little aggravated by the fact that they knock one finger and often are mistaken therefore it is necessary to come back. What does not add special pleasure because of the same logs.

And so, they learned that that wonderful piece of iron which with a pump was dragged through all plant on the second floor is the optimizer of a traffic. In general, they would understand nothing if they did not stand behind the back of those who tested this device in the next office where the same riverbed was connected on the pilot. At this moment they realized everything at once and asked for couple of days for the detailed test there. They were let.

As a result they in 2 last days of April made all necessary report. It very much impressed them because to steam of last years they hung out at plant for May holidays at least to the 6th. Following the results of they decided that it is advisable such piece of iron, maybe, even for the to purchase, having chipped in together from a salary. There was no wish to remain overtime though you burst.

Naturally, they just did not wait for such prices.

Through a couple of hours we leave to tell loaders that everything is ready, it is possible to take away. On an output two of our heroes already wait:
— Men, and give you it you will leave for tests for a month, and? We will give the BOX.

What came to an end in – I do not know. The manual of holding after test implementation purchased several tens iron optimizers. Whether the piece of iron as a result got to our guys or not, I cannot tell, but very much I want to believe that it got. Well or at least set them a software agent which is able to answer the "big" server.


This article is a translation of the original post at
If you have any questions regarding the material covered in the article above, please, contact the original author of the post.
If you have any complaints about this article or you want this article to be deleted, please, drop an email here:

We believe that the knowledge, which is available at the most popular Russian IT blog, should be accessed by everyone, even though it is poorly translated.
Shared knowledge makes the world better.
Best wishes.

comments powered by Disqus