Developers Club geek daily blog

1 year ago
I submit the second article from a series, the "continuing" system administrators oriented on, for experienced I will hardly open something new.
In these articles we will consider creation of the Internet gateway on linux allowing to connect several offices of the company and to provide limited access in a network, a prioritetzation of a traffic (QoS) and simple balancing of loading with reservation of the channel between two providers.
Specifically in this part:
  • More detailed Shorewall setup
  • Terrible and not clear QoS
  • Balancing of loading and reservation


And in the previous part were considered:
  • The elementary Shorewall setup
  • Awfully difficult dnsmasq setup
  • Not less difficult OpenVPN setup
  • And for many continuing administrators atypical, dynamic routing, on the example of OSPF


Everything described below is fair for CentOS 7.1 (above, the 6th series will approach too, but with small features)

Profound Shorewall setup



Last time we configured quite trustful and primitive operation mode, now we will include a little sensible paranoia.
It is not difficult to make it, let's make changes in the file:
policy
#
# Shorewall -- /etc/shorewall/policy
#
# For information about entries in this file, type "man shorewall-policy"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-policy.html
#
###############################################################################
#SOURCE	DEST	POLICY		LOG	LIMIT:		CONNLIMIT:
#				LEVEL	BURST		MASK
$FW	all	REJECT
grn	all	REJECT
tun	all	REJECT
red	all	DROP


The new configuration became simpler, but to rejoice early, the rules file considerably will swell:
/ etc/shorewall/rules
#
# Shorewall -- /etc/shorewall/rules
#
# For information on the settings in this file, type "man shorewall-rules"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-rules.html
#
######################################################################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE		ORIGINAL	RATE		USER/	MARK	CONNLIMIT	TIME		HEADERS		SWITCH		HELPER
#							PORT(S)	PORT(S)		DEST		LIMIT		GROUP
?SECTION ALL
?SECTION ESTABLISHED
?SECTION RELATED
?SECTION INVALID
?SECTION UNTRACKED
?SECTION NEW

INCLUDE rules.fw
INCLUDE rules.grn
INCLUDE rules.red
INCLUDE rules.red-dnat
INCLUDE rules.tun


Also we will use the directive INCLUDE for diversity of rules according to several files:
rules.fw
#
# Shorewall -- /etc/shorewall/rules.fw
#
# For information on the settings in this file, type "man shorewall-rules"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-rules.html
#
######################################################################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE		ORIGINAL	RATE		USER/	MARK	CONNLIMIT	TIME		HEADERS		SWITCH		HELPER
#							PORT(S)	PORT(S)		DEST		LIMIT		GROUP
DNS(ACCEPT)	$FW		red
Web(ACCEPT)	$FW		red
FTP(ACCEPT)	$FW		red
OpenVPN(ACCEPT)	$FW	red
Ping(ACCEPT)	$FW	all
OSPF(ACCEPT)	$FW	tun
SSH(ACCEPT)	$FW		all



rules.grn
#
# Shorewall -- /etc/shorewall/rules.grn
#
# For information on the settings in this file, type "man shorewall-rules"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-rules.html
#
######################################################################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE		ORIGINAL	RATE		USER/	MARK	CONNLIMIT	TIME		HEADERS		SWITCH		HELPER
#							PORT(S)	PORT(S)		DEST		LIMIT		GROUP
DNS(ACCEPT)	grn		$FW
Web(ACCEPT)	grn		red
FTP(ACCEPT)	grn		red
Ping(ACCEPT)	grn	all
SSH(ACCEPT)	grn		all	-	-	-	-	s:3/min



rules.red
#
# Shorewall -- /etc/shorewall/rules.red
#
# For information on the settings in this file, type "man shorewall-rules"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-rules.html
#
######################################################################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE		ORIGINAL	RATE		USER/	MARK	CONNLIMIT	TIME		HEADERS		SWITCH		HELPER
#							PORT(S)	PORT(S)		DEST		LIMIT		GROUP
OpenVPN(ACCEPT)	red	$FW
SSH(ACCEPT)	red		$FW	-	-	-	-	s:3/min



rules.tun
#
# Shorewall -- /etc/shorewall/rules.tun
#
# For information on the settings in this file, type "man shorewall-rules"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-rules.html
#
######################################################################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE		ORIGINAL	RATE		USER/	MARK	CONNLIMIT	TIME		HEADERS		SWITCH		HELPER
#							PORT(S)	PORT(S)		DEST		LIMIT		GROUP
Ping(ACCEPT)	tun	$FW
OSPF(ACCEPT)	tun $FW
SSH(ACCEPT)	tun		$FW	-	-	-	-	s:3/min
SSH(ACCEPT)	tun		grn	-	-	-	-	s:3/min



Now all traffic under our control, we start up only those protocols that we want. Pay attention to rules for SSH, we limited the frequency of connections to the 3rd in a minute, from each ip of a source. But it is necessary to be very careful with this parameter, having incorrectly specified s key: or d: can make your service easily giving in DDoS to attack. And for traffic Web (and in general any), it is necessary to consider that many potential clients can sit at NAT, and then one IP, a source of connections, can generate considerable number of connections.
If we need to forward ports to internal servers, it is not difficult to make it:
rules.red-dnat
#
# Shorewall -- /etc/shorewall/rules.red-dnat
#
# For information on the settings in this file, type "man shorewall-rules"
#
# The manpage is also online at
# http://www.shorewall.net/manpages/shorewall-rules.html
#
######################################################################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE		ORIGINAL	RATE		USER/	MARK	CONNLIMIT	TIME		HEADERS		SWITCH		HELPER
#							PORT(S)	PORT(S)		DEST		LIMIT		GROUP
Web(DNAT)	red	grn:172.16.0.2
#Вариант, если у нас несколько внешних адресов, а мы хотим DNAT только с одного из них:
#Web(DNAT)	red	grn:172.16.0.2	-	-	-	192.168.10.37
#Варианты, все руками:
#DNAT		red			grn:172.16.0.2	tcp	80,443
#DNAT		red			grn:172.16.0.2	tcp	80,443	-	192.168.10.37
#Вариант с нестандартными портами:
#DNAT		red			grn:172.16.0.2:80	tcp	8080



Simple balancing of loading and reservation in Shorewall



I will tell at once, both balancing and reservation, are only configured by means of Shorewall, it does not provide any functionality on detection of loss of the channel or uniform distributions of real loading.
All setup is reduced to editing one file:
providers
#
# Shorewall -- /etc/shorewall/providers
#
# For information about entries in this file, type "man shorewall-providers"
#
# For additional information, see http://shorewall.net/MultiISP.html
#
############################################################################################
#NAME	NUMBER	MARK	DUPLICATE	INTERFACE	GATEWAY		OPTIONS		COPY
pr1	1	0x10000	-		$IF_RED1	$GW_RED1	fallback=1
pr2	2	0x20000	-		$IF_RED2	$GW_RED2	fallback=4


If who remembers, in last article there was a section from "Shorewall.conf". It is necessary just for setup of marking of packets and work with providers. There we set what bits from a packet tag, set the identifier of provider, what just tags (I such so far do not use anywhere, but Shorewall involves them, for the needs), and what for tracking of connections.
Here we described two providers, set how to mark packets for them on what interface everyone sits and what gateway at whom.
There is an OPTIONS column, it is necessary to explain a little. Here the key of fallback forces to generate additional rules of routing, on a case if balancing are not able to process a packet (the interface lay down for example), and the tsiferka sets interface weight. The weight is more, the more often new connections will come to the interface (relative weight at whom in a proportion weight is higher, is used as a result there and the traffic is more often, it is not recommended to set great value of weight). Balancing happens by the principle of roundrobin already kernel and the client-server is based on the connection establishment fact on route (as the client your router, and the server respectively, a remote server). At the same time routes are cached for some time, and such effect is gained: someone in a lokalka, got for example on a certain website (at which one IP), the traffic went through provider 1, then someone else got there, and the traffic will run through provider 1 again (if the cache was not in time will be reset). Still can it will turn out that you have not symmetric providers as in an example, and so will carry, every fourth connection, getting on the first provider, will be also "heaviest", and we wanted it to avoid … There are no already simple solutions, and this article will not help you in any way.
Having given command:
shorewall show routing

It is possible to see the constructed scheme of routing:
Example of routing
Shorewall 5.0.2.1 Routing at cent1.domain.local - Пт янв  8 23:41:30 MSK 2016


Routing Rules

0:      from all lookup local
999:    from all lookup main
10000:  from all fwmark 0x10000/0xff0000 lookup pr1
10001:  from all fwmark 0x20000/0xff0000 lookup pr2
20000:  from 192.168.10.37 lookup pr1
20000:  from 192.168.10.36 lookup pr2
32765:  from all lookup balance
32767:  from all lookup default

Table balance:


Table default:

default nexthop via 192.168.10.1 dev eth0 weight 1 nexthop via 192.168.10.1 dev eth2 weight 1

Table local:

local 192.168.10.37 dev eth0 proto kernel scope host src 192.168.10.37
local 192.168.10.36 dev eth2 proto kernel scope host src 192.168.10.36
local 172.16.3.1 dev tap0 proto kernel scope host src 172.16.3.1
local 172.16.3.129 dev tap1 proto kernel scope host src 172.16.3.129
local 172.16.248.1 dev lo proto kernel scope host src 172.16.248.1
local 172.16.0.1 dev eth1 proto kernel scope host src 172.16.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 192.168.10.255 dev eth2 proto kernel scope link src 192.168.10.36
broadcast 192.168.10.255 dev eth0 proto kernel scope link src 192.168.10.37
broadcast 192.168.10.0 dev eth2 proto kernel scope link src 192.168.10.36
broadcast 192.168.10.0 dev eth0 proto kernel scope link src 192.168.10.37
broadcast 172.16.3.255 dev tap1 proto kernel scope link src 172.16.3.129
broadcast 172.16.3.128 dev tap1 proto kernel scope link src 172.16.3.129
broadcast 172.16.3.127 dev tap0 proto kernel scope link src 172.16.3.1
broadcast 172.16.3.0 dev tap0 proto kernel scope link src 172.16.3.1
broadcast 172.16.248.1 dev lo proto kernel scope link src 172.16.248.1
broadcast 172.16.1.255 dev eth1 proto kernel scope link src 172.16.0.1
broadcast 172.16.0.0 dev eth1 proto kernel scope link src 172.16.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1

Table main:

192.168.10.1 dev eth2 scope link src 192.168.10.36
172.16.3.1 dev tap0 proto zebra
172.16.3.129 dev tap1 proto zebra
172.16.248.2 via 172.16.3.2 dev tap0 proto zebra metric 13
172.16.12.129 via 172.16.3.2 dev tap0 proto zebra metric 3
172.16.11.1 via 172.16.3.2 dev tap0 proto zebra metric 3
172.16.3.128/25 dev tap1 proto kernel scope link src 172.16.3.129
172.16.3.0/25 dev tap0 proto kernel scope link src 172.16.3.1
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.36 metric 101
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.37 metric 100
172.16.8.0/23 via 172.16.3.2 dev tap0 proto zebra metric 13
172.16.0.0/23 dev eth1 proto kernel scope link src 172.16.0.1 metric 100

Table pr1:

192.168.10.1 dev eth0 scope link src 192.168.10.37
default via 192.168.10.1 dev eth0 src 192.168.10.37

Table pr2:

192.168.10.1 dev eth2 scope link src 192.168.10.36
default via 192.168.10.1 dev eth2 src 192.168.10.36


But, not everything is so bad, and even so it works adequately (by the way, so it works for the majority of solutions, for "fair" balancing very not simple solutions which often without support do not work at the party of providers (both) are used).
The added file of variables:
params
#
# Shorewall -- /etc/shorewall/params
#
# Assign any variables that you need here.
#
# It is suggested that variable names begin with an upper case letter
# to distinguish them from variables used internally within the
# Shorewall programs
#
# Example:
#
#	NET_IF=eth0
#	NET_BCAST=130.252.100.255
#	NET_OPTIONS=routefilter,norfc1918
#
# Example (/etc/shorewall/interfaces record):
#
#	net	$NET_IF		$NET_BCAST	$NET_OPTIONS
#
# The result will be the same as if the record had been written
#
#	net	eth0		130.252.100.255	routefilter,norfc1918
#
###############################################################################
IF_RED1=eth0
GW_RED1=192.168.10.1
IF_RED2=eth2
GW_RED2=detect
IF_GRN=eth1
NET_GRN=172.16.0.0/23
IF_TUN=tap+
#LAST LINE -- DO NOT REMOVE


It is possible to notice that at the second provider the gateway is specified as a key word of "detect" which works at connections with dynamic addressing. For some cases (for example PPtP), Shorewall cannot correctly define the gateway for what the file with an auxiliary script is used:
findgw
#
# Shorewall version 4 - Findgw File
#
# /etc/shorewall/findgw
#
# The code in this file is executed when Shorewall is trying to detect the
# gateway through an interface in /etc/shorewall/providers that has GATEWAY
# specified as 'detect'.
#
# The function should echo the IP address of the gateway if it knows what
# it is; the name of the interface is in $1.
#
# See http://shorewall.net/shorewall_extension_scripts.htm for additional
# information.
#
###############################################################################
LANG='C' nmcli --terse --fields IP6.GATEWAY device show ${1} | cut -f2- -d':' #IPv6
LANG='C' nmcli --terse --fields IP4.GATEWAY device show ${1} | cut -f2- -d':' #IPv4


Also the script for NetworkManager will add all this (simpler version was in last article, and it did not consider that after a raising of the interface, Shorewall not always correctly forms policy of routing therefore we for such interfaces just restart it).
/etc/NetworkManager/dispatcher.d/30-shorewall.sh
#!/bin/bash
 
IF=$1      # имя сетевого интерфейса, с которым связано событие
STATUS=$2  # новое состояние сетевого интерфейса
function check_prov() {
	PARAM=$(grep -v '^#' /etc/shorewall/params | grep $1 | cut -d '=' -f 1)
	if [ -z "$PARAM" ]; then
		grep -v '^#' /etc/shorewall/providers | grep -q $1
		[[ $? == 0 ]] &&shorewall restart
	else
		grep -v '^#' /etc/shorewall/providers | grep -q $PARAM
		[[ $? == 0 ]] &&shorewall restart
	fi
}
case $STATUS in
	up)
		# команды выполняемые после установления соединения
		shorewall enable $IF
		shorewall6 enable $IF
		check_prov $IF
	;;
	down)
		# команды выполняемые после разрыва соединения
		shorewall disable $IF
		shorewall6 disable $IF
		check_prov $IF
	;;
esac


But, whether it is possible to direct any traffic on specific provider? Answer of Da!
mangle
#
# Shorewall -- /etc/shorewall/mangle
#
# For information about entries in this file, type "man shorewall-mangle"
#
# See http://shorewall.net/traffic_shaping.htm for additional information.
# For usage in selecting among multiple ISPs, see
# http://shorewall.net/MultiISP.html
#
# See http://shorewall.net/PacketMarking.html for a detailed description of
# the Netfilter/Shorewall packet marking mechanism.
#
####################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE	USER	TEST	LENGTH	TOS	CONNBYTES	HELPER	PROBABILITY	DSCP
#							PORT(S)	PORT(S)
MARK(0x20000):P	172.16.0.4	0.0.0.0/0!172.16.0.0/12


Here we marked all traffic from 172.16.0.4 in any network, except 172.16.0.0/12, with a tag of the second provider. Conditions can be more cunning, it is only necessary to consider, for the traffic generated on our gateway, from the rule it is necessary to clean ":P".

Terrible and awful QoS



At once it is necessary to explain here that: on the present we can regulate speed only when sending. The incoming traffic, already reached our interface passing all bottlenecks, and being pushed in queues, for the right to appear at us on a threshold.
But to despair early, and there are mechanisms, not so graceful and reliable as it would be desirable, but solving this problem. In networks based on protocols of IP family, it is solved thus:
The source smoothly increases sending speed while answers about delivery (ACK packets in the TCP protocol) in general come or come with a normal delay. If there are losses or ACK delay grows, speed decreases. Then, later some period, speed try to lift again. And so occurs to an out-time.
And how UDP? And with it everything is simple, there is no delivery control, there is no headache. Sent also OK (let the receiver suffers).
Of course, in pure form UDP usually do not use in complex challenges of data transmission. This protocol usually use as a basis, at implementation of the option of control of delivery (it is possible to tell, the implementations TCP, in cases when standard does not arrange). Therefore in many protocols of UDP working atop, delivery control is too. What does not cancel an opportunity to send a continuous flow of UDP to all power, hammering a purpose communication channel (that DDoS option).
How we can organize prioritizing of a traffic and selection of bandpass range of the incoming traffic? The answer is above: to create a delay in obtaining on the party (as a result, a delay in generation of ACK) or if the delay is indecently high, to discard a packet.

In Linux it is implemented by creation of the IFB pseudo-interface which as if rises between the physical interface and the gateway, passing through itself the incoming traffic. The traffic enters the physical interface (we already accepted it on it), and right there goes to IFB which already regulates with what speed, and in what order to pass this traffic (or in general to discard).
With setup we will be helped by Shorewall (though it is possible and in / to prescribe etc/modprobe.d):
/ etc/shorewall/init
#
# Shorewall -- /etc/shorewall/init
#
# Add commands below that you want to be executed at the beginning of
# a "shorewall start", "shorewall-reload" or "shorewall restart" command.
#
# For additional information, see
# http://shorewall.net/shorewall_extension_scripts.htm
#
###############################################################################
modprobe ifb numifbs=3
ip link set ifb0 up
ip link set ifb1 up
ip link set ifb2 up


Here it is trivial, created three IFB pseudo-interfaces and they were lifted.
Further we will describe interfaces on which we will regulate a traffic:
tcdevices
#
# Shorewall -- /etc/shorewall/tcdevices
#
# For information about entries in this file, type "man shorewall-tcdevices"
#
# See http://shorewall.net/traffic_shaping.htm for additional information.
#
###############################################################################
#NUMBER:	IN-BANDWITH	OUT-BANDWIDTH	OPTIONS		REDIRECTED
#INTERFACE							INTERFACES
1:$NET_GRN	-		1000mbit	hfsc,classify
2:ifb1		-		1000mbit	hfsc,classify	$NET_GRN
3:$NET_RED1	-		10mbit		hfsc,classify
4:ifb0		-		10mbit		hfsc,classify	$NET_RED1
5:$NET_RED2	-		10mbit		hfsc,classify
6:ifb2		-		10mbit		hfsc,classify	$NET_RED2


Here we in an explicit form, set numbers of the used interfaces (if not to set, Shorewall will number them as the declaration the file), associated IFB with real interfaces, set the maximum outgoing speed (we remember, only we also regulate it, and the ifb interface in fact is an incoming line) and set disciplines of classification and that we will classify a traffic exactly.
Let's describe those classes:
tcclasses
#
# Shorewall -- /etc/shorewall/tcclasses
#
# For information about entries in this file, type "man shorewall-tcclasses"
#
# See http://shorewall.net/traffic_shaping.htm for additional information.
#
###############################################################################
#INTERFACE:CLASS	MARK	RATE:		CEIL	PRIORITY	OPTIONS
#				DMAX:UMAX
1:1:2			-	1mbit		3mbit	2		default
1:1:3			-	256kbit		full	1
2:1:2			-	1mbit		3mbit	2		default
2:1:3			-	256kbit		full	1
3:1:2			-	1mbit		3mbit	2		default
4:1:3			-	256kbit		full	1
5:1:2			-	1mbit		3mbit	2		default
5:1:3			-	256kbit		full	1
6:1:2			-	1mbit		3mbit	2		default
6:1:3			-	256kbit		full	1


Yet we will not do anything difficult (so interesting and useful) until we tie to each interface (including IFB) two classes.
In the first column we associate the interface with classes.<номер интерфейса><номер родительского класса>::<номер описываемого класса>.
On the interface there is always a class 1 which we in fact, described in tcdevices.
Further, we did not mark packets (at classification of it and it is impossible to make) and therefore we will not use a column, there is the most interesting, minimum guaranteed bandpass range further, and greatest possible (than that is no more at the parent's class), for this class. The priority will set an order of permission of a disputable situation (at whom less, that and will go the first if went beyond side-altars of the guaranteed band, and she already another is busy by someone completely). In the conclusion there are options, default says that if nothing is found filters (packets are not carried to classes), then to appropriate them a class by default).
Further, so historically it developed that corrected classifications of real interfaces, are in the file:
mangle
#
# Shorewall -- /etc/shorewall/mangle
#
# For information about entries in this file, type "man shorewall-mangle"
#
# See http://shorewall.net/traffic_shaping.htm for additional information.
# For usage in selecting among multiple ISPs, see
# http://shorewall.net/MultiISP.html
#
# See http://shorewall.net/PacketMarking.html for a detailed description of
# the Netfilter/Shorewall packet marking mechanism.
#
####################################################################################################################################################
#ACTION		SOURCE		DEST		PROTO	DEST	SOURCE	USER	TEST	LENGTH	TOS	CONNBYTES	HELPER	PROBABILITY	DSCP
#							PORT(S)	PORT(S)
CLASSIFY(1:3)	0.0.0.0/0	0.0.0.0/0	tcp	-	80,443
CLASSIFY(3:3)	0.0.0.0/0	0.0.0.0/0	tcp	80,443


And for virtual IFB in:
tcfilters
#
# Shorewall -- /etc/shorewall/tcfilters
#
# For information about entries in this file, type "man shorewall-tcfilters"
#
# See http://shorewall.net/traffic_shaping.htm for additional information.
#
########################################################################################################
#INTERFACE:	SOURCE		DEST		PROTO	DEST	SOURCE	TOS		LENGTH	PRIORITY
#CLASS							PORT(S)	PORT(S)
2:3		0.0.0.0/0	0.0.0.0/0	tcp	-	80,443
4:3		0.0.0.0/0	0.0.0.0/0	tcp	80,443


In an example above, I placed the incoming traffic from HTTP(S) of the server in a class with number 3 on the physical interface, and on virtual, associated with it, and outgoing I made similarly, but "having unrolled ports". Very much show consideration for the fact that connections are often bidirectional, and it is necessary to paint their classification separately for each direction, it is not dependent on the one who initiated the client or the server.
Exactly here, the roof also begins to crack. For understanding the picture will help (I ask not to beat especially with legs, in Visio I am not able to work with sense).
Program Internet gateway for not the small company (Shorewall, OpenVPN, OSPF). Part 2
It is possible to cut a traffic as a result on different sections, without applying IFB if you have one provider, and the gateway, a traffic does not accept (is not his receiver). If it is more providers, and the gateway itself actively accepts a traffic (it for example services VPN), then without IFB it is not easy to get out.

P.S.
In the following article I plan will dwell upon QoS, especially in the light of distribution of VoIP of technologies. The subject is big, and it is necessary to plan everything properly. If you are interested in a certain aspect in more detail, write request to comments, I will consider wishes in the following article.
Whether these articles are informative? (The motivating poll)

19 people voted. 13 people refrained.

The users only registered can participate in poll. Enter, please.


This article is a translation of the original post at habrahabr.ru/post/274677/
If you have any questions regarding the material covered in the article above, please, contact the original author of the post.
If you have any complaints about this article or you want this article to be deleted, please, drop an email here: sysmagazine.com@gmail.com.

We believe that the knowledge, which is available at the most popular Russian IT blog habrahabr.ru, should be accessed by everyone, even though it is poorly translated.
Shared knowledge makes the world better.
Best wishes.

comments powered by Disqus