Developers Club geek daily blog

Operation of injections in Hibernate ORM

1 year, 2 months ago
image
The report on this subject was submitted at the ZeroNights 0x05 conference on the section FastTrack. Work was very actual and caused a great interest as recently the problem of operation of HQL injections interested many security-researchers specializing in web safety. Therefore I decided to write article which discloses the additional details allowing to understand better results of work.

The modern applications written in the Java language, as a rule, work with DBMS not directly, and use Java Persistence API (JPA). JPA is API which was added to structure of the Java SE and Java EE platforms, since version 5 Java in order that it was convenient to save Java-objects in the database and to retrieve them from the database. There is a large number of ORM libraries (ORM – Object-Relational Mapping) for JAVA which implement the JPA specification. For today latest version of the specification 2.1.

One of popular ORM libraries — Hibernate ORM. At the moment Hibernate is the RedHat project. Servers of the WildFly and JBoss applications use Hibernate as ORM.

Hibernate ORM uses an object-oriented language of requests of Hibernate Query Language (HQL) for writing of requests to entities of Hibernate which are kept in the database.

Read more »


The first steps in Spring framework

1 year, 2 months ago


In September of this year the group of our developers visited the SpringOne2GX Java-conference which was taking place in Washington. It is one of the largest similar conferences, and we could not pass by. We bring to your attention the story that useful and interesting we took out for ourselves by results of this conference.

Read more »


Spring MVC/Security, REST, Hibernate, Liquibase is started in two lines

1 year, 2 months ago

Modern assembly systems allow to automate completely compile process and an application launch from source codes. By the target machine only JDK is necessary, including all the rest and the collector will be loaded to a raid. It is necessary to construct only correctly build process and on start of two commands to receive, for example, the following: start of the database, execution of SQL of scripts, compilation of Java, Javascript and CSS files, start of the container of servlets. It is implemented by means of Gradle, by HSQLDB, Liquibase, Google closure compile and Gretty. In more detail in article.

Read more »


Java 8 in a parallel. We learn to create subtasks and to control their execution

1 year, 2 months ago
All hi.
We continue the cycle of articles devoted to processing of large volumes of data in a parallel (the beautiful word, a lie?).
In the previous article we got acquainted also the interesting Fork/Join Framework tools allowing to break processing into several parts and to start side-by-side execution of separate tasks. What new is in this article – you ask? I will answer – more informative examples and new mechanisms for high-quality information processing. In parallel :) I will tell you about resource and other features of work in this mode.



I invite all interested under kat:

Read more »


About Kirill Tolkachyov and Alexander Tarasov's report about microservices on jug.msk.ru

1 year, 2 months ago
On November 26, 2015 at a meeting of the Moscow Java User Group passing in office of the KROK company, Kirill Tolkachyov and Alexey Tarasov made the report "MICROSERVICES: fire, water and copper pipes". What there was a speech about what impressions of performance, about all this it is possible to read further.


Read more »


Creation of system of scenes for the game engine

1 year, 2 months ago

Preface


Now I work on own game engine. With use of the minimum quantity of third-party libraries, after implementation of a game cycle (game loop), drawing of a frame, the update function, loading of textures and so forth, main "fittings" of the engine were ready. Time of implementation of one more important component — a scene (scene) came.

Introduction


In article I assume that the engine is already equipped with a game cycle with "callback" - functions. All code will be written on Java, but can be easily postponed for any other language supporting garbage collection. Well, we will start.

Read more »


We write tactical game about digits under Android

1 year, 2 months ago
When I only undertook programming (3 months ago), I quickly understood that it is better at once to begin to be engaged in the projects. It is impossible to be all day long at books or courses, but if you begin to do something special, then easily stay behind development from morning to the morning.

This article — the small tutorial how to make logical game with a bot. Game will look here so:


* I will in detail describe rules once again in the section about AI.

Conditionally I separate readers of article into three groups.
  1. Began to program a few hours ago.
    To you it will be difficult, better previously complete some small course on introduction to Android-development, deal with two-dimensional arrays and interfaces. And then load the project from a gitkhab. Comments and this article will help you to understand as as works.
  2. Already you are able to program, but you cannot call yourself experienced yet.
    It will be interesting to you because you very quickly will be able to make the game. I undertook a dirty job on creation of logic of game and an ui-component, I leave you creative part. You can make other mode of game (2 on 2, online, etc.), to change algorithms of a bot, to create levels, etc.
  3. Experienced.
    To you can be interesting to think over AI — to write it not so easily as it seems at first sight. Also I would be very glad to receive from you notes on a code — is sure, I made not all optimum.

Read more »


The first steps about Java 9 and the Jigsaw project – part one

1 year, 2 months ago
Good morning, Habr!

Since the book "Java. New generation of development" we monitor development of the announced new opportunities of this language integrated under the general name "Project Jigsaw" long ago. Today we offer transfer of article of November 24 instilling sufficient confidence that in the Java 9 Jigsaw version after all will take place.

Read more »


Pagination of lists in Android with RxJava. Part II

1 year, 2 months ago
All good afternoon!
Approximately a month ago I wrote article about the organization of a pagination of lists (RecyclerView) by means of RxJava. What there is a pagination simply? It is an automatic podgruzka of data to the list at its scrolling.
The solution which I presented in that article was quite working, steady against errors in replies to the requests on a podgruzka of data and steady against reorientation of the screen (correct saving of a status).
But thanks to comments habrovchan, to their notes and sentences, I understood that the solution has a number of shortcomings which quite on forces to eliminate.
Many thanks to Matvei Malkov for detailed comments and excellent ideas. Without it refactoring of last solution would not take place.
I ask all become interested under kat.

Read more »


Development of fast mobile applications on Android. Part second

1 year, 2 months ago
In Edison we often face optimization of mobile applications and we want to share material which we consider extremely useful if you solve one of two tasks: a) you want that the application braked less; b) you want to make the beautiful, soft and smooth interface for the mass user.

We bring to your attention the first part of transfer of the article Udi Cohen which we used as a benefit for training of young colleagues of optimization under Android.

(To read the first part)



General councils for work with memory


There are several simple recommendations which I use when writing a code.

  • Transfers are already a subject of hot discussions about performance. There is video in which the extent of memory which is spent by transfers, and discussion of this video and some information which is potentially misleading is discussed. Whether more memory, than normal constants use transfers? Definitely. Whether it is bad? It is not obligatory. If you write library and need a strong tipobezopasnost, it could justify their use in comparison with other solutions, such as @IntDef. If you just have a heap of constants which can be grouped together, use of transfers will be not really wise solution. As usual, there is a compromise which needs to be considered at decision-making.

  • The wrapper is an automatic converting of primitive types to their object representation (for example, int-> Integer). Each primitive type "is wrapped" in object representation, the new object is created (shocks, I know). If we have many such objects, the challenge of a garbage collector will be executed more often. It is easy not to notice quantity of a reversing because it becomes automatically for us at assignment to a primitive object type. As a solution, try to use the corresponding types. If you use primitive types in the application, try to avoid their wrapper without need, real on that. You can use instruments of profiling of memory to find the objects representing primitive types. You can also use Traceview and look for Integer.valueOf (), Long.valueOf () and so forth.

  • HashMap vs ArrayMap/Sparse*Array — also, as well as in a case with wrappers, use of HashMap demands use of objects as keys. If we use the primitive int type in the application, he automatically turns around in Integer at interaction with HashMap, in this case we could use SparseIntArray. In a case when we still use objects as keys, we can use ArrayMap. Both options demand the smaller memory size, than HashMap, but they work in a different way that does them by more effective in consumption of memory at the price of reduction of speed. Both alternatives have a smaller print of memory, than HashMap, but time demanded for extraction of an element or memory allocation is a little higher, than at HashMap. If you have no more than 1000 elements, distinctions in runtime are not essential that does viable by these two option.

  • Understanding of a context — as you saw earlier, it is rather easy to create memory leaks. You, perhaps, will not be surprised, having learned that aktivit are the most frequent reason of memory leaks in Android (!). Their leaks cost also very much as they contain all representations of hierarchy of their UI which in itself can take a lot of place. Be convinced that you understand what occurs in aktivit. If object reference and this object lives in a cache longer, than yours aktivit, without cleaning of this link you receive memory leak.

  • Avoid use of nonstatic internal classes. During creation of a nonstatic internal class and its copy you create the implicit reference to your external class. If the copy of an internal class is necessary for longer time frame, than an external class, the external class will continue to be in memory even if it is not necessary any more. For example, it is created nonstatic a class which inherits AsyncTask in the class Activity, then there is a transition to a new asynchronous task and while it lasts, completion of work of an aktivita. While this asynchronous task lasts, it will save aktivit live. A solution simple — do not do it, declare an internal static class if it is necessary.

Profiling of GPU


New adding in Android Studio 1.4 is profiling of GPU of rendering.
Under the Android window go to the GPU tab, and you will see the diagram showing time which was required to make each photo on the screen:

image

Each panel on the diagram represents one drawn frame, and color represents different process steps.

  • Draw (blue) — represents the View#onDraw method (). This part builds/updates objects of DisplayList which then will be converted into the command OpenGL clear for GPU. High rates can be because of the difficult used representations demanding more than time for creation of their list of display, or many representations became invalid in a short period.

  • Prepare (violet) — was added to Lollipop one more flow to help to draw UI quicker. It is called RenderThread. He is responsible for conversion of lists of display to OpenGL of command and sending to their graphic processor. As soon as it occurs, the flow of UI can pass to processing of the following frame. Time spent by UI flow to transfer all necessary RenderThread resources, is reflected in this stage. If we have long/heavy lists of display, this step can take more time.

  • Process (red) — execution of lists of display for creation of OpenGL of commands. This step can take more time if lists long or difficult because it is necessary to redraw a set of elements. Representation can be redrawn as became invalid or was open after movement of the representation hiding it.

  • Execute (yellow) — sending the OpenGL commands for GPU. This stage can last long as CPU sends the buffer with commands for GPU, expecting to receive back pure buffer for the following frame. The number of buffers is limited and if GPU is too occupied, the processor will wait until the first is released. Therefore if we see high values on this step, it can mean that GPU was occupied with drawing of our interface which can be too difficult to be drawn for shorter time.

More flowers for display of bigger quantity of steps, such as Measure/Layout, Input Handing and others were added to Marshmallow:
image

EDIT 09/29/2015: John Reck, a framework the engineer from Google, added the following information about some of these flowers:
"Exact determination of the word "animation" — all this that is registered with Choreographer as CALLBACK_ANIMATION. It includes Choreographer#postFrameCallback and View#postOnAnimation which are used by view.animate (), ObjectAnimator, Transitions and others … And yes, it the same, as is marked out by a tag "animation" in systrace.

"Misc" is a delay between a time mark of vsync and the current time mark when it was received. If you already saw a log from Choreographer "Missed vsync by blabla ms skipping blabla frames", now it will be provided as "misc". There is a distinction between INTENDED_VSYNC and VSYNC in framestats dump (https://developer.android.com/preview/testing/performance.html#timing-info)."


But before beginning to use this opportunity, we need to include the rendering GPU mode in the menu of the developer:
image

This will allow the tool to use ADB commands to get all the information it needs, and so are we (!), using:

 adb shell dumpsys gfxinfo <PACKAGE_NAME>


we can obtain information and construct the diagram independently. The command will output useful information, such as a number of elements in hierarchies, the size of all lists of display and other. In Marshmallow we will obtain more information.

image

If we have an automated testing of UI of our application, we can make the assembly starting this command after certain actions (scrolling of the list, heavy animation, etc.) and to see whether there are changes in values, such as "Janky Frames", eventually. It can help to define decline of performance after several changes in a code, allowing us to fix a problem until the application entered the market. We can obtain even more exact information when we use a key word of "framestats" as it is shown here.

But it is not the only way to see such diagram!

Apparently in the menu of the Profile GPU Rendering developer, there is also an On screen as bars option. Its choice will show the diagram for each window on your screen together with the green line indicating a threshold of 16 ms.

image

On an example on the right we can see that some frames cross the green line that means that they demand more than 16 ms for drawing. As blue color, apparently, dominates, we understand that there were many representations for drawing, or they were difficult. In this case, I scroll a tape which supports different types of representation. Some elements become invalid, and some — more difficult in comparison with others. Perhaps, the reason for which some frames crossed this line, that the element, difficult for display, got.

Hierarchy Viewer


I adore this tool, and it is a pity to me that many do not use it!
Using Hierarchy Viewer, we can receive performance statistics, see complete hierarchy of representation on the screen and have access to all member properties. You can also make a dump of all data of a subject, see all parameters used for each attribute of style, but it is possible only if Hierarchy Viewer is started by standalone, not from Android Monitor. I use this tool when I create models of the application and I want to optimize them.

image

On the center we can see the tree reflecting hierarchy of representation. The hierarchy of representation can be wide, but if it too deep (~ 10 levels), it can cost much to us. Every time when there are measurements of representation in View#onMeasure () or when his descendants in View#onLayout accommodate (), these commands extend to descendants of these representations, doing the same. Some models will take each step twice, for example, RelativeLayout and some LinearLayout of a configuration and if they are enclosed — number of passes increases in geometrical progression.

In the lower right corner we can see "plan" of our model, noting where each representation is located. We can select representation here, or on a tree and see at the left all properties. At design of the model I sometimes am not sure why the certain notion comes to an end where it comes to an end. Using this tool, I can trace on a tree, select it and see where it is in a window of print preview. I can make interesting animations, looking at final measurements of representations on the screen and to use this information precisely to place elements around. I can find the lost representations which were blocked by other representations inadvertently.

image

For each representation we have time which was required for measurement/creation of its model/drawing and all his descendants. Colors reflect as these representations are executed in comparison with other representations in a tree, it is excellent method to find a weak link. As we also see a representation preview, we can pass on a tree and follow the instructions creating them, finding excess steps which we could delete. One of such things which influences performance is called Overdraw.

Overdraw


Apparently in the section of profiling of GPU — the phase Execute provided in the yellow color on the diagram can take more time to come to the end if GPU has to draw many elements, increasing time demanded for drawing of each frame. Overdraw occurs when we draw something over something another, for example the yellow button on a red background. GPU is required to draw, first, a red background and then the yellow button over it that does to inevitable overdraw. If we have too many layers of overdraw, it can become the reason for which GPU works more intensively and is not in time in 16 ms.

image

Using the Debug GPU Overdraw setup in the menu of the developer, all overdraw will be tinted to show overdraw degree in the field. If overdraw has degree 1x/2x, it is normal, even small red areas it is quite good too, but if we have too many red elements on the screen, it is probable we have a problem. Let's look at several examples:

image

At an example there is a list colored by green at the left, it is normal normally, but there is overdraw above which does the screen red, and it already becomes a problem. On an example on the right all list light-red. In both examples there is an opaque list with 2x/3x overdraw degree. These can happen if there is a background color on all screen in a window which blocks yours aktivit or a fragment, and also on submission of the list and representation of each its element.

Notice: the subject by default sets background color for all your window. If you have aktivit with the opaque model which blocks all screen, you can clean a background for this window to clean one layer of overdraw. It can be made in a subject or in a code by getWindow challenge () by .setBackgroundDrawable (null) in onCreate ().

Using Hierarchy Viewer, you can export all your layers of hierarchy to the PSD file to open it in Photoshop. As a result of research of different layers overdraw in the model will be opened all. Use this information to clean excess overdraws, and do not agree to green, aim at blue!

Alpha


Use of transparency can influence performance and to understand — why, let's look what occurs at installation of parameter an alpha in representation. Let's consider the following structure:

image

We see the model which contains 3 ImageView which block each other. In direct implementation, setting to an alpha by means of setAlpha (), the command will be called to extend to all representations of descendants, ImageView in this case. Then these ImageView will be drawn with the alpha parameter in the frame buffer. Result:

image

It not that we want to see.

As each ImageView was drawn with value an alpha, all blocked images will merge. Fortunately, OS have a solution of such problem. The model will be copied in separate off-screen the buffer, the alpha will be applied in this buffer in general and the result will be copied in the frame buffer. Result:

image

But … we will pay for it the price.

Additional drawing of representation in off-screen the buffer before it is made in the frame buffer, adds one more imperceptible overdraw a layer. OS does not know when to use this or direct approach earlier so by default always uses one complex. But there are still methods allowing to set an alpha and to avoid difficulties with adding of off-screen of the buffer:
  • TextViews — use setTextColor () instead of setAlpha (). The alpha for the text will bring use of the channel to that, the text will be redrawn.
  • ImageView — use setImageAlpha () instead of setAlpha (). The same reasons, as for TextView.
  • The user representations — if our user representation does not support overlapping, this end-to-end solution does not matter for us. There is no method by which our representations descendants will be connected together as it is shown on an example above. Having redefined the hasOverlappingRendering method () so that it returned false, we OS signals to take directly a way to our representation. Also there is an option manually to process what occurs at installation an alpha by redefinition of the onSetAlpha method () so that it returned true.

Hardware acceleration


When hardware acceleration was included in Honeycomb, we received new model of drawing for display of our application on the screen. It includes structure of DisplayList which writes commands of drawing of representation for faster receipt of the image. But there is still other interesting feature about which developers usually forget — the presentation layers.

Using them, we can draw representation in off-screen the buffer (as we saw earlier when we used the alpha channel) and to manipulate with it as we will wish. It perfectly is suitable for animation as we can animate difficult representations quicker. Without levels animation of representations would become unavailable after change of properties of animation (such as the X-coordinate, scale, alpha value, etc.) . For difficult representations this inability extends to representations of descendants, and they will redraw in turn themselves that is expensive operation. Using the presentation layers, relying on hardware, the texture for our representation is created in the graphic processor. There are several operations which we can apply to texture, without the need for its replacement, such as change of provision on axes x/y, rotation, an alpha and others. All this means that we can animate difficult representations on our screen without their replacement in the course of animation! It does animation by more smooth. There is a code sample showing as it can be made:


// Using the Object animator
view.setLayerType(View.LAYER_TYPE_HARDWARE, null);
ObjectAnimator objectAnimator = ObjectAnimator.ofFloat(view, View.TRANSLATION_X, 20f);
objectAnimator.addListener(new AnimatorListenerAdapter() {
    @Override
    public void onAnimationEnd(Animator animation) {
        view.setLayerType(View.LAYER_TYPE_NONE, null);
    }
});
objectAnimator.start();

// Using the Property animator
view.animate().translationX(20f).withLayer().start();



Simply, truly?

Yes, but there are several moments which it is necessary to remember when using hardware level.

  • Do cleaning after work of your representation — hardware level uses the place on limited memory space on your GPU. Try to use it only in cases when it are necessary, for example, in animation, and do cleaning later. In an example above with ObjectAnimator I used the withLayers method () which automatically creates a layer at the beginning and deletes it when animation comes to an end.
  • If you change your representation after use of hardware level, it will result in its unfitness and it is necessary to reproduce representation on off-screen the buffer again. It will occur when the property which is not optimized for hardware level changes (are at the moment optimized: turn, scaling, movements on axes x/y, an alpha). For example, if you animate representation, using hardware level, and move representation on the screen while background color is updated, it will lead to permanent updates at the hardware level. It demands additional expenses which can make its use unjustified.

For the second case there is a method to visualize this updating. Using settings of the developer, we can activate "Show hardware layers updates".

image

After that representation will be is highlighted in the green color when updating at the hardware level. I used it some time ago when I had ViewPager which did not glance over the page with that smoothing which I expected. After I included this option, I returned back and thumbed through ViewPager, and here that I saw:

image

Both pages were green throughout all scrolling!

It means that for optimization of pages hardware level was used, and pages became useless when we glanced over ViewPager. I refreshed pages and thumbed through them, using parallax effect at a background and gradually recovered elements on the page. What I did not make, so it was not created by a hardware layer for the ViewPager pages. After studying of the ViewPager code I found out that when the user begins scrolling, at the hardware level both pages are created and are removed after scrolling comes to an end.

While it makes sense to use hardware levels for scrolling of pages, in my case it did not approach. Usually pages do not change when we glance over ViewPager and as they can be rather difficult, hardware level help to do drawing much quicker. In the application on which I worked it was not so, and I cleaned this hardware layer, using small Haque who I wrote.

Hardware layer — not a silver bullet. It is important to understand how it works and to use properly, otherwise you can face a big problem.

Make


By preparation for all these examples shown here I wrote a set of a code for simulation of these situations. You can find them on Github repository, and also on Google Play. I separated scenarios for different aktivit, and tried to document them as far as possible to help to understand from what sort problems you can face, using aktivit. Read documentation on aktivit, open tools and play with the application.

It is more than information


In process of development of Android OS also methods by means of which you can optimize your applications develop. New tools are entered into Android SDK, also new functions are added to OS (for example, hardware levels). It is very important to master new and to study compromises before deciding to change something.

There is a remarkable playlist on YouTube which is called Android Performance Patterns, with a set of short videos from Google, explaining the different things connected with performance. You can find comparisons between different data structures (HashMap vs ArrayMap), optimization of Bitmap and even about optimization of network requests. I very much recommend to look at all of them.

Enter Android Performance Patterns Google + community and discuss performance with other participants, including the staff of Google to share ideas, articles and questions.

Other interesting links


  • Study how Graphics Architecture in Android works. There is everything that you should know about drawing of your UI in Android, different system components, such as SurfaceFlinger and as all of them with each other interact speak.

  • Talk from Google IO 2012 shows as the model of drawing and as/why we receive garbage when our UI is drawn works.

  • Android Performance Workshop talk from Devoxx 2013 shows some to optimization which were made in Android 4.4 in drawing model, and represents different tools for optimization of performance (Systrace, Overdraw, etc.).

  • Remarkable post about Preventive optimization and what it differ from Premature optimization in. Many developers do not optimize the code because they think that this change is insignificant. One thought which you have to remember is that all this in the sum becomes a big problem. If you have an opportunity to optimize only small part of a code, it can seem insignificant, I do not exclude.

  • Memory management in Android — old video with Google IO 2011 which is still actual. It shows as Android manages memory of our applications and how to use tools, such as Eclipse MAT to reveal problems.

  • The case study made by the engineer of Google Romain Guy for optimization of popular twitter client. In this research Romain shows how it found performance concerns in the application and that it recommends to make to correct them. There is a following post showing other problems after it was processed.

I hope, you have enough information, and more confidence now to begin optimization of applications already today!

Begin trace, or include some other available options of the developer, and just begin with them. I suggest to share your thoughts in comments.

Read more »