I have known object (simple rectangle) in cameras field of view, so I have acquired it's 3d position using:

solvePnP(pts3d,ptsimg,camMatrix, distCoeffs ,Rot,Trans,true);

Later, I have created disparity map using StereoSGBM.
Afterwards I have used reprojectImageto3d do obtain cloud of points. When I plotted that cloud of points and vector Trans obtained by solvePnP, I have noticed that distance between Trans and cloud of points is always around 50mm.
I have tried to solve that problem for a couple of days until i stumbled upon this post ->

#898 https://code.ros.org/trac/opencv/ticket/898

Bearing that in mind, and without any other idea, I have added a single line to my code:

Q.at<double>(3,2)=-Q.at<double>(3,2);

where Q is reprojection matrix obtained from stereoRectify.
Now everything works just fine, and just the way it should be.

I am sure that there is a bug somewhere in stereoRectify, probably at the place specified in already mentioned post.