In this work, an automatic robotic manipulator control based on visual information for object tracking is presented. The tracking process employ Recurrent Neural Network and Kalman filter techniques to predict the location of target object which is used to calculate movement of the robot. A manipulator control process was developed to provide consistent motion and reduce vibration. We also propose an object analysis based on image processing technique for object size determination. We implemented the proposed system with an actual robotic manipulator with a gripper for experiments. The experiments shown that object size estimation method results the maximum accuracy at 1.13%. The proposed object tracking and manipulator control process achieved vibration reduction 3.05 times compared to simple control method and reduced the time of end-effector reaching the target object by 1.29sec.