It is observed that specially abled people are unable to communicate properly due to the lack of knowledge about their sign language amongst common people. This system aims at encountering this problem and proposing a feasible solution. The aim is to build a human responsive system that can aid specially abled (blind, deaf and aphonic) people by interpreting the sign language (i.e. hand gestures). Sign language of such people is mostly composed of hand gestures. Gesture recognition is a technique that uses the concepts of computer vision which is used for the interaction between human and computer or it can be for any automation implementation purpose. Further it is proposed to convert these gestures to text and speech so that it can be easily understood by others.