Emotions are considered to be the reflection of human thinking and decision-making process which increase his/her performance by producing an intelligent outcome. Hence it is a challenging task to embed the emotional intelligence in machine as well so that it could respond appropriately. However, present human computer interfaces still don’t fully utilize emotion feedback to create a more natural environment because the performance of the emotion recognition is still not very robust and reliable and far from real life experience. In this paper, we present an attempt in addressing this aspect and identifying the major challenges in the process. We introduce the concept of ‘emotion profile’ to evaluate an individual feature as each feature irrespective of the modality has different capability for differentiating among the various subsets of emotions. To capture the discrimination across target emotions w.r.t. each feature we propose a framework for emotion recognition built around if-then rules using certainty factors to represent uncertainty and unreliability of individual features. This technique appears to be simple and effective for these kind of problems.