This paper investigates the potential of stress recognition using the data from heterogeneous sources. Not only physiological signals but also reaction time (RT) is used to recognize different stress states. To acquire the data related to the stress of an individual, we design the experiments with two different stressors: visual stressor (Stroop test) and auditory stressor. During the experiments, the subjects perform RT task. Three physiological signals, Electrodermal activity (EDA), Electrocardiography (ECG) and Electromyography (EMG) as well as RTs are recorded. We develop the classifier based on the Support Vector Machines (SVM) for the stress recognition given the physiological signals and RT respectively. An overall good recognition performance of the SVM classifier is obtained. Besides, we present the strategy of recognition using the decision fusion. The recognition is thus achieved by fusing the classification results of physiological signals and RT with the voting method and a further improvement of recognition accuracy is observed. Results indicate that RT is efficient for stress recognition and the fusion of physiological signals and RT can bring in a more satisfied recognition performance.