The Brain-computer interface (BCI) is used to enhance the human capabilities. The hybrid-BCI (hBCI) is a novel concept for subtly hybridizing multiple monitoring schemes to maximize the advantages of each while minimizing the drawbacks of individual methods. Recently, researchers have started focusing on the Electroencephalogram (EEG) and “Functional Near-Infrared Spectroscopy” (fNIRS) based hBCI. The main reason is due to the development of artificial intelligence (AI) algorithms such as machine learning approaches to better process the brain signals. An original EEG-fNIRS based hBCI system is devised by using the non-linear features mining and ensemble learning (EL) approach. We first diminish the noise and artifacts from the input EEG-fNIRS signals using digital filtering. After that, we use the signals for non-linear features mining. These features are “Fractal Dimension” (FD), “Higher Order Spectra” (HOS), “Recurrence Quantification Analysis” (RQA) features, and Entropy features. Onward, the Genetic Algorithm (GA) is employed for Features Selection (FS). Lastly, we employ a novel Machine Learning (ML) technique using several algorithms namely, the “Naïve Bayes” (NB), “Support Vector Machine” (SVM), “Random Forest” (RF), and “K-Nearest Neighbor” (KNN). These classifiers are combined as an ensemble for recognizing the intended brain activities. The applicability is tested by using a publicly available multi-subject and multiclass EEG-fNIRS dataset. Our method has reached the highest accuracy, F1-score, and sensitivity of 95.48%, 97.67% and 97.83% respectively.