Despite progress in visual reconstruction from fMRI signals, evaluating reconstruction quality remains challenging due to noisy, low-resolution data and semantic ambiguity. Conventional metrics often overlook perceptual and structural alignment with the original stimuli. To address this, we propose Graph-based Semantic and Structural Similarity (GSS), a novel evaluation approach that represents both stimuli and reconstructions as patch-wise graphs using CLIP-derived features. By applying graph matching, GSS captures spatial and semantic relationships beyond pixel-level similarities. Our approach aligns with neuroscientific models of visual processing and demonstrates robust, interpretable results that complement existing metrics.