README file
Data Set Title: RGB-MS Matching dataset
Data Set Author/s: Fabio Tosi (University of Bologna), 0000-0002-6276-5282; Pierluigi Zama Ramirez (University of Bologna), 0000-0001-7734-5064; Matteo Poggi (University of Bologna), 0000-0002-3337-2236; Samuele Salti (University of Bologna), 0000-0001-5609-426X; Stefano Mattoccia (University of Bologna), 0000-0002-3681-7704; Luigi Di Stefano (University of Bologna), 0000-0001-6014-6421;

Data Set Contact Person/s: Samuele Salti (University of Bologna), 0000-0001-5609-426X, samuele.salti@unibo.it; Stefano Mattoccia (University of Bologna), 0000-0002-3681-7704, stefano.mattoccia@unibo.it; Luigi Di Stefano (University of Bologna), 0000-0001-6014-6421, luigi.distefano@unibo.it;

Data Set License: this data set is distributed under a 
Creative Commons Attribution Non-Commercial 4.0 International license
Publication Year: 2022
Project Info: RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation, funded by HUAWEI TECHNOLOGIES OY (FINLAND); https://cvlab-unibo.github.io/rgb-ms-web/ . 
Data set Contents
The data set consists of a set of images collected in different environments. Training images are provided together with proxy labels. Testing images are provided together with ground-truth disparity labels. 

labaled folder
It contains 14 testing sequences, with available ground-truth labels
o	000 folder
o	001 folder
o	...
unlabaled folder 
It contains N sequences, annotated with proxy-labels

Data set Documentation
Abstract
We address the problem of registering synchronized color (RGB) and multi-spectral (MS) images featuring very different resolution by solving stereo matching correspondences. Purposely, we introduce a novel RGB-MS dataset framing 13 different scenes in indoor environments and providing a total of 34 image pairs annotated with semi-dense, high-resolution ground-truth labels in the form of disparity maps. To tackle the task, we propose a deep learning architecture trained in a self-supervised manner by exploiting a further RGB camera required only during training data acquisition. In this setup, we can conveniently learn cross-modal matching in the absence of ground-truth labels by distilling knowledge from an easier RGB-RGB matching task based on a collection of about 11K unlabeled image triplets. Experiments show that the proposed pipeline sets a good performance bar (1.16 pixels average registration error) for future research on this novel, challenging task.
Each testing sequence folder contains the following folders and files:
•	camera_00 folder
It contains left camera images (RGB), under different illumination
o	000.png
o	...
•	camera_01 folder
It contains right camera images (MS), under different illumination
o	000.raw
o	000.hdr
o	…
•	camera_01_gray folder
It contains right camera images (MS) averaged over channels, under different illumination
o	000.png
o	...
•	calib_00-0X.xml
Calibration file containing intrinsic/extrinsic parameters (X=2 balanced, X=1 unbalanced)
•	00-01.npy
Left ground-truth disparity map
•	00-01.png
Left ground-truth disparity map, for qualitative visualization

File specifics
Ground-truth disparity map are provided as .npy files, encoding floating point disparities.
Notes 
The dataset is release in two main archives: 
•	labeled examples, providing both training images (for which ground-truth annotations are provided) and testing images (for which ground-truth annotations are withheld).
•	unlabeled examples, for which images are provided with proxy labels.

Data sources 
RGB images have been collected with Sony IMX253LQR-C 12.4 Mpx sensors.
MS images have been collected with Sony IM-SM4X4-VIS2 2.2 Mpx sensor.

Methodologies 
RGB images can be opened with any python package among opencv, pil, etc.
MS images can be opened with spectral python package.
Ground-truth disparity maps can be opened with numpy.

List of variables