We discuss the problem of restructuring visual data from different heterogeneous sources to analyze an event of interest. We present X-coherence: a pipeline aiming at organizing and representing pieces of data, tying them coherently with the real world, and with one another. Finally, we outline research challenges while seeking X-coherence.