Deep learning introduced a dramatic improvement in methods for automated image recognition in Computer Vision. Despite this progress, there is an immense gap between the accuracy of trained models during production and after deployment. One of the contributing factors for this gap is how machine learning models are designed and evaluated: without the expectation that an unknown class during training will be experienced during operation — the “open set problem”. To reduce this gap, it is important to know how to detect the problem and how to tackle it theoretically and in practice. The purpose of this talk is to introduce the open set problem in the context of visual computing, with a case study on the classification of LEGO Minifigures for Hidden Side.