Title: Dynamics-based Invariants for Video Understanding

Abstract: The power of geometric invariants to provide solutions to computer vision problems has been recognized for a long time. On the other hand, dynamics-based invariants are often overlooked. Yet, visual data come in streams: videos are temporal sequences of frames, images are ordered sequences of rows of pixels and contours are chained sequences of edges. In this talk, I will discuss the key role that systems theory can play in timely extracting and exploiting dynamics-based invariants to capture actionable information that is very sparsely encoded in high dimensional data streams. The central theme of this approach is the use of dynamical models, and their associated invariants, as an information-encoding paradigm. We will show that embedding problems in the conceptual world of dynamical systems makes available a rich, extremely powerful resource base, leading to robust solutions, or, in cases where the underlying problem is intrinsically hard, to computationally tractable approximations with sub optimality certificates. We will illustrate these ideas in the context of several practical applications: crowd-sourcing video, activity recognition, human re-identification and video prediction.

Bio: Octavia Camps received a B.S. degree in computer science and a B.S. degree in electrical engineering from the Universidad de la Republica (Uruguay), and a M.S. and a Ph.D. degree in electrical engineering from the University of Washington. Since 2006, she is a Professor in the Electrical and Computer Engineering Department at Northeastern University. From 1991 to 2006 she was a faculty of Electrical Engineering and of Computer Science and Engineering at The Pennsylvania State University. Prof. Camps was a visiting researcher at the Computer Science Department at Boston University during Spring 2013 and in 2000, she was a visiting faculty at the California Institute of Technology and at the University of Southern California. She is an associate editor of Computer Vision and Image Understanding (CVIU). Her main research interests include robust computer vision, image processing, and machine learning.

Title: Capture and Animation of Human Motion

Abstract: In this talk, Jessica Hodgins will present research on constructing controllers for human motion from first principles, on capturing data of human motion and conclude with recent results on using captured data to learn controllers. Throughout the talk, she will reflect on the lessons learned, both from the research itself and how the projects came to be.

Bio: Jessica Hodgins is a Professor in the Robotics Institute and Computer Science Department at Carnegie Mellon University. From 2008-2016, she founded and ran research labs for Disney, rising to VP of Research and leading the labs in Pittsburgh and Los Angeles. From 2005-2015, she was Associate Director for Faculty in the Robotics Institute, running the promotion and tenure process and creating a mentoring program for pre-tenure faculty. Prior to moving to Carnegie Mellon in 2000, she was an Associate Professor and Assistant Dean in the College of Computing at Georgia Institute of Technology. She received her Ph.D. in Computer Science from Carnegie Mellon University in 1989. Her research focuses on computer graphics, animation, and robotics with an emphasis on generating and analyzing human motion. She has received a NSF Young Investigator Award, a Packard Fellowship, and a Sloan Fellowship. She was editor-in-chief of ACM Transactions on Graphics from 2000-2002 and ACM SIGGRAPH Papers Chair in 2003. She was an elected director at large on the ACM SIGGRAPH Executive Committee from 2012-2017 and in 2017 she was elected ACM SIGGRAPH President. In 2010, she was awarded the ACM SIGGRAPH Computer Graphics Achievement Award and in 2017 she was awarded the Steven Anson Coons Award for Outstanding Creative Contributions to Computer Graphics.

Title: CNN vs SIFT-based localization: out with the old?

Abstract: Recently Deep Learning has achieved such a great performance in so many vision tasks that researcher are starting to use it to solve any imaginable task, disregarding previous methods optimised over decades of research work. Image-based localization is one of those problems: a classic task in Computer Vision where given an image, we are asked to find its camera pose and orientation with respect to a given model which can have a scale ranging from a city to a small room. Recently researchers have turned into CNNs for pose estimation disregarding previous literature and even disregarding basic epipolar geometry. In our first work, we put both methods to test: on the one hand, we clearly show CNN-based localization has still an incredibly long way to go with respect to SIFT-based methods; on the other hand, we present an indoor dataset in which classic methods suffer due to textureless surfaces and repetitive structures. Afterwards, I will also present our first attempt towards a fully scalable method based on relative pose estimation that allows us to localize a camera in any given scene with a single network (and even use epipolar geometry!). I will show that a marriage between CNN and Computer Vision classic knowledge is still possible and very much desirable in today's research landscape.

Bio: Prof. Laura Leal-Taixé is leading the Dynamic Vision and Learning group at the Technical University of Munich, Germany. She received her Bachelor and Master degrees in Telecommunications Engineering from the Technical University of Catalonia (UPC), Barcelona. She did her Master Thesis at Northeastern University, Boston, USA and received her PhD degree (Dr.-Ing.) from the Leibniz University Hannover, Germany. During her PhD she did a one-year visit at the Vision Lab at the University of Michigan, USA. She also spent two years as a postdoc at the Institute of Geodesy and Photogrammetry of ETH Zurich, Switzerland and one year at the Technical University of Munich. Her research interests are dynamic scene understanding, in particular multiple object tracking and segmentation, as well as machine learning for video analysis.

Title: When bias in AI and product design means life or death.

Abstract: Description: A talk about bias in AI products could impact humanity - the state of where we are and the largest pitfalls all throughout the process. Will go through a history of AI products and discuss where bias creeps in and how to create a safer, more delightful products for the world. Since we're at the start of the AI revolution, this lays groundwork for how to be thoughtful as a company and as a consumer.

Bio: Carol E. Reiley has been at the forefront of robotics and AI for over 15 years and was the youngest member on the IEEE Robotics and Automation board. She's worked on highly regulated products in a variety of different applications such as space, underwater and medical. She did her graduate work at Johns Hopkins University as an NSF Fellow researching how humans and robots interact, worked on the da Vinci system at Intuitive Surgical, and most recently cofounded and was the President of a self-driving car startup drive.ai. She's raised over $77M and built a partnership with Lyft, Grab, and automotive companies. Reiley was selected by Forbes as one of the twenty incredible women advancing artificial intelligence research. She currently holds over six patents, over a dozen publications, and was the first female engineer on the cover of MAKE magazine. She’s been featured in major publications such as MIT Tech Review, NYTimes, Harper’s Bazaar and given several Tedx Talks. She is also a published author and is now starting a new startup.

Bio: Michael Black received his B.Sc. from the University of British Columbia (1985), his M.S. from Stanford (1989), and his Ph.D. from Yale University (1992). After post-doctoral research at the University of Toronto, he worked at Xerox PARC as a member of research staff and area manager. From 2000 to 2010 he was on the faculty of Brown University in the Department of Computer Science (Assoc. Prof. 2000-2004, Prof. 2004-2010). He is one of the founding directors at the Max Planck Institute for Intelligent Systems in Tübingen, Germany, where he leads the Perceiving Systems department and serves as Managing Director. He is also a Distinguished Amazon Scholar, an Honorarprofessor at the University of Tuebingen, and Adjunct Professor at Brown University. His work has won several awards including the IEEE Computer Society Outstanding Paper Award (1991), Honorable Mention for the Marr Prize (1999 and 2005), the 2010 Koenderink Prize for Fundamental Contributions in Computer Vision, and the 2013 Helmholtz Prize for work that has stood the test of time. He is a foreign member of the Royal Swedish Academy of Sciences. In 2013 he co-founded Body Labs Inc., which was acquired by Amazon in 2017.

Bio: Octavia Camps received a B.S. degree in computer science and a B.S. degree in electrical engineering from the Universidad de la Republica (Uruguay), and a M.S. and a Ph.D. degree in electrical engineering from the University of Washington. Since 2006, she is a Professor in the Electrical and Computer Engineering Department at Northeastern University. From 1991 to 2006 she was a faculty of Electrical Engineering and of Computer Science and Engineering at The Pennsylvania State University. Prof. Camps was a visiting researcher at the Computer Science Department at Boston University during Spring 2013 and in 2000, she was a visiting faculty at the California Institute of Technology and at the University of Southern California. She is an associate editor of Computer Vision and Image Understanding (CVIU). Her main research interests include robust computer vision, image processing, and machine learning.

Bio: Dima Damen is a Senior Lecturer (Associate Professor) in Computer Vision at the University of Bristol, United Kingdom. Received her PhD from the University of Leeds (2009). Dima's research interests are in the automatic understanding of object interactions, actions and activities using static and wearable visual (and depth) sensors. Dima co-chaired BMVC 2013, is area chair for BMVC (2014-2018), associate editor of Pattern Recognition (2017-). She was selected as a Nokia Research collaborator in 2016, and as an Outstanding Reviewer in ICCV17, CVPR13 and CVPR12. She currently supervises 9 PhD students, and 3 postdoctoral researchers.

Bio: Jessica Hodgins is a Professor in the Robotics Institute and Computer Science Department at Carnegie Mellon University. From 2008-2016, she founded and ran research labs for Disney, rising to VP of Research and leading the labs in Pittsburgh and Los Angeles. From 2005-2015, she was Associate Director for Faculty in the Robotics Institute, running the promotion and tenure process and creating a mentoring program for pre-tenure faculty. Prior to moving to Carnegie Mellon in 2000, she was an Associate Professor and Assistant Dean in the College of Computing at Georgia Institute of Technology. She received her Ph.D. in Computer Science from Carnegie Mellon University in 1989. Her research focuses on computer graphics, animation, and robotics with an emphasis on generating and analyzing human motion. She has received a NSF Young Investigator Award, a Packard Fellowship, and a Sloan Fellowship. She was editor-in-chief of ACM Transactions on Graphics from 2000-2002 and ACM SIGGRAPH Papers Chair in 2003. She was an elected director at large on the ACM SIGGRAPH Executive Committee from 2012-2017 and in 2017 she was elected ACM SIGGRAPH President. In 2010, she was awarded the ACM SIGGRAPH Computer Graphics Achievement Award and in 2017 she was awarded the Steven Anson Coons Award for Outstanding Creative Contributions to Computer Graphics.

Bio: Prof. Laura Leal-Taixé is leading the Dynamic Vision and Learning group at the Technical University of Munich, Germany. She received her Bachelor and Master degrees in Telecommunications Engineering from the Technical University of Catalonia (UPC), Barcelona. She did her Master Thesis at Northeastern University, Boston, USA and received her PhD degree (Dr.-Ing.) from the Leibniz University Hannover, Germany. During her PhD she did a one-year visit at the Vision Lab at the University of Michigan, USA. She also spent two years as a postdoc at the Institute of Geodesy and Photogrammetry of ETH Zurich, Switzerland and one year at the Technical University of Munich. Her research interests are dynamic scene understanding, in particular multiple object tracking and segmentation, as well as machine learning for video analysis.

Bio: Xin Lu is researcher, developer, and manager at Adobe. She has been working on deep learning and its application on computer vision and image processing. Her recent research interests include deep neural network architecture optimization, neural network pruning, and image generation. Her research work has been deployed across Adobe desktop and mobile products. Prior to joining Adobe, Xin received her Ph.D. from The Pennsylvania State University. Her thesis focused on image aesthetics assessment, emotion recognition and image denoising.

Bio: Jitendra Malik is Arthur J. Chick Professor of EECS at UC Berkeley, and a Director of Research and Site Lead at Facebook AI Research in Menlo Park. He has published widely in computer vision, computational modeling of human vision, and machine learning. Several well-known concepts and algorithms arose in this research, such as anisotropic diffusion, normalized cuts, high dynamic range imaging, shape contexts and R-CNN. Jitendra received the Distinguished Researcher in Computer Vision Award from IEEE, the K.S. Fu Prize from IAPR, and the Allen Newell award from ACM and AAAI. He has been elected to the National Academy of Sciences, the National Academy of Engineering and the American Academy of Arts and Sciences.

Bio: Carol E. Reiley has been at the forefront of robotics and AI for over 15 years and was the youngest member on the IEEE Robotics and Automation board. She's worked on highly regulated products in a variety of different applications such as space, underwater and medical. She did her graduate work at Johns Hopkins University as an NSF Fellow researching how humans and robots interact, worked on the da Vinci system at Intuitive Surgical, and most recently cofounded and was the President of a self-driving car startup drive.ai. She's raised over $77M and built a partnership with Lyft, Grab, and automotive companies. Reiley was selected by Forbes as one of the twenty incredible women advancing artificial intelligence research. She currently holds over six patents, over a dozen publications, and was the first female engineer on the cover of MAKE magazine. She’s been featured in major publications such as MIT Tech Review, NYTimes, Harper’s Bazaar and given several Tedx Talks. She is also a published author and is now starting a new startup.

Abstract: State-of-the-art semantic segmentation approaches increase the receptive field of their models by using either a downsampling path composed of poolings/strided convolutions or successive dilated convolutions. However, it is not clear which operation leads to best results. In this paper, we systematically study the differences introduced by distinct receptive field enlargement methods and their impact on the performance of a novel architecture, called Fully Convolutional DenseResNet (FC-DRN). FC-DRN has a densely connected backbone composed of residual networks. Following standard image segmentation architectures, receptive field enlargement operations that change the representation level are interleaved among residual networks. This allows the model to exploit the benefits of both residual and dense connectivity patterns, namely: gradient flow, iterative refinement of representations, multi-scale feature combination and deep supervision. In order to highlight the potential of our model, we test it on the challenging CamVid urban scene understanding benchmark and make the following observations: 1) downsampling operations outperform dilations when the model is trained from scratch, 2) dilations are useful during the finetuning step of the model, 3) coarser representations require less refinement steps, and 4) ResNets (by model construction) are good regularizers, since they can reduce the model capacity when needed. Finally, we compare our architecture to alternative methods and report state-of-the-art result on the Camvid dataset, with at least twice fewer parameters.

Abstract: Dense video captioning is a fine-grained video understanding task that involves two sub-problems: localizing distinct events in a long video stream, and generating captions for the localized events. We propose the Joint Event Detection and Description Network (JEDDi-Net), which solves the dense video captioning task in an end-to-end fashion. Our model continuously encodes the input video stream with three-dimensional convolutional layers, proposes variable-length temporal events based on pooled features, and generates their captions. Unlike existing approaches, our event proposal generation and language captioning networks are trained jointly and end-to-end, allowing for improved temporal segmentation. In order to explicitly model temporal relationships between visual events and their captions in a single video, we also propose a two-level hierarchical captioning module that keeps track of context. On the large-scale ActivityNet Captions dataset, JEDDi-Net demonstrates improved results as measured by standard metrics. We also present the first dense captioning results on the TACoS-MultiLevel dataset.

Abstract: Many structured prediction tasks arising in computer vision and natural language processing tractably reduce to making minimum cost cuts in graphs with edge weights learned us- ing maximum margin methods. Unfortunately, the hinge loss used to construct these methods often provides a particularly loose bound on the loss function of interest (e.g., the Hamming loss). We develop Adversarial Robust Cuts (ARC), an approach that poses the learning task as a minimax game between predictor and “label approximator” based on minimum cost graph cuts. Unlike maximum margin methods, this game-theoretic perspective always provides meaningful bounds on the Hamming loss. We conduct multi-label and semi-supervised binary prediction experiments that demonstrate the benefits of our approach.

Abstract: We propose and investigate an identity sensitive joint embedding of face and voice. Such an embedding enables cross-modal retrieval from voice to face and from face to voice. We make the following four contributions: first, we show that the embedding can be learnt from videos of talking faces, without requiring any identity labels, using a form of cross-modal self-supervision; second, we develop a curriculum learning schedule for hard negative mining targeted to this task, that is essential for learning to proceed successfully; third, we demonstrate and evaluate cross-modal retrieval for identities unseen and unheard during training over a number of scenarios and establish a benchmark for this novel task; finally, we show an application of using the joint embedding for automatically retrieving and labelling characters in TV dramas.

Abstract: We develop a method for policy architecture search and adaptation via gradient-free optimization which can learn to perform autonomous driving tasks. By learning from both demonstration and environmental reward we develop a model that can learn with relatively few early catastrophic failures. We first learn an architecture of appropriate complexity to perceive aspects of world state relevant to the expert demonstration, and then mitigate the effect of domain-shift during deployment by adapting a policy demonstrated in a source domain to rewards obtained in a target environment. We show that our approach allows safer learning than baseline methods, offering a reduced cumulative crash metric over the agent's lifetime as it learns to drive in a realistic simulated environment.

Abstract: A number of studies have found that today's Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from 'cheating' by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model -- Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.

Bio: Timnit Gebru works in the Fairness Accountability Transparency and Ethics (FATE) group at Microsoft Research, New York. Prior to joining Microsoft Research, she was a PhD student in the Stanford Artificial Intelligence Laboratory, studying computer vision under Fei-Fei Li. Her main research interest is in data mining large-scale, publicly available images to gain sociological insight, and working on computer vision problems that arise as a result, including fine-grained image recognition, scalable annotation of images, and domain adaptation. She is currently studying the ethical considerations underlying any data mining project, and methods of auditing and mitigating bias in sociotechnical systems. The New York Times, MIT Tech Review and others have recently covered her work. As a cofounder of the group Black in AI, she works to both increase diversity in the field and reduce the negative impacts of racial bias in training data used for human-centric machine learning models.

Bio: Dima Damen is a Senior Lecturer (Associate Professor) in Computer Vision at the University of Bristol, United Kingdom. Received her PhD from the University of Leeds (2009). Dima's research interests are in the automatic understanding of object interactions, actions and activities using static and wearable visual (and depth) sensors. Dima co-chaired BMVC 2013, is area chair for BMVC (2014-2018), associate editor of Pattern Recognition (2017-). She was selected as a Nokia Research collaborator in 2016, and as an Outstanding Reviewer in ICCV17, CVPR13 and CVPR12. She currently supervises 9 PhD students, and 3 postdoctoral researchers.

Bio: Xin Lu is researcher, developer, and manager at Adobe. She has been working on deep learning and its application on computer vision and image processing. Her recent research interests include deep neural network architecture optimization, neural network pruning, and image generation. Her research work has been deployed across Adobe desktop and mobile products. Prior to joining Adobe, Xin received her Ph.D. from The Pennsylvania State University. Her thesis focused on image aesthetics assessment, emotion recognition and image denoising.