Systems and services equipped with sophisticated prediction functions based on machine learning are rapidly expanding, attracting substantial interest. But did you know that there exists a huge barrier of training data development to the introduction of machine learning, particularly deep learning? Currently, deep learning is successfully introduced, primarily in the area of “supervised learning.” However, there is a need to prepare massive amounts of “training data” (data with correct answers) to achieve high prediction accuracy using the supervised learning. In numerous applications, sufficient training data cannot be prepared when attempts are made to introduce new machine learning processes, which may exclude developers and users from the benefits of training data. Even if the data required for building training data were available, the preparation process would require enormous cost and specialized knowledge, thereby posing as a major obstacle to its introduction. The construction of a system that allows the introduction of machine learning even with limited information and the reduction of costs for creating high-quality training data are therefore the foremost challenges in the universal use of intelligent systems in practice.

To solve the challenge, this research aims to establish underlying technology for automatically building high precision prediction models from limited training data. The technology will fundamentally solve diverse problems related to the developing of training data, which has posed as a barrier to the introduction of machine learning, from areas where machine learning could not be introduced due to the failure to gather large amounts of high-quality training data to ones where introduction of machine learning had to be given up due to human costs and lack of specialized knowledge required for the development of training data.
In this research, we plan to establish innovative underlying technology from three perspectives. Our research team has already obtained world-leading achievements and evaluation results for different processes. By maintaining and accelerating this advantage, we aim to create more predominant core technologies.

Our research team has built world-leading learning theories and general-purpose algorithms in a field called “weakly supervised learning.” Weakly supervised learning is a learning method in which the label information of the training data is inaccurate or given only partially. In recent years, it has attracted interests across the world as a formidable solution for overcoming the massive costs required for collecting labels for medical information, etc. This project thus aims to establish a unified theory on weakly supervised learning by further developing the research results achieved thus far and refining the theory. To enable practical application of the unified theory in the real world, we also promote research and development to build general-purpose algorithms and pursue robustness for dealing with noise and abnormal values.

Another approach of our focus is the development of knowledge transfer technology called domain adaptation. Domain adaptation is the technique of diverting a prediction model learned in a certain area to a target with different properties. This makes it possible to convert the artificial knowledge generated in large quantities by simulation into knowledge that can be used in real environments. A challenge faced with current domain adaptation is the strong restriction requiring the source and target categories to match. In this project, we promote research on technologies that can ease the restriction. Other research and development endeavors include technologies for increasing the number and diversity of source areas and selecting and adapting appropriate knowledge from them, as well as methods for combining [1] weakly supervised learning and domain adaptation.

We are also engaged on the development of integrated automatic learning underlying technology combining [1] weakly supervised learning and [2] knowledge transfer technology. To build high precision models in machine learning, it is imperative to appropriately decide model structures, preprocess data, set parameters, etc. We therefore aim to build an efficient parallel distribution processing platform that automatically performs these settings. Particularly, for [1] weakly supervised learning, we will focus on automating the selection of optimum algorithms based on the quality of information assigned to training data (e.g., labeled, unlabeled, label reliability, similarity). For [2] knowledge transfer technology, we will focus on automating the selection of appropriate knowledge to be transferred as well as develop integrated methods for these.

To overcome the training data development barrier to the introduction of machine learning, this project focuses on the development of underlying technology for leveraging artificial intelligence (AI), which can be applied extensively. We hope that our endeavors will pave new avenues for introducing machine learning, and enable machine learning to be applied to an even wider range of industries and services than ever before. We also believe that the underlying technology will significantly contribute to the building of a better society that “Beyond” AI strives for by enabling the wide use of the AI-based intellectual systems around the world.
Furthermore, the underlying technology is expected to produce critical results in terms of the development of science and technology. It has the potential to change conventional methodologies of scientific research, with which it has been difficult to obtain massive amounts of training data due to restrictions on the number of experiments that can be conducted, as well as the potential to help acquire never-imagined knowledge in various areas of natural science.

The goal of this project is to develop methods for learning prediction models from limited supervised information and for automatic construction of such models. To achieve this goal, we have developed theories and algorithms for learning from limited supervised information, theories and algorithms for knowledge transfer, and a framework for automatic construction of highly accurate prediction models. In addition, we have worked on spatio-temporal modeling with the goal of constructing world models based on the real environment. Sugiyama group is mainly responsible for developing the theory and algorithms for learning from limited supervised information, while Harada group is mainly responsible for the other topics. An overview of the results is given below.

In the first subproject, we aim to develop theory and algorithm for learning from limited supervision. More specifically, we have been exploring the problems of learning from weak supervision, noisy labels, and biased data.
For weakly supervised learning, we have been developing a unified framework that estimates the classification risk in an unbiased manner only from available weak supervision and corrects the risk estimator to mitigate overfitting. We successfully developed such methods for weakly supervised learning problems from partial labels, pairwise comparison labels, pairwise similarity confidence labels, and multiple sets of unlabeled data. We summarized such a unified framework in a monograph and published it from the MIT Press. Furthermore, we applied a weakly supervised learning method to an audio signal enhancement problem and demonstrated its practical usefulness.
For label-noise problems, we have also been estimating the classification risk in an unbiased manner based on the noise transition matrix. The main technical challenge was estimation of the noise transition matrix since it is non-identifiable without additional assumptions. We conducted theoretical analysis of this problem and developed theoretically justifiable algorithms. Furthermore, we considered an even more challenging scenario of instance-dependent noise and developed practically useful algorithms. In addition, we also considered the problem of coping with adversarial noise. We explored various adversarial scenarios and developed robust learning algorithms.
For data bias, we have been tackling a learning problem under continuous distribution shifts, where data generating distributions are changing over time. For this problem, we developed online learning methods for label shift and covariate shift that adaptively update our model only from unlabeled data without knowing the distribution shift. We theoretically proved that the proposed methods achieve the same dynamic regret as the case where the distribution shift is known.

In the second subproject, we aim to develop theories and algorithms for knowledge transfer. When a model learned in one domain is applied to another domain, e.g. when a model learned in a simulation is applied to a real-world environment, the expected prediction accuracy in the target domain may not be achieved due to domain differences (domain shift). Unsupervised domain adaptation (UDA) is an effective approach to solve the domain shift problem by using information from both the source domain where the model was trained and the target domain where the model is applied. Most recent domain adaptation methods are based on the theory of Ben-David et al. which simultaneously minimizes the error in the source domain and the distance between the marginal distributions, but typically ignores the joint error due to its difficulty in estimation. To solve this problem, we propose a new objective function related to the upper bound of the joint error. In addition, most existing UDA methods assume that the raw source data is directly available during the training of the target domain when transferring knowledge from the source to the target domain. In these days of increasing privacy concerns, direct availability of source data is not always possible when applying UDA methods in a new target domain. To solve this problem, we address Source Data Free Unsupervised Domain Adaptation (SF-UDA), which does not use source data and model parameters during training of the target domain.

Regarding the framework for automatic construction of highly accurate prediction models, we focused on efficient models due to the high computational cost of automatic model construction. To achieve this goal, we introduced hyperbolic space neural networks, which can model hierarchically structured data in low dimensions. These networks use a non-Euclidean geometry, hyperbolic space, which allows continuous embedding of tree structures with low distortion. The study generalizes basic components of neural networks under a unified mathematical interpretation using a single hyperbolic geometric model, the Poincaré ball model. We also worked on spiking neural networks (SNNs) to increase speed and reduce power consumption. SNNs are binary, event-driven, and can run on neuromorphic devices with high speed and low power. In this study, a variational autoencoder (VAE) based on SNNs is constructed for image generation.
Image data as it is would have several million dimensions, and it would be extremely difficult to collect training images on a scale appropriate for that number of dimensions. However, the real world has a 3-dimensional structure, or a 4-dimensional low-dimensional structure if time is included, which can be used as prior knowledge to significantly reduce the cost of annotation. Therefore, with the goal of reducing the amount of data required by exploiting this low-dimensional structure of the real world, we have worked on the matching and registration of partial point clouds, new implicit and explicit shape primitive representation methods for 3D shapes, and the acquisition of controllable radiance fields for articulated objects from sparse observational images and videos. We also proposed a new method to deform the radiance field, namely free-form deformation of the radiance field.

※For more detail, please refer to the URL below.
https://beyondai.jp/contents/wp-content/uploads/2024/01/Research-report_Tatsuya-Harada_202309.pdf