Machine Learning and Reliable System Analysis with Astah

This is an introduction to our new modeling framework that makes high level requirements for safety, high reliability and availability of the system align with machine learning pipelines consistently. We will be featuring a clear overview lecture video by Professor Hiroyoshi Washizaki from Waseda University. Please turn the English subtitle on, and if you speak Japanese, here’s the Japanese version of this blog.

Q: What is the problem you are trying to solve in this research?

The current typical development of machine learning-based systems using deep neural networks often tends to be ad hoc. It involves coding in Python, training with labeled data, evaluating performance, and when the desired results don’t come easily, adjusting hyperparameters, performing data augmentation, and so on. It’s a cycle of trial and error in a way. There are two mountains to climb here (we call them twin peaks), but we often struggle predominantly with the peak on the right. We train, evaluate, and iterate on deep learning models, but without knowing for sure if it’s going well at a higher-level view like objectives or values of the system to the users. This often leads to a loop of activity, turning without certainty.

We aim to make this whole process an ‘engineering’ process. That is to say, we want to understand what purpose this deep learning model serves in the first place, how far we’re aiming for impact in terms of project value, what kind of data characteristics it deals with, and what the expected performance and accuracy should ideally be. Considering these higher-level project requirements and values (including safety), we plan to approach the objective while conducting both the implementation on the right peak and contemplating the meaning and value within the system as a whole on the left peak, going back and forth between the two. For this reason, a left view described in terms of purpose, meaning, and value for the stakeholders is crucial for the project.

Q: What kind of diagrams or models do you use?

We have integrated industry-standard models for requirements, systems, and safety analysis using methods such as STPA/STAMP, KAOS Goal Model, GSN (Safety Case), and SysML(Architecture). In addition to these, we have developed custom views similar to a ‘Business Model Canvas’ at the objective level (referred to as AI Project Canvas and ML Canvas), which link the entire system together.

Q: What is the reason for choosing Astah System Safety as the platform?

Astah System Safety has been widely used in our department (Waseda University, Faculty of Engineering) for classes. It is user-friendly, and furthermore, it was easy to customize by connecting various models through its APIs.

Q: What platform is used on the right mountain(ML Pipeline)?

We have chosen DVC as a robust choice for open-source software that allows for version control and pipeline creation for models. It enables version control for training, validation data, hyperparameters, and automation of workflows.

Q: What are the advantages of using this platform?

For example, in image classification, consider distinguishing between cars and pedestrians using machine learning. With a right-side approach alone, if you increase the recognition rate for pedestrians, you might see a decrease in the recognition rate for cars.

Instead of blindly focusing on the right side, let’s first organize things in the left-side world. In other words, we want to clearly understand what is being fulfilled and what is not, based on the initially set values of the project.

By visualizing the fulfillment of goals, requirements, and safety levels, stakeholders can have a clear understanding of sufficiency to see which direction to move for the next step.

AI and SE

Prof. Washizaki conducts research in a field that bridges machine learning and software engineering in both directions. He has also authored the several books in the area of “SE for AI” (Software Engineering for AI) and “AI for SE” (AI for Software Engineering).

It is anticipated that fields like these, which combine software engineering and artificial intelligence, will continue to advance as required from the industry.

Interviewed by Kenji Hiranabe