NDSU: Mixed-Reality & eKanban

Mixed-Reality & eKanban
Mixed Reality is the topic of this year’s “Capstone Project”. Inwerken has been active as a project sponsor for several years. Numerous pioneering projects with the North Dakota State University (NDSU) have already been implemented. From the broad spectrum of possible areas of application for mixed reality, the team of students and trainees focused on the topic of industrial IoT – eKanban. The goal was that the sensor information, such as the last signal, the battery or signal strength should be displayed on an IoT device visually.

Our partner company WSN Technologies AG has committed itself in providing the required sensors hardware. In addition to Inwerken, various other companies such as Microsoft, John Deere or IBM also offer sponsorships for Capstone projects. But the students are able to choose their own preferences.

Way of Communication

North Dakota and Germany are 7,000 kilometers apart with a time difference of seven hours. That has not presented a communication problem because most of the communication has been done with the help of the “Microsoft Teams” tool. This tool also includes document exchange, status updates with Kanban board, 24-hour chats and weekly video meetings.
The project is the practical part of the students’ final exam, while Inwerken benefits from new innovative products and features. Both sides learn through the respective and unique project experiences. Some of the challenges that both parties might face during the project are the time difference as well as cultural differences.

Combination: Mixed-Reality & eKanban

Mixed reality is the expansion and supplementation of the human perception system through technically supported information processing. In the current project, the aim is to improve Industrial IoT in the form of eKanban with the help of mixed reality and thereby levering out modern production problems.

Use scenario A: The shelf to be refilled is first located in order to make navigation easier for an employee working in vast logistics hall. It receives navigation instructions and the shelf’s current fill levels via the mixed reality glasses.

Application scenario B: A technician can see the telemetry data of the sensors through the glasses. He therefore knows when the batteries are empty, whether there are signal problems and if the sensor is defective.

The five students work effectively and are always motivated in finding a solution. For this reason they have been divided into two teams. Team A deals with the user interface and Team B with the so-called AR markers – (this detects the sensors, the same as barcodes being used on products in a supermarket).

The „Marker-System“

Every sensor in a logistics hall has its own unique barcode. However, due to technical variables, the type and number of markers available is dynamic. Depending on the height of the shelves, the markers must have a certain size or the camera of the end device must have a corresponding resolution in order to recognize them as AR objects.

The MR-Interface

Our application should also be designed for the use of MR glasses but the question was how to operate it. An additional remote control would make little sense since the technician would only have one hand at his disposal. This problem we solved with “Virtual Buttons” which means real objects or virtual buttons, are touched by the technician with his finger and then act like buttons to control the interface.

How does a company benefit from this solution?

Companies benefit from the integration because the flow of information is accelerated. Whether with tablets, smartphones or glasses – a technician does not need to have the information prior to his work task. The data is made available to him when he needs it. Shop floor employees can also find their way around better and more easily. Error rates are reduced. Nobody needs to search anymore in executing certain refill orders as the data is projected directly into the field of vision.

When mixed reality becomes a part of everyday working life the question will arise as to how quickly it can be integrated into existing systems. Before implementing this with currently expensive MR glasses, tablets and alternative solutions can also be considered.

Current status & mid-terms presentation

The students have recently given their mid-term presentation. This always takes place halfway through the project and has been very successful this year:

We are thrilled with the progress of the project so far and we are looking forward to the result.  We are definitely envisioning numerous application scenarios.