All right. Thanks everybody for joining us today on July 14th
2020, for the Light Up Virtual Conference, 24 Hours of Microsoft Technology Learning and
Fundraising to Fight COVID-19. We'll be talking about Connect to Azure for Developers
My name is Andreas Erben. That's important, take a picture of this slide
You can donate in support of the Universal Crisis, COVID-19 Crisis, whose contribution matters
We will probably go back to the slide at the very end and you can also
go to the Light Up Conference website to get this donation link
We have got some great sponsors. I will not read all the names individually
but without sponsors, an event like this would be impossible. So thanks a lot to every, every sponsor
Thanks also for Sivak Kone and to the Tech Platform to facilitate and set up this event
My name again, Andreas Erben. I'm the CTO for Applied AI and History Literacy
and CEO of Americas of Bayonets. I have 25 years of experience in this field and have been
developing VR and interactive user interfaces early in the mid-1990s, but then a lot of years ago, nothing was happening
I worked with companies of all sizes and all types, and started to get enterprises
So, but I love PC technology. My contact information is there if you have questions
You can also contact me after this event if you don't have anything to learn more
Talking about the Connect Center a little bit, right? I believe my camera feed is shared
and if not, then I will go back to it again. This is the Connect Center I'm holding into the camera
It's a little camera that has, as you can see, some light emitter and you see probably some speckles and
pixels on the camera and the camera sensors, and has a few cameras in there
It has connectors for USB-C, has a different connector for power. If you would unscrew this
it would have another connector for signal waves. But I have a lot of content to cover
so let's go back to my slides first, and let's look at what are the history of Connect
Connect initially was released for the Xbox 360. It was completely still gaming device
motion control, video games, and so on. But hackers, good hackers, started to dabble with this and started to write
applications where the Microsoft had a different instruction to solve. What do we do? Do we support the hackers or do we hide them
They decided to support the hackers and decided to make their API to develop for Microsoft public and
develop a product out of the Connect for Windows version 1.1, and which has skeleton tracking enabled
what that is, we will see later, with 20 different landmarks. This was a strong operation at that time
Then 2013, Connect 1 came out of Xbox One, and following 2014
Connect for Windows U2 was released, which improved on body tracking and a lot of other things
and the core sensor technology was also changed. Finally, in 2018 was a mixed year because we learned that
Connect for Xbox One didn't have the future, but then the project Connect for Azure was announced
and 2019, it launched as Average Connect Development Kit at the Mobile World Congress
and this new body tracking SDK and skeleton-free and zero-compose-mode video is joined
So this is, again, how it looks like. If you would blow it up
then that would look like this. So I believe you can see my mouse cursor
This is the main board and all the different components. You see the lenses here
you see the RGB camera, and you see the desk camera. This is the light emitting module
which is laser light, modulated laser light that's been emitted. You also have a microphone array here on top
where Connect can locate sounds in space, and theoretically in a specific direction
or you can use it for noise cancellation of ambient sound. Little bit extra and early content that we
can read about on this paper, so the public information, how does it actually work
We have a so-called time of light-based desk camera in there. It means it modulates light coming out of light emitters
and it knows how that light is modulated. So by knowing how it is modulated and knowing the exact timing of
that modulation, it can then calculate and integrate for every pixel that it sees of this light signal
and it's an incorrect light signal. It can then know based on this modulation
how far is something away from the camera in a specific spatial desk resolution
There are two resolution modes in simple steps, where you get a little bit more detail
on a narrow point of view, or you get a less detail but have a wider view
Some of the math inside works. However, a few other interesting factors also
Those are the pencil screens that you see from it, has an active IR in it
What does it mean, active IR in it? It means that it's driven by
the infrared light emitters of that device. It's infrared, but it's not heat
This camera cannot do heat specifically. But because of the sensory light modulation
it actually can remove ambient light from the picture. So if I would shine another infrared source on the same spot here
you wouldn't see a bright spot coming through, because it will only show
the light it knows about in the modulation in the camera. This is the so-called depth map
It's a visualization about how far something is away. It just shows the color screen for it
You don't want to worry about it. It has a full color in it. It has an IMU, so an acceleration unit
that you also will find in columns. That's very useful if you want to mount this on a robot
for example, or get an idea of which angle the camera is positioned
Here's the microphone array, the raw data. It has seven channels, so it's seven microphones
It has a software stack. Very, very robust. You have the EtherConnect2K device that's going to be used to see
But you see access to this via just regular USB portable. There's no driver
If it's a driver you're really working with, the sensor SDK accesses that device pretty directly
Each can be processed through the microphone screen. It's identical, so you can access the microphone screen
On top of the sensor SDK, there are a bunch of things you can do on top of that
What the sensor SDK provides, we will look at a little bit in detail. But we can use Azure Vision Services
like HoloLens Services Custom Vision to just look at the camera images from the camera
and get information out of it. While body tracking SDK, we will look at what that is
just that the magic they can understand what a human body does or posture
Then any type of user build skills that you would want to build on top of that SDK
that's then actually in the ability to just use the SDK. The sensor SDK, that's also something that
all Microsoft would never have done, completely open source. So it's broadcast from a low-level access
You can write software on Linux for that. Currently, as far as I know
it's Linux and Windows, and it's not yet on macOS. I don't know if that will ever happen
At least one of the challenges is that the component that is being used to calculate the 3D location
so basically the mathematical magic that works on the sensor stream and stick to the desk
that's the only part that's not open source. I don't think we can export that
because that's also accelerated by the CPU. Then I have all the other things
I don't know if it's accessible on Linux and Windows. The vision is to have it eventually even on Raspberry Pi type of devices
if they have enough processing power. So it should not be trained to purely desktop systems
As a language that's available, we can develop most natively in C
the whole thing is written in C. There's a C++ ref on top of this
and then a C-start wrapper. All my sample specifications are going to be in this C-start wrapper
By the way, I will not be able to monitor the chat myself, but the chat will be monitored by the moderators of the session
and we will have questions at the end for the answers. You don't need to hold them necessarily
but maybe it's better to hold them a little bit, and the moderator will do that. So there's tools, and it's a fantastic thing
it comes with tools. You can download it, you can install it
and then you'll have it on your hard drive. Somewhere here, I can put some files
and everything else is okay. Then tools. Tools is, you have a tool for updating the firmware
which I will install, and you have a recorder to record. Yes, it records a data stream
and then the Vuel component. So the Vuel component, if you just launch it
looks like this. And when you start it, it shows you a list of devices
It looks at the USB bus, where is what on the device, and then you open the device
When you open the device, no other software can access it. That's the difference compared to Kinect for Windows
the version two, where you could have multiple pieces of software accessing using Microsoft's same time
at the expense of having this centralized driver component that's installed on the computer
So they wanted to get rid of that. There's a bunch of settings you can choose here
Something called pinning, and it's basically, rather, combines Mabel and Pixel to get better information, and it's noisy
and better written, in that sense. And you can choose between a wide view and a narrow view
and you have a testifier camera. You can choose the color format
Three colors being full, too. The resolution of the colors is green
And the frame, how many frames you want to take, and 8 or 9 pixels, and so on, and so on, and so on
And let's start it with this. So this is a view, similar to what you're seeing
on the side, how it looks like. Now, let's walk into the picture real quickly
And yeah, and you can choose also between 2D mode and 3D mode
So you see here that it actually has maps, it's able to map all this video screen into 3D space
Okay, so do not forget to close the device, otherwise, if I show you a report later on
I will get an error, because my demo probably won't be able to access it
All right. So, Adder Connect, again, it's open-sourced from GitHub, and it's also the best way to report issues
or to ask questions, is to just do it right, GitHub. And you can also use the custom reports
to report new features, and TrackOverflow also tracks from the team, the Connect team actively monitors
So rather than trying to trace down those features inside the Microsoft tool, why don't they help me use GitHub
and they really are pretty responsive on that. So let's go to body tracking
Body tracking has been designed from the ground up new for Adder Connect 3D space
It provides what they call an instance segmentation map. It means that we give you the pixel
all of the image, the location of those pixels, that belong to a body
It gives you 3D joint positions for a person, and gives you unique IDs for each of those persons
to track temporally, meaning if you have frame one, and the next frame, frame two
you will get the same identifier, the same numbers for the same skeleton
even if the moves were out. It is important if you want to write software
The quality has been improved, and that's almost more correct as compared to Connect version B2
Has higher joint accuracy and sufficiency, improved robustness for side-screw bending and gluing
That's actually a big thing, because for Connect for Windows, for the Xbox, was designed primarily
to interact with the Xbox control. And so it was trained on people facing the camera
facing the screen, and interacting with the screen. So it would still track you relatively well
if you looked at this from the side, or if you were bending the angle or lying down
but it was never designed for that. And it can track up to 15 skeletons
which is not a hard limit, but a practical limit of Connect for Azure
You can do practical from development. I think Windows works, Linux works right now
it's not free anymore, actually. And the SDKs available in C, C++ and C++
It runs on top of what they call the Onyx Runtime. Onyx Runtime is a cross-platform engine, actually
that Microsoft uses for often using network exchange format to rely on arbitrary hardware
But supported right now is the NVIDIA hardware platform, and you need at the least an NVIDIA 1060 Ti
or better for hardware acceleration. Yes, it also will work with the NVIDIA 2080s
a newer device, so it doesn't have to complain to the thousand numbers of NVIDIA devices
So how did they do it? They trained it with synthetic data, right? So you have this device here
and what they did is, basically, those are not real people here, right
This is artificially rendered people based on the emotions of real people
and they put, basically, fake clothing on them, and so they know exactly
for each of those artificial synthetic people, like synthetic people like in a Pixar movie
Think about it, right? So they know exactly where each point on those bodies are
because it's all computer-simulated. And then they projected that, what they're rendering here
into how would the Kinect sensors be, like artificially, right? They try to approximate it as well as possible
How would the Kinect sensors do that exact same thing? And then train AI on top of that, right
So they get all those perfect positions. They know it's the perfect labeling. They know exactly where to land
because it's an artificial model. And this caused a lot of problems
For example, you need to make it like all body shapes and sizes, all skin colors, all, yeah
All types of humans that we have in reality can be simulated in the platform
and so they can train their lives. And they can do that also
in multiple simulated environments, not just in a gym or in a living room
but also in a patient's room or in an OR operating room and so on
So that's just the fundamental approach. Currently, it is trained really just for humans
No, we cannot track the raws of it. That's very special. We get almost every time I talk about it
But yeah, theoretically, they could enable that by that they would be used as a model for software
to train different things for other environments. If you compare with Kinect versus Zoom
I will not talk too much about this slide. In the video, we can pause it and then move on
Fundamentally, it's a newer, now it's used as a newer network compared to before
which is in part has a better synthetic strong tools, has improved robustness
is supported on more hardware, or more software platforms such as Windows
has super-sub-wrappers. It provides a need to load the power and more powerful hardware
For Kinect for Windows version two, the original Surface Pro was enough to really run it
So that's a good example. But now, you need to have a
well, to really run it sufficiently, it doesn't generate any interest for us
I suppose it's about memory and just in video graphics, not as real
So as for the limit of how many skeletons there are, to be fast
we can take the hard limit of six. This is a practical limit
We cannot really put more than 15 people into the view of Kinect in specific
and fix them all. That's the practical limit of it. And so, again, the C-API is on top of the Center SDK
It's easy to wrap around other languages and there's the C-PART API
A body frame. So how are the body represented? The production of body structs
What are the body structs? Has the ID of that body abstract
has the bone positions in 3D, which is the vector, and has bone orientation as a so-called fraternia
I have a slide later on what a fraternia is, but it's very common in game development
and 3D development because it's very easy computationally to represent locations in a fraternia
The math behind it is a little bit funky. Think about it as complex numbers on steroids
This is the index map I mentioned before. This is the outline of the body that the Kinect summarizes
And you can use it, for example, for a green screen in the text
to show the background and so on. And input chapters. You can capture everything the camera sees
and you can play it off if you want. The skeleton comes 3D
so you can rotate that in and out. And it gives you the information for all those images
The architecture. How does the whole thing work with body structure? So it starts with the Azure Kinect development kit
On top of that is the sensor SDK. And then we have the camera screen
The active infrared camera screen, with just a black and white image
is used to do convolutional neural networks on the Onyx front side
to then get two-dimensional joint coordinates. For information, there's also, for regular web cameras
there's also an Onyx model you can just download. You can just load it if you want, or things like that
It gives you 2D joints, if you want to write little apps that are going to just activate
And it gives you segmentation maps. So this is just two-dimensional. So here the depth image comes to rescue
And it gives you a note all about the distance from the camera and face
And then we do a step called model fitting. Model fitting basically combines those joint coordinates
from 3D with the zero distance and face. And beautiful knowledge about the typical human anatomy
about how far something will be. So even if the sensor screen
might be a little bit messed up, because it knows how a body is
that the arm length doesn't suddenly change between two images. You can use that to get good approximations
of 3D joints, even if it's what we call occlusion or partial occlusion of the joint
where the camera cannot see it. Yeah? So processing, well, you get the capture object
from the SDK, and then you execute this capture to be processed on the GPU
which you will use in your network. And then the GPU takes over that result
and does some model fitting. And that result that ends up in a queue
And the application developer will pop the result of operation from the queue
Yeah, skeleton 3.0, a little bit more detail. Those are the coordinates that you get
So, the placement of the landmarks has improved and positions and take the landmarks
and so on, and so on, and so on. Normally, I can show here, by the way
one of the quirks that Kinect for Windows version 2 had, if you would look at a body and look at the hip joint
and that body would raise their arm from low to high, then the hip joint
and you would also move up as the arms are raised. Even though the body can be resting at the pain
this is something that also the infections in Kinect were added. A joint definition, this is how it roughly looks like
It's a float, presents a vector from the position, and it's returning for the orientation
with the coordinate, you know, float. And that gets the joint. What can you do
And sorry that I'm roughing over the concept a little bit, we have no time for this
The able skill to do certain things. For example, kinematic analysis is analyzing all those motions
all those things you can see from a perspective of understanding what's going on with the human body
Analyze the posture, then use it for physical rehabilitation to see do people do what they're supposed to do
and calculate mathematically what it means in terms of the camera
Can be used for fitness apps, or for sports instruction. A lot of people ask about the golf swing
can I use my golf swing? It's theoretically possible, I would not recommend it purely
because part of the golf swing is a very, very fast motion, and that might not fit completely
into the 30 frame, maximum frame that the camera has. But other types of sports
maybe you want to trust people that are performed, meaning the body posture, exercises, very good
Can be used for patient monitoring of people, where they are supposed to be
and are they in bed, do they move, do they breathe? Does their chest move when they're lying down
Really does it help? Often used for fall detection. I can talk about fall detection a little bit later
maybe towards the end. Human understanding is a different scenario, as we're still looking at human bodies
but we want to not understand something intrinsic to bodies, but their behavior
How long do they stand in front of a specific shopping district
How often do they go in a specific area so we can detect them
How can we distract people as they move to a place? Maybe we want to know where they are
or maybe we shouldn't be. A very common scenario is also in the industry
If you have a machine, and nobody could be near that machine that the machine is running
you could have a Kinect sensors that, for example, only able the machine of bodies
are only outside that area, but not inside the area. And you can use it for smart faces interaction
This is similar to the next slide, which is human interaction. You put interactive information signs
and video walls, wave to the video wall, and it does something back to you
Interactive audience form is to just transform all those sensor data into something creative
You use it for intense machines, and for this, and for a custom of having a fitting machine
basically, with social tools. These last topics are pretty related in what we can do with them
Like a little walkthrough about body-fragmenting CD viewer. Yeah, I mean, well, actually
let me use a pen, actually, to show you the 3D body-fragmenting
Similar to how you had installation of the SDK, you also have the installation of the body-fragment
And a simple man-slide app here that you install in the folder
When you install that, it's popping up. Yeah, it pops up and starts with 3D visualization
And you can move that around, as I said before. Figure. Figure
Maximize this. Let me go into the picture. So the camera starts seeing me
and it starts taking my point, as I move around. As you
if I change that angle a little bit, so now it sees me from the side
but of course it doesn't know the picture on my side. So it doesn't also know what's behind me
because it only sees me from the front. All right, so, from a coding perspective
how does it all start? The first operation is really, hey, give me the device configuration
verify and start the camera. So let's start with a little demo
as it stands for basic. So we have an imager. We have
this, this one here. This one here. I guess, yeah, okay
Oh, I already have the device. What about that? So
you see that here, there'll be, press to open the device, and I hope actually that I
yeah, it brings my demo. So it's, press to open the device
there's a short arrow, so you cannot open the device here. Then it starts the camera
with the device configuration setting, which is about the parameters of the API
Again, this is a pre-start web work. This is an operation that can take a second
So, yeah. So it starts the camera, and I'll just add on to camera
and you can get the metadata from the camera. So we know the resolution is 1920x1080
And we initialize the disk map. And then when we, when the component runs
the disk map is, the disk map is, so I can't move the window yet
because I have a break point here. But essentially what it does
is then uses the API, gets a capture object from the API
And it's not right now. Okay, that's the Murphy's Law
Try to restart the video. Okay, yeah
Yeah, so, so it repeats the capture object, and has a device timestamp
which you get from the capture. This timestamp you can use in applications
to establish map based on time. We can also calculate velocity, or how fast something moves
to check the difference between timestamps. And you can calculate the distance of movement
And we can use the device timestamp. It also identifies the image unit
And then as a result, it takes, basically it writes the buffer
it gets back from the API into the disk map. So, this is something with this code
but I don't know, it's kind of a debugger. So it marks the disk map as 30
and then displays that. So let's do this break point, so you can see it happening
And this page here is pretty boring, actually just a video camera
storing the video. But the fundamentals are the same for everything. It's pretty straightforward to access the sensor data
from the database engine. So let's look at body tracking as a more advanced example
because we don't have time for all the times that are possible
Again, the process is the same, right? I mean, you start the camera, you create a body tracker object
You do the query capture, which is the object you get back from the camera
And remember, here you need both the desk screen and the information data screen
When you take any capture, then instead of any capture, you skew that capture to the body tracker
And then after that, try to pop the latest results from the body tracker
which could happen only in the next frame of processing, or just once in the background
And then you process that result. Yeah, that's about it. Well, the demo will come in a little bit later
So again, input capture, you get this selection body trust, and then you do the body index map
This is an enumeration you get as an application developer, which is very helpful to let you do the operations
based on the structure of the body. So you can say, hey, give me the distance between the elbow and the wrist
or the back of the elbow and the wrist, but there's no way that's pointing based on
those positions of the, feeding positions of those indexes, right? Again, we're going to have to make an echo test
So now let's do the demo application. So if you see
here's what's the same as above. I start the camera, I do the device calibration
Then I initialize the tracking, which is the components of body tracking
And then I do the sample capture, right? And here, so that when I break one
it starts right there. And after that, we try to get the results from the tracker
So you see the tracker's initialized, and we got the capture app object
If you look into it, you see it has a depth image
Yeah, that's all the goodness and the IR image. You can just use them both as you want
And then we include that. So at one point you get the results from that
So let me remove this from here, so we can actually get a look at the results
as we get back to the tracker. Yeah, so you got a frame back
and the frame actually, sorry. Here. Here, the frame might be here now, right
So that means all we asked was the tracker, but it didn't get anything back. So at the one point, the processing will
leave it, and see the frame, get the frame back finally. And the time spent, but there's no bodies to be found
Just for the simple reason that I haven't been in the camera
So let me use the breakpoint. And here, so this is the video stream
based on the depth image, the photo depth image here. And now let's just break it in
And it's, yeah. And it's in the visualizer, it's at the frame data
Now let's look at the visualizer. This is the visualizer. It's basically checking the track of the data
It actually moves into the picture, and I hope it's, forget the breakpoint
Okay, I didn't forget the breakpoint. Yeah, so here's a bunch of math
that's happening in the render window. And in the render window, like we looked at
hey, did I get any number of bodies? Here we got a number of bodies. And we can look at, now let's get that body
That's the skeleton and the body ID. We set the body color
And the skeleton is, has pretty good drawing. Yeah, it's pretty good
And so it's a buffer here. So we can look at the skeleton
it's a buffer here. So we need to actually unwrap that buffer
and get the specific drone that we wanna have access to. I mean, the API lets you do it
but again, the wrapper itself doesn't provide a good inspector experience
So that's one of the downsides is that you can use the body ID once
So let's get one of the drones. That's the first drone here
So, my mouse has a strong enough battery that it will take it later on
So we get it going. And, yeah, that's the drone's position
It's a vector, yeah? And the vector is the coordinate. It's the coordinate
And the orientation is the coordinate, which has X, Y, Z, and W as parameters
And then, so we do some process, all those drones that we get
So this is three dimensional vectors and orientations. What we do is basically
so every drone type A, do we also have the value of the parent
meaning, for the wrist, I need to elbow to be able to draw the lower part of the arm
And then let's render those two drone positions into the space. You walk into the image
And of course, I select the WUHOT breakpoint and follow the instrument
Yes, okay. You walk into the image. The problem is you just have a way in to see
So it renders what I mentioned before. So this is a wrist drone. This is the elbow drone that renders that
Those lines take from the data to get indexed into the poly-fragment
Okay. Okay. So back to the slides. So let's discuss briefly an example of use case, right
I mean, what can you do when you have a system like that
On an abstract level, you have a system that the files are not completely updated
I should say, unfortunately. But on an abstract level, you have a central system and you have a high level of central processing
And then you can do specific things. And you have, you can do posture classification on it
and you can have, do something activity in fall detection. How do you do all this? Right
I mean, you have a concept of the world. You have physical formulas. As a model, understanding the body
the parameters, and there's math to do on that body, like calculation, velocity, acceleration
Then do the real life calculation. Let's look at body going from acceleration
What you wanna do, you wanna look at the tracking state. It's available for each of those zones
So you get a certain level of confidence how well are my tracking
You wanna do math on it. So you wanna maybe convert it to a numerical form
of convenience for the math you're doing. You want to understand the drawn hierarchy
as I mentioned, the list and the other. For example, and then do all those math and so on
In general, you want to read about gaming engines and physics engines and game development
Those are the best references to do math and study data. As a guidance
what I used before is I used the model from the NASA
of a very typical body and the distribution of weight in a body
So to do math on, let's say the energy that's being used on the elbow joint
you need to understand the length and how heavy is the arm
upper and lower part of the arm. And they got this by looking at models they used
to calculate post-tune astronauts or astronaut uniforms, or astronaut cases and so on. Yeah
And then you make them a construct. Okay, like a bone is supposed to rotate
and then you can calculate an inertia by formula of this. You can use multi efficiency
efficiently open this function to do a dense. This is the side of a quaternion
If you overload it, it's a statistical parameter. You can see the position of the quaternion
that enables you to project things onto it. You as a patient can provide it
I've got wonderful YouTube videos about it, what quaternions mean to the patient
In 2020, the best thing I can tell you, look it up on YouTube
quaternions to patients and you will, you're eye to the open. Yeah
Now about the math, you can do linear motion and angular motion. Those are formulas you can use
from velocity to force, force energy. And it matters if it will go in a second
This is a video I did of an app I did looking for the old connect center
And you see that those little bubbles represent energy calculations. I did take the most
like the green one is the angular motion and the yellow one is the absolute linear motion
You can see that they have differences. I thought it doesn't move
when I lower the arm, raise the arm like this. But there's still energy to be spent
because I move my arm. So you want to do math maybe
by comparing both. Those different vectors, the line, the acceleration and the velocity
One thing, it is a center-based system, so you need to handle something called divots
Every center system in the world, including computer-based systems, they do not give you
consistently super precise readings all the time. There might be a difference
in one pixel or the other between two frames. And if you do math on that
like acceleration, you take the distance of velocity between two frames, you get errors
There's errors that get worse over time. Here I smoothed, I used the so-called double exponential filter
And you can see those lines stay relatively, well, even as I move slowly
Now if I were to move that, so there's a centering here
and do that same motion again, it's a bit jolting as I'm talking to it
So you see how everything jumps around. So this error in this would be
too much exponential, and then you, yeah, the two would be less than that
So, simple as that. Again, the last slide I will just run through
You can calculate activity. You can calculate activity levels if you first calculate the energy expenditure
based on the positive and positive flow. You can do all these types of different scenarios
thinking about, for example, how fast does my head move towards the floor
If it has a sudden fast movement towards the floor, and you know the position of the floor
and suddenly stops moving, maybe something happens. Or it could be slow flow. Maybe you just want to see, hey
a body has been on the floor for quite a while, something might be wrong
In general, quality detection is not easy at the beginning, because there are a lot of false positives
Somebody might go down to the floor to pick something up, or take a nap
This again could distinguish from an actual fall. Or somebody might bump on the sofa on the bed
because they're tired, and that bump might be classified as a normal fall
Popper, also very simple, and already complex. You can use advanced machine learning
or you can make very simple assumptions. You want to detect a popper
Now, look at the angle of your leg towards the floor. Look at the angle of your leg towards your hip
And you get some information. If your upper leg is like this to the floor
and your lower leg is like this to the floor, your spine is like this to the floor, maybe you're sitting
As the head may stand up. Posture is simply sitting over there
And then this is a summary of all the math I did in the previous application
I validated energy expenditure estimation with the view of motion sensing arm bands
The gold standard would be, actually measures the CO2 that are produced in our exercise
but I didn't have that. And I found that what I'm doing with the Kinect
is similar in profile. What I'm doing is calculating the profile as those arm bands would capture
But there is some differences. So it depends on how you do motion type
So if you write an application, what you want to do eventually is to correct, not just with the formula
which I did in my case. I corrected my formula and made it a little bit better
But in the end, what you want to do is you want to add formulas for those energy expenditure estimations
or detecting motion in general. It's depending on the type of motion we expect
That's what we tried to do. But that's what a lot of content to cover
a lot of ground to cover. And I hope it helps you to understand a little bit
how to develop software, how to get to ask questions for it and what you can do with it
Yeah, this is all I have in the content. If there are any questions
I'll allow the moderator to ask them now. I want to thank the sponsors again
Really, really thank the sponsors. And there's also feedback you can provide
All right, I think two minutes over so far. If there are questions, please address them now
Hi, Andrew, it's Aru here. And your question was really interesting. Very futuristic kind of thing we are able to see
And very good, the efficiency of body tracking was very amazing. I just want to ask you
for people who are in the, want to learn more, how we can come with learning resources
How we can do some investment to how the cameras are? So what kind of things we can do to learn more
Well, the good thing, as I mentioned, everything is on GitHub. So all the examples I showed
are available as source code on GitHub. So, and let me actually GitHub
X SDK. Just give me one second. Very exciting. Okay, GitHub
Example. Okay. So I'm showing you my browser right now. So this is the GitHub repository
You can start really just like understand even how the sensors became more developed
And there's examples in the sensors that you can use. There's also ways to write
There's also documentation, documentation, as you connect, as available as this landing page here
It's a very, this is basically start with everything. And this sends you to the download for the SDK
to your first application. But it's available on GitHub. And there's also a test repository available on GitHub
for all those connections. Yes, I mean, all this content I gave you
and the types of stuff that I developed myself, it's all available publicly
I mean, and beyond that, you really need to look into a gaming engine
3D maps and quaternions and all that. Is that a good summary that helps some of the audience
to get started? Yeah, yeah, it's very important because if they want to learn more about the mixed reality
and how it works behind the scenes, it's a springboard for them to learn new stuff
how they can do themselves. So is it, this is the most important starting page
to just, to, you will be able to get to pretty much everything through that page
So, I mean, let me, please, I think that's my spot
That's the connected page. All right, are there any further questions? Because I understand that you might need to get
the next speakers ready and so on. Yes, Andrew, I think this is
a lot of stuff is amazing. And we just wait for other speakers
And thanks a lot for your amazing demo today. Oh, yeah, thanks for having me
and thank you for your patience through some breakpoints and bug issues
But, good things happen. Thank you so much, Andrew. All right, thank you
Hope to talk to you all soon and find me on, email me or find me on social media
if anybody wants to. Yes. So, be good, yeah. Thank you, thank you for. Bye-bye