Episode 5 - The Human Agent Connection

I got to join George Sims and James Woodall on their 'GoHAndsFree' podcast to chat about RealWear and the future of Human and AI collaborative systems.

I interviewed my AI self live.
No edits, no cuts, no pre-processed answers, just a short conversation with myself...

This uses a brand spanking new live interactive avatar from HeyGen Labs, and OpenAI with access to a small knowledge base about me. Now I can finally be in two places at once! HeyGen has already included Zoom integration, so it's easy for digital me to turn up to digital meetings. Fun for a prank, or revolutionary for 1-1 customer Question Answering, you decide.

DISCLAIMER - real me has never pronounced data like that and I never will 😤

I've been playing a little with AI generated videos.

I used a very basic setup, taking a photo of me, and extending it to the correct frame dimensions with photoshop's generative fill.

became:

Then I simply ran it through runway, and looked at the results.

This was the output with the least deformation of my face. (note I had a slightly different room option to start this one from).

But there were some more exciting examples:

and:

These include some fun, dynamic, if not a bit mad, movements from Rory the robot.

There is quite a lot of unwanted deformation here, and certainly, improvements to be made, but human faces and features are tough to get right, as our brains are incredibly good at detecting them and noticing things that go wrong. I am not a fan of the ageing process it seems to put me through when I move my head, though...

Some incredible things to note though, are the really accurate shadows cast on the wall from the robot as it moves, these sorts of details will really add to the believability as the tools improve, the pace of which by the way, is rapid.

I've recently become interested in AI, specifically generative AI, doing things it really wasn't built to do. I also love a good visualisation, so I combined the two things to teach GPT-V to game and then used this to explore how we can better outcomes from generic models, using a bit of sensible structure and guidance.
Using an open-source game example from pregame, I asked GPT-V to defend this open-source planet from the alien invasion.

I started by showing GPT-V a single frame of the game and asking it to decide whether to fire the gun, wait, move left, or move right.

The result is less than stellar. Although it can do basic movements, it fails to come up with any strategy, and just sits in the corner (this will be a common thread...)

In this example, the AI is only shown one frame. It has no capacity to remember previous states or previous instructions. Looking at the log just shows a lot of 'FIRE!' as it has no reference that it has exceeded the total number of shots. It gets stuck in the corner, and whilst there is a glimmer of hope as it attempts to run from the kill-shot bomb, it is ruined by the decision to retreat to the corner instantaneously to its death.

The next thing I tried was expanding its memory to 10 frames so that it could see its last positions and moves. This is always a good idea for added context, but it doesn't help in our case, as the 'hide in the corner' strategy that is reached by accident, is fundamentally flawed.

The next step was to introduce a new 'agent' to the scenario. This 'agent' is simply a piece of code that feeds GPT-V some relevant text-based information, like how many shots have been fired, how many bombs are active, and where on the screen the player is. (My hope was that it would recognise that it was getting stuck in the corner.

This slightly improves our outlook. We get stuck in the corner less, and the gameplay is a bit more dynamic. There is some movement to the shots, but ultimately, the vision AI fails to see the obvious threat above the player, and simply waits for its impending doom.

This is strange behaviour though, because if you ask GPT-V where the bombs are, it can accurately tell you that a bomb is in a threatening place, but the decision isn't made to move out of the way based on visual information alone.
This is where a truly multi-agent approach comes in. This time, I set it up so that not only do we have an 'intelligent' agent playing the game and a basic agent giving some live stats, but now I introduce a separate 'intelligent' strategy agent. This agent is the same GPT-V technology, but rather than ask to give an order to move or fire, it is asked to give a strategy and explain to the player agent where the bombs are and where the danger comes from.

This was interesting, as the strategy agent would add information it gathered from the frame to add additional context.

This additional context, genuinely changes the way the player operates. Keeping it more likely to be central, and move dynamically and consistently away from threats.

This time the gameplay is revolutionised. The player moves much more dynamically and knows when to move away from bombs. The corner strategy is still strong, but made less prevalent by the strategy agent. The player also seems to be much better at killing the aliens in this scenario than others. We see around score 17/18 the player actively moving out of the corner to avoid bombs, because it actually knows:

1. It will get stuck in the corner
2. The bomb is above the player.

All thanks to the strategy agent.

The player eventually gets trapped in a tight spot and loses the game, but I do think this fun demonstration shows that multi-agent approaches, using both smart and dumber agents, can increase the effectiveness just by using the same pieces of technology.

Next up in my interviewing AI series... inspired by the OpenAI robotic announcements a week or two ago, I wanted to see how far I could get to replicating their video. We've not managed a robotic transformer for our Hello Robot Inc Stretch RE1 (we named Rory) yet, but watch this space.

This is a quick interview powered by GPT4 on Microsoft Azure including GPT-V powered vision processing. The voice is powered by OpenAI's text-to-speech models. I built the robot a self-aware persona with a touch of humour sprinkled in.

An AI description of the video using GPT-4 Vision and Azure Vision AI:
In the ever-evolving landscape of artificial intelligence, the fusion of human interaction with AI personas has reached new heights. A remarkable example of this synergy is encapsulated in a video that showcases an intriguing interview between Fergus Kid, an R&D engineering lead at Avanade, and his AI counterpart. This innovative interaction not only highlights the capabilities of AI but also demonstrates the creative possibilities when leveraging Azure OpenAI's GPT-4 and the HeyGen avatar creation platform.

The video, set against the backdrop of a cozy and well-lit living room, starts with Fergus leaning forward, his curiosity evident as he prepares to dive into a conversation with a digital version of himself. The scene seamlessly transitions to a more formal setting where Fergus's AI persona is comfortably seated on an orange couch, embodying a serene and professional ambiance within what appears to be a modern office space.

The interview delves into the personal and professional aspects of Fergus's life, from his birth in Winchester to his current life in London, his hobbies like gaming and beekeeping, and his profound interest in space and AI. These details not only serve to humanise the AI but also ground the conversation in tangible reality.

The technology underpinning this fascinating exchange is Azure OpenAI's GPT-4, a state-of-the-art machine learning platform that utilizes advanced natural language processing to craft responses that are remarkably human-like. GPT-4's capabilities allow the AI version of Fergus to engage in a fluid dialogue, discussing complex topics such as the ethical implications of AI and the potential benefits of AI personas for society.

Complementing GPT-4's linguistic prowess is the HeyGen avatar creation tool, which has been instrumental in bringing the AI persona to life visually. The attention to detail in the avatar's creation is evident as it mirrors Fergus's movements and mannerisms, creating an immersive and convincing experience. The avatar's realistic presence on the orange couch, interacting with its environment in a natural way, underscores the potential of AI in humanizing technology.

Throughout the video, the interplay of light and shadow, the strategic placement of indoor plants, and the modern decor contribute to a setting that is both inviting and forward-thinking, echoing the innovative spirit of the conversation. The shifts between the living room and office scenes provide a visual rhythm to the dialogue, maintaining viewer engagement.

This video serves as a testament to the strides made in AI and avatar technology, specifically through the integration of Azure OpenAI's GPT-4 and HeyGen's avatar platform. It offers a glimpse into the future of human-AI interaction, where digital personas can not only replicate human conversation but also provide unique insights and augment human capabilities. As the AI Fergus aptly puts it, the key lies in approaching AI development responsibly, ensuring it paves the way for positive outcomes for humanity.

In creating such a video, the creators have not only pushed the boundaries of what's possible with AI but have also presented a compelling narrative that encourages viewers to ponder the role of AI in our lives. It's a blend of technical marvel and creative storytelling that beckons us to consider the harmonious potential of humans and AI coexisting and collaborating.

As we step into a future where AI becomes increasingly integrated into our daily lives, videos like these are pivotal in shaping our understanding and expectations of this technology. It's a conversation starter, an educational tool, and a peek into what might soon become a new normal in the interaction between humans and their AI counterparts.

Make your own: https://app.heygen.com/

One exciting thing I worked on very early on was creating my own custom voice font with Microsoft's custom voice font to enable text to speech. In the years since we have seen some amazing advancements in neural text to speech which give a much better output.

When I found out neural custom voices were available I couldn't resist trying one out. I was eager to see how accurate and natural the results would be compared to my previous attempts, which although recognisable, were not quite there.

For those who are unfamiliar, Microsoft Neural TTS is a cloud-based service that aims to provide highly realistic and expressive speech synthesis using deep neural networks. Voice Fonts, on the other hand, allow users to create custom voices by training the TTS engine on their own voice recordings.

To begin my experiment, I signed up for the Microsoft Azure Cognitive Services and followed the provided guidelines for recording my 50 utterances. The process was relatively straightforward, with clear instructions on how to maintain consistent pitch, tone, and pacing throughout the recordings. These are recorded right in the portal, which is super easy. I did use a much better quality microphone than my attempts several years ago. The training is pretty quick, but to use the font you need to record a short statement that you agree to have your voice replicated.
The moment of truth arrived as I typed a sample text into the Neural TTS interface and selected my custom voice. Listen to the results yourself:

An excerpt from Mary Shelley's Frankenstein

Audio file

0:00

/0:28

1×

This quick experiment has left me eager to explore further and see how much more realistic the Voice Font can become with additional recordings on the professional version, although up to 1000 are required. I'm also looking forward to seeing how Microsoft and other companies continue to push the boundaries of TTS technology in the coming years. It looks like there will be a version native in the next apple iOS iteration later this year. The future of personalized, expressive speech synthesis is here, and it'll be interesting to see exactly how it gets used... For good and bad...

As a self-proclaimed robotics enthusiast, I could not have been more thrilled to attend and present a late breaking poster at the International Conference on Robotics and Automation (ICRA) 2023. The event exceeded my expectations, with fascinating keynotes, talks, and demonstrations that showcased the cutting edge of robotics technology.

One of the most striking aspects of ICRA 2023 was the sheer number of robotic dogs running around the conference venue! It seemed like every other exhibitor had their own version of a robotic canine companion, each with unique features and abilities and custom sensor set. Some were designed for search and rescue missions, while others were more focused on indoor mapping with LIDAR. Regardless of their purpose, seeing so many robot dogs in one place was an unforgettable experience.

However, the most intriguing encounters at the conference were the human-robot interactions. As I roamed the exhibition hall, I stumbled upon a stall showcasing a humanoid robot designed to mimic human expressions and emotions from Ameca.

It was a fascinating and, admittedly, somewhat creepy experience to witness this human-like robot move so naturally with expression. It was a stark reminder of the rapid advancements being made in the field of robotics and how far we can come when we link this with technologies in like voice fonts and generative AI.

Another highlight of ICRA 2023 was the incredible array of keynote speakers and talks. Some of the most influential figures in the robotics and automation industry shared their insights on the current state of the field, as well as their predictions for the future. Topics ranged from ethical considerations in agricultural robotics to the latest developments in machine learning and computer vision.

A surprise at ICRA 2023 was the sheer number of people showcasing simulation technologies. There was a notable emphasis on using virtual environments to train and test robotic systems, with several exhibitors demonstrating their simulation platforms. These tools are becoming increasingly important as the complexity and capabilities of robots continue to grow, allowing developers to refine their creations in a controlled environment before deploying them in the real world. Some of these had fantastic graphical layers that can be used for training computer vision models with synthetic data.

ICRA 2023 was a robotic wonderland that left me in awe of the innovations and advancements in the field of robotics and automation. From the charming robot dogs to the thought-provoking keynotes and talks, the conference was a testament to the exciting future that lies ahead in this rapidly evolving industry. I can't wait to see what ICRA 2024 has in store in Japan!

OpenAI released it's model and API that powers the product we see when we use Chat GPT. This means we can start to implement our own solutions around the function by adding it into our custom apps or services. One thing I wanted to do right away was to get a version working for robots.

That's right, a version that will work with voice recognition and voice synthesis. Check it out on GitHub - it's a basic implementation for now.

I use the voice services from Microsoft Azure to take input from the device's microphone (could be your laptop, could be a robot) and send it off to OpenAI's  GPT 3.5 model for chat. The nice thing about this is that you can include the messaging history to allow for a more natural conversation, as OpenAI can "remember" what you have been talking about. It should work mostly the same, however there are obvious limitations like code generation. You wouldn't want anyone to dictate code to you for example....

Anyway, I hope this will be useful to someone learning about GPT and speech, and I'll enable it on our robot, Rory, soon, to make 'him' even more knowledgeble than before.

One very cool open source project has taken Stable diffusion into Blender. This means you can generate AI art, whether it be concept art, or a texture, right where you need it.

One epic feature of this project that was just released though is a new model that can deal with the depth it receives from Blender. So not only does it generate something new, it can also understand your model and wrap the texture around it. It seems to do this by asking the model for a mew flat image, knowing what surfaces it meeds to wrap. It makes for a nice addition, but can drastically reduce the quality of the image if you aren't zoomed in on your Blender viewport.

Here I've asked for a simple texture of a cereal box on a simple mesh. The output is obviously not perfect, generative AI has never yet been great at rendering text nicely, but it's very clearly a cereal box that we have rendered out.

As a distant background object this would be more than enough to quickly fill out a scene, or make interesting new models on the fly. It can use cloud compute if you have a Dream Studio, or locally on your own GPU.

Here it is in action, going from a blank cuboid to a fully textured cereal box!

Blender is using my Nvidia 3080 to render this nice and quickly locally. The nice addition here is that makes it completely free!

Gaming technology and the technology that supports it has come a long way. With more powerful GPUs in our devices than ever, revolutions in chip manufacturing, and efficiencies in calculations allow us to do incredible things like run ray tracing engines in real time, simulate realistic physics, and so much more. All of this work has huge impact on gaming and immersive experience, but outside of creative industries what can we do?

I've done a lot of work and learning around the metaverse. As part of this learning I've made worlds, designed objects, and learnt a lot about 3D rendering. One thing I did to support some of this research was to 3D model my own building for some experimentation around MetaHumans - more on that soon. Once I had a render of my bedroom behind my workstation, I uploaded it as a Teams camera background. After talking with a few people, I realised not everyone cottoned on that it was even a digital render (the real thing is much messier normally).

This gave me an idea.

If people can't really tell what's real once the stream has been compressed and sent over the internet, could a machine vision model? I've always been interested in synthetic data creation for machine vision, but usually my starting point has been something real, like a photograph. What if we could make a working machine learning model with no real data at all?

The use case I chose was to find a tennis ball. Why? Well, it's about the size that our stretch RE1 robot can grab well, and it won't matter if it's dropped or bumps into anything, so, I was hoping the resulting model could be used to help the robot find and grab the tennis ball in the room. That's a challenge for another day though.

I started off with the 3D model of my room, and settled on the view from behind my workstation.

Then dropped a tennis ball onto the bed. (virtually). I wrote a script that can move the ball around the room, change the lighting conditions, and then automatically tag the ball location and upload it straight to Azure custom vision.

This uses the cycles engine, with a de-noised image generation capped at about 30 seconds on my machine. The beauty of this is you can easily change or limit the quality output, or scale your compute power to generate many thousands of quality pre-tagged images much faster than you could ever photograph a real scene and manually tag it, assuming you don't have to build an environment from scratch like I did here.

Playing around with the lighting conditions allows you to simulate lots of different conditions, which from my past research, really help to build a more resilient object detection model.

All that's left to do is to click train, and test the results!

The real thing

I used the minimum number of images required by Azure custom vision, just to push it to the max, but in reality with this method, you can keep adding camera views, lighting setups and noising to you heart's content.

First, let's test a real image from a similar viewpoint:

There we have it. The model has never seen a real tennis ball, but can identify the real one with 93.1% confidence, even with the lowest amount of synthetic data possibly generated.

What if we show it an image it wasn't trained on, with more objects than expected?

Well, we had to reduce the confidence limit, but we have still correctly identified all the tennis balls, even one from the reflection in the mirror.

Next, the real thing from a different viewpoint:

Not the highest confidences again, but still correct guesses - so this approach is still useable for a first pass where the context of the images meeds to change.

Finally, the same object but in a totally different context:

So, in a totally different context, it still works!

Imagine what we could do with many more renders, with no meed for any manual tagging, and looking at as many objects as we liked. There are great things going on in this sector from the likes of Unity as well, but for me, Blender gives the greatest flexibility on 3D rendering, with beautiful rendering available completely for free.

We discussed AI generated textures using DALL-E-2 before. However there have been two significant updates.

1. The Dall-E-2 API is finally here!
2. I figured out how to easily get from a flat texture to a full 3D one using Materialize.

The Dall-E-2 API make it infinitely easier to modify image in the way we meed to. Instead of manually doing all the image generation, we can thread it into a single python operation.

I'll show you step by step what we generate. First let's ask through the API for a texture we might want to see. In this case I'm asking for a yellow brick texture.

Pretty good start. The problem is if we tile this nicely over a large 3D surface, it won't look good because the edges won't match up. It 'tiles' horribly when we replicate the image many times. These always will. Let's move the 'seams' to the middle so we can see it better.

All we've done here is move the outside in, by chopping the image into 4 and swapping the positions. Now we can see the seam in the middle as if we were looking at 4 tiled images. It's not the worst seam ever, DALL-E-2 has done a very good job, but we can easily tell it's not right.

The next step we can do is to create an alpha mask on this image as we know where the seam is, it's in a nice cross shape in the centre.

This is a representation of the mask, with areas to keep showing through (in blue_ and areas to re-do in purple. So what can we do with this? Well the great thing about the DALL-E-2 API, is that we can also edit images. Sending the original image with the seams in the middle off with this mask and the original prompt will allow us to regenerate the seam area for hopefully a lovely smooth transition.

And there we have it. You can clearly see some new artefacts like the blue smudge on the brick in the middle, where the mask was. DALL-E-2 has regenerated the gaps we marked out, so now the image is pretty much seamless and tile-able. Great!

Let's make it 3D

For this to really be useable it doesn't just need to be seamless, it needs to have depth, roughness, metallic layers. All sorts of additional information about how light would interact with this texture in the real world, so we can better simulate it in a render engine. I previously thought this would be a manual and labour intensive process until i found Materialzie from Bounding Box Software. This FREE tool takes 2D flat textures from a photo, or in our case AI, and makes them pop by generating the other information we need.

This is a huge improvement on our flat tile-able image. Let's have a look at what that looks like rendered.

And there we go! A now usable seamless 3D texture, completely AI generated. This opens up a word of opportunities for amazing new on-demand texture rendering!

Here's some more example outputs just for fun:

Notice we may want to do some colour stabilisation accross the segments on these in the future.

A good example of how things with regular patterns or directionality can go very wrong very quickly...