Demo: Building the world’s biggest database at Microsoft Inspire 2019

Demo: Building the world’s biggest database at Microsoft Inspire 2019


>>For the purposes of this demo, please imagine that I’m an operations manager in
a European auto company. We’re going to launch a new line
of autonomous vehicles shortly, and our team of solution architects
has been working on an application to monitor
these connected cars. As you see on the screen, we have a few hundred connected cars being monitored already in Europe. Let’s deep dive into
one of the cars to see what type and volume of
data we’re collecting. As you can see on the screen, we’re collecting hundreds of
gigabytes of sensor data every day. Now, to support a scenario
like autonomous driving, the data processing engine inside the car needs to support two things. First, it needs to be
Edge-Optimized to run really well on the intelligent
control unit of the car. Second, it needs to
be powerful enough to support real-time decision
making right inside the car. Azure SQL Database Edge is
the perfect fit for this scenario. It’s still supports native
scoring right inside the relational database to
enable real-time decisions. It can handle tens of thousands
of events per second. It shares the same relational engine as the SQL Server that
you know and love, enabling a level of consistency
for the developers across the intelligent Cloud and the intelligent Edge that is
unparalleled in the industry. Best of all, Azure SQL Database Edge, a fully functional performant
relational database now can run really well on ARM devices like the one
I’m holding in my hand. We’re also streaming
a small subset of the process data in every car
to a relational database in the Cloud to generate
operational insights on every car and across
the entire fleet. Let’s take a look at
a dashboard that captures the peak key performance KPIs of this relational
database in the Cloud. As you can tell, we
are currently running this application against
AWS RDS for SQL Server. More and more cars are coming
online in our simulation and the volume of data being
ingested is growing rapidly. As the database grows, it’s become very clear that we’ve reached the limits
with the current design, which cannot handle data volumes
greater than 16 terabytes. As we’ve reached these limits, all the right transactions or
the updates to the database fail, which essentially means we cannot add anymore cars to the fleet
being monitored. This is a very common challenge
with all relational databases. Managing very large relational
databases is a complex challenge, and more often than not, developers are left with no choice, but to completely rewrite and
rearchitect their application, with concepts like
application level database sharding which are very complex to get right. The reality is there is
a very clear need for a relational database that can
break through all these limits. Do you guys think we have
such a database in Azure? Yes, we absolutely do with
Azure SQL Database Hyperscale. Now, let’s take a look. I also have the same application
running against Hyperscale. Let’s see how that performs. As you can tell, you’re very easily able to go past
the 16 terabyte limit, both the read and
write performance of the application workloads are doing great and the database is healthy. Well, we’re currently
monitoring just a 1,000 cars. What if we wanted to
monitor a million, can Azure SQL Database Hyperscale
handle that? Let’s take a look. This is the part of the demo, I start feeling a little
nervous by the way. We’re going from 1,000 cars to a
million. Let’s see if that works. Well, a million cars is not a problem for Azure SQL
Database Hyperscale. As you can tell,
the number of cars are rapidly getting added to
the fleet being monitored, the read and write performance
is looking healthy. Now, of course, it’s going
to take a while before all the million cars get to
a steady state of data ingestion. Now, in the interest of time, we actually run this demo
earlier against the different Hyperscale
relational database and got it to a steady state. Any guesses on how large a relational database backed
by Hyperscale in Azure is when it gets data being ingested from a million
cars? Let’s take a look. It’s a little over 200 terabytes. That’s 200 terabytes in a single relational database
backed by a Hyperscale in Azure. Now, you may be wondering what’s
the magic behind Hyperscale? There are three key pieces
of innovation that we worked on for
the last several years. The first one, we’ve
completely rearchitected the core database engine to separate out the query
processing layer, compute layer, from the storage
management layer to be totally Cloud-native so that each layer
can be scaled out independently. Now, this is an extremely hard
challenge to get right for relational databases that support high performant
operational workloads. Second, we built out an intelligent caching hierarchy to leverage the memory hierarchy and
SSDs in a very efficient way, to get you the best price performance for workloads of any size and scale. Finally, we have technology now that guarantees that
databases of any size, scale, and state can
recover in constant time. We made all these
groundbreaking changes while maintaining
complete compatibility with the SQL Server ecosystem
that has served our customers well for
the last 25 plus years. Now, some of you may be wondering, what about core database
operations like backup and restore on
these very large databases? Well, backup is a managed service. We utilized snapshots,
it’s instantaneous, and almost has zero impact
in the higher bandwidth. But what about restores? How long does it take to restore such a large database
backed by Hyperscale? Well, my friends, it
doesn’t take weeks, it doesn’t take days, it doesn’t even take
hours as you may be used to as managing these
large databases On-prem. With the Hyperscale architecture, it takes less than 10 minutes. Now, finally, we’d like to
provide personalized insights to all our millions of users of
these autonomous vehicles. Now, you can imagine the amount of query
processing power you would need to actually get that to
scale on a relational database. Let’s see if our Azure SQL Database
Hyperscale can handle that. Now, what you’re seeing
on the screen is the Azure portal experience
which allows me to create 30 replicas on a single relational database
backed by Hyperscale. To put that in perspective, that’s 2,400 cores of processing power on a single relational database
backed by Hyperscale. With that amount of
computing power in a single relational database,
you can handle, you can easily satisfy
the scenario to give out personalized recommendations
to all the millions of users. Azure is the only public
Cloud that supports scaling SQL relational workloads
at this scale and magnitude. Our customers can build
their solutions on Azure with confidence that we’ll be able to meet their needs today
and in the future. We can’t wait to see what kind of innovation you build on
the Azure data platform.

Danny Hutson

8 thoughts on “Demo: Building the world’s biggest database at Microsoft Inspire 2019

  1. HI,
    I found one bug in SSMS 2017 .I am not able to reach to exact microsoft support .Please look onto below bug .I have written one case statement, Where i am checking if network_type='medicaid' then display corresponding Effective date else display blank. There is no any network_type='medicaid' in table.

    The bug here is query going into else part and converting varchar blank to date instead of showing

    ''Conversion failed when converting date and/or time from character string".

    If same blank string i changed with something else like 'ABC' then it is showing correct error.

    Please update it for blank also. For blank it should display error instead of converting it to datetime.

    Below is my query

    select

    Case when NL.NETWORK_LICENSE_TYPE='Medicaid' then NL.EFFECTIVE_DT else '' END as [Start Date]

    –,Case when NL.NETWORK_LICENSE_TYPE='Medicaid' then NL.TERMINATION_DT else 'xyz' END as [Start Date]

    from D_NETWORK NL

Leave a Reply

Your email address will not be published. Required fields are marked *