dbatools - new functions for Database Mail (2021-11-01)

Daily Coping Tip

You can’t do everything. What are the top three priorities this week?

I also have a thread at SQLServerCentral dealing with coping mechanisms and resources. Feel free to participate.

For many of you out there working in a new way, I'm including a thought of the day on how to cope in this challenging time from The Action for Happiness Coping Calendar. My items will be on my blog, feel free to share yours.

Moving to DataOps

I've worked in a number of organizations and every one of them was constantly searching for a way to better ensure data is up to date and consolidated across systems. There has been no shortage of projects trying to assemble a "single view of the truth." Using ETL, data warehouses, external master data systems, and more, companies somehow want to ensure that they get the most value from their data.

That sounds a lot like DataOps, which is discussed in this piece. It's trying to more efficiently ensure that we manage data data better, getting it to the right system, updated, and safely. It's not quite the same as normal data operations, such as running jobs and fixing data a user broke. This is more like the overreaching we in which we move data through a pipeline.

In some sense, as a build pipeline is to the software, DataOps is to the DBA's job.

In the past, most organizations were not very coordinated in how they managed data. Instead, each application or each team was responsible for their own data, often without much communication with others. I think a lot of master data management projects in the past haven't gotten much penetration inside a company for this reason.

These days, with more governance and concern about security, I see a renewed interest in better managing data, as well as ensuring metadata is up to date and used. Companies are realizing that they can get more value from data when they know more about it, especially when the metadata can be queried programmatically. This also helps organizations ensure they are aware of their data assets, understand their risk in how the data is handled and used, and even be aware of when data might cross a security boundary.

Much of the XOps world is concerned with automating the process. CI automates the build and test of software, reducing the load on developers. In the DataOps world, I'd expect we could automate a lot, but I wouldn't think we could do it all. The structures of data assets, the ETL processes that move them, and the ways in which people consume data will constantly change. Often at a pace that isn't easily automated.

Steve Jones - SSC Editor

Join the debate, and respond to today's editorial on the forums