How to Set up an Agile Workflow for Data Management

By Gilad David Maayan

A traditional workflow follows a waterfall process. It is linear with each phase relying on the completion of the previous phase before it can be started. This process is highly structured and requires significant planning to implement. This highly structured nature makes it difficult to adapt to changes mid-project and create significant delays or duplication of work.

To improve upon the traditional process, agile was developed. Instead of trying to fully plan and implement a static process, agile workflows focus on dynamic iterations. Each iteration prioritizes a small portion of the larger product. Once an iteration is finished, the next is planned based on feedback from the previous iteration and changing project requirements.

In this article, you will learn how to apply agile values for data management, including process goals, real-life examples, and a quick guide for setting up your own workflow.

Agile Values for Data Management

Although agile was designed for development processes, you can apply agile practices to a wide variety of projects, including data management. Some of the most useful values are explained below.

Evolution over definition

To remain competitive, you need to be able to efficiently and effectively modify data sources to fix quality issues or adapt to changing functionalities. The challenge is that this ability needs to be balanced against the benefits and requirements of having well-defined metadata and data models. 

Sufficiency over perfection

While it may be tempting to try to perfect your data management methods, it is a mistake to do so. First, perfection is not possible. Second, focusing on perfection costs more time and effort than the rewards it pays back. 

Instead, you should focus on ensuring that your data management processes are “good enough”. Production databases can and should evolve as you go. Rather than trying to predict any and all needs a database may need to serve, focus on integrating feedback over time. In the meantime, “good enough” methods enable you to progress with your work faster.

Collaboration over documentation

To ensure clear and effective communication, focus on face-to-face interactions instead of documentation. This can help ensure that your team and data users are working collaboratively and that everyone is on the same page. It can also prevent you from endlessly creating and updating documentation. 

This doesn’t mean that documentation should be ignored entirely, however. It is still important to create clear core documentation that can be accessed by all stakeholders. The key is to focus on information that is directly actionable. 

Automation over manual processes

Automation can increase productivity and standardize processes. It can also help eliminate bottlenecks by streamlining or removing authorization processes. For example, you can use scripting to verify the format of data prior to integration. This eliminates manual verification process while ensuring that data remains reliable. 

Process Goals of Agile Data Management

When implementing agile practices in your data management strategies, it helps to have clear process goals. Some particular goals to include are covered below. 

Constantly improving data quality

There are multiple methods that have been developed to help you improve the quality of your data. These include database refactoring and continuous database integration. Whichever methods you use, the goal is to collect and apply feedback on current processes and performance. You can then apply this information to improve future procedures and methods.

Developing data assets

Data assets represent your efforts in data collection and management. These assets need to be refined and protected to ensure that data is as productive and valuable as possible. Protecting and improving assets requires continuously knowing both what data you have and where it is stored. 

Redefining data structures

To ensure that data can be used effectively, you need to ensure that it is fit to purpose and highly accessible. This involves adapting data structures and access to meet current needs. Additionally, data should be easy to find and manipulate regardless of the format you store it in..

Real-Life Example: Agile Workflow for a Cloud-Based Database

To better understand the challenges of incorporating agile practices in data projects, consider the following situation. This example project focuses on building a database in the cloud used to analyze data and share reports. 

This sort of database requires extract, transform, and load (ETL) processes and data investigation and analysis that your customers will never see. While the end result (the reports) has obvious value and can be easily prioritized, the work needed to produce the reports can be harder to organize and define. 

Putting these work requirements into agile terms can be done in one of two ways:

• I want to create an ETL process to load data from the application to the reporting system so that you can easily manipulate data.
• I want to enable users to view active customer records in the reporting system so they can create reports based on a list of customers.

The former more clearly defines what needs to happen to enable the reporting the customer needs but doesn’t focus on the end result. The latter doesn’t lay out specifics but prioritizes the required end functionality and is better suited to agile goals. Although the second story doesn’t break down the task, it also doesn’t restrict the team to a specific solution.

Determining priorities of tasks is also an issue with projects like the example given here. While the development team may view the creation of ETL processes as more important, the customer may prioritize a new report or interface. To resolve this, you need to rely on a product owner.

A product owner is a person responsible for negotiating and balancing the needs of agile teams and customers. To correctly prioritize, a product owner needs to both understand the implications of the tasks they are prioritizing and the limitations of the agile team. To achieve this, teams and customers need to be able to clearly and reliably communicate. 

How to Set up an Agile Workflow for Data Management

When implementing agile practices in your data management workflows, keep in mind the end goals of agile. You want to continually improve your processes for greater efficiency and values.

One method for incorporating agile you can try is the plan-do-check-act (PDCA) method. This method employs a cycle of the following steps:
1. Plan—identify the issues you want to address. You need to pinpoint your problem, analyze it, identify areas of improvement, and determine potential solutions. Take your time in this phase to ensure that you sufficiently understand the issue you are trying to address. You need to find the balance between having enough information to create a plan and getting stuck trying to determine every detail. 

2. Do—apply your solutions and improvements. You should implement this on a small scale until the change is verified to provide benefit. Making system wide changes can create bigger issues.

3. Check—evaluate and analyze your implemented fixes. You should compare current productivity or performance against past benchmarks. Make sure to evaluate if the changes you made are sustainable rather than a short term fix.

4. Act—apply your verified improvements on a larger scale. If you run into issues along the way it may be an indication that you overlooked something in the previous phase. Regardless of the outcome, make sure that you continue to evaluate your processes. Even after you have improved a system, you can likely continue to make improvements. 

Conclusion

Much has been said about the bottlenecks caused by data processes, but not much has been proposed to solve the issue. Yes, there’s been talk about DataOps, which is a suggestion to merge data management and operations. However, before data can join the dynamic workflows of today’s Ops, it will need to become more agile. Hopefully, this article provided you with concrete and practical advice on how to create agile workflows for data management.

Author Bio

Gilad David Maayan is a technology writer who has worked with over 150 technology companies including SAP, Samsung NEXT, NetApp and Imperva, producing technical and thought leadership content that elucidates technical solutions for developers and IT leadership.