Column-level lineage is not just another piece of software; it's a pivotal element in how your organization manages, understands, and utilizes its data. It impacts everything from data governance and operational efficiency to your decision-making processes.
However, realizing you need these tools is only half the battle. The real challenge is convincing your organization's decision-makers to invest in them.
To help you make your case, we asked Jisan Zaman, the Data Engineering Manager at Xometry, how he successfully advocated for such a tool at his own organization. Here’s a sneak peek at what he had to say about building a business case to get the buy-in you need.
Why did you implement a column-level lineage tool (vs a table-level lineage tool)?
We chose a column-level lineage tool over a table-level one because we understood how important our data accuracy was. With tools like Snowflake, Looker, Tableau, and dbt already in our data stack, it wasn't enough to just track tables; we needed to see how changes in one part of our data affected everything down the line, right down to each individual column.
This visibility ensured that the data feeding our processes was accurate, relevant, and reliable, so we can make changes faster and more safely. Plus, it was easier to show other teams how their work impacted the overall data flow.
How did you translate the data team’s needs to larger business objectives?
Previously, unexpected data issues led to resource-intensive troubleshooting and costly downtime. Now, because we can track our data transformations down to the column level, the team understands and manages our data stack much more efficiently.
We experience fewer errors because our data is more accurate, and when we do experience errors, it’s much quicker to debug the root of the issue because of our clear visibility. This reduces both the time and resources needed for fixes which only further improves our efficiency.
We tied this error reduction to its direct business impact—cost savings, improved performance, and therefore, improved customer satisfaction.
How did you build a business case to convince stakeholders to buy in?
We focused on a clear, tactical, and demonstrative approach to get buy-in for our team:
- Engaged a technical leader (in our case, the CTO). He was more likely to understand exactly how the product worked and why it would benefit the organization.
- Presented a demo to show how the column-level lineage tool worked in real-time, highlighting its time-to-value, accuracy, and ease of setup.
- Tailored this demo specifically to the executive presence, emphasizing specific advantages for our data and business teams. We then tied those team advantages to business ROI.
- Emphasized the cost and resource efficiency of buying a tool compared to building a similar solution in-house, considering our team's existing workload and responsibilities.
By showcasing Select Star’s ability to prevent issues like recent data breakages, we provided tangible examples of its value—and the CTO quickly bought in.
What follow-up did you need to validate the efficacy of your column-level lineage tool?
We didn't need extensive follow-up. We validated our purchase by proving Select Star’s value in our daily data operations. Besides helping us complete error-free data migrations, it also helped us shift from a reactive to a proactive data management approach.
Column-level lineage enhanced the planning and execution of all our data migrations and deprecations with clear documentation. That documentation helped our data team support business partners make more informed decisions—and that was all the validation we needed for Select Star’s efficacy.
How does column-level lineage remain relevant and useful as your organization's data architecture evolves?
The best part of Select Star’s column-level lineage is that it enables us to evolve our architecture. We couldn’t make changes efficiently without it. We actively use it to facilitate and manage changes in our data systems, and to transition from outdated, less efficient systems to newer, better ones. For example, we recently deprecated an old business layer database, and column-level lineage helped us confidently identify and communicate with the few users affected by this change.
It's integrated into our daily operations, constantly updating and providing clear insights into our data flow. This integration is crucial when making substantial changes or updates, like modifying sales commission reports because it offers complete lineage visibility for accurate and informed decision-making.
And to keep it top of mind, we're leveraging all its new features and updates like dbt docs Sync and AI-generated documentation. This ensures that our data management processes remain robust, adaptable, and aligned with our evolving data architecture.
When is a data lineage tool most useful for an organization?
You have to feel a bit of pain from your data complexity first. You don't necessarily start with a lineage tool right away. It’s more top-of-mind when you have enough data and your systems are getting out of control—when you're using various systems and realize you don't have the manpower to connect all your BI APIs and manage your data warehouse effectively. Or, in our case, when you've experienced issues like data breakages or find yourself overwhelmed by the sheer volume of tables and data processes. It's important to recognize when there's a lack of understanding of your own system—that's a key indicator it's time for a lineage tool.
We waited until we felt these pains and understood our needs clearly before deciding on a lineage tool. You should see the clear value of the tool before trying to onboard it.
In retrospect, is there anything you would have done differently during the selection or advocacy process for the tool?
One thing I would have done differently is to make sure I stay up-to-date with the tool's evolving features. After we got everything up and running in the first year, we didn't fully utilize some of the newer features, like tagging and automatic descriptions, which could have added more value to our processes.
What advice would you give to other organizations considering a similar tool to address their data lineage challenges?
Understand the scope and depth of your data challenges first. Make sure you choose a data lineage tool that aligns with your existing data ecosystem. A lineage tool should complement your data governance and metadata management efforts, not replace them. It can help highlight which columns and tables lack descriptions, moving your governance forward (rather than slowing it down).
Use column-level lineage to gain better insights into your systems, identify redundancies, and streamline processes. And remember: while investing in a column-level lineage tool is an upfront cost, it saves resources in the long run. Engineers are expensive, and you want them working on high-value tasks, not bogged down in debugging. Column-level lineage helps shift your team from a reactive to a proactive stance, which is a significant cost-saving aspect in itself.
Book a demo to see how Select Star’s best-in-class column-level lineage fits your organization’s needs!
Want to hear more from Jisan? Watch his full interview now!
