First Experiences with an Agile Data Project

Ryan Desmond   •   06.27.2019

I have worked primarily as a Database Administrator for seven years and in that role had never been an actual team member in the agile development process. My only exposure to Agile was attending some stand-ups at a previous employer, but as the DBA I wasn’t on a team, so I didn’t have hours or tasks. It was more to make sure I knew what the development team was doing so I could anticipate what they would need from our group and identify issues about what SQL Server could and couldn’t do depending on the version we were using.
In this experience, I didn’t know anything about Agile methodology. I had no idea how the tasks assigned to developers were created. I just figured that someone just knew everything!

In transitioning to a BI consultant, I knew that some clients would be employing the Agile methodology, so I did more research into Agile and was excited to learn how some of those tasks came to be.

Agile: Defined by The Agile Alliance as the ability to create and respond to change. It is a way of dealing with, and ultimately succeeding in, an uncertain and turbulent environment.

Ok, so Agile seems to be a process that allows flexibility, great!

I learned about the “12 Principles” defined from the Agile Manifesto. Here is how they played out in my first project.

1) Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
As this is the highest priority I can see now where we missed the mark early on. A lot of the early work was necessary to get to a place where we could deliver valuable software. But to get there we spent a few sprints not delivering ANYTHING. Eventually, of course, we started to produce deliverables, but there were opportunities to give them something. Alternatively, we focused on getting EVERYTHING prepared.

2) Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
My second exposure to Agile development was with a client as a BI Consultant. On day one with the client, I attended the retro from the sprint they completed before we joined them.
In this meeting, there was a lot of turbulence and it was because they struggled to properly plan the sprint, and didn’t complete the tasks, and also, they had to do a lot of rework/direction change work, etc. It was very stressful to sit in, and it seemed no one at the time was “ok” with changing direction. It was funny to experience because I had only read (recently) that Agile was flexible to allow direction changes!

3) Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Our sprints were set for 2 weeks, 10 working days, 80 hours. During these sprints, we rarely had anything to “show” at our large show and tell meeting. We missed some deadlines because I would get 80 hours of work assigned, but I had 20 hours of meetings (slight exaggeration), but we weren’t accounting for meetings. So, we shifted down to 60 hours of development and started to complete all of the tasks assigned.

4) Business people and developers must work together daily throughout the project.
In the first few sprints there were a lot of changes and missed marks, but the client slowly began to understand that because they were struggling to define the features properly, they were unable to define the stories, and therefore the tasks were also not defined to accomplish the goal of meeting the story and feature requirements.
It got better after a few sprints; it really did. We started to flush out the features, “What does the client or stakeholder want?” We ended up having better stories and saw an increase of clearly-defined tasks. Things were improving slowly.

5) Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.
Largely this seemed to be supported at the client. There were environmental issues that we had to deal with, but for the most part, I felt like this principle was pretty well supported.
One thing that I noticed was in the group makeup of the team. Different people have different skills and levels of those skills. When story planning, don’t let the loudest voice in the room dictate points or hours. Maybe give the estimate of the person assigned the task a little more weight.

6) The most efficient and effective method of conveying information to and within a development team is a face-to-face conversation.
This one was interesting, we met daily and the meetings consisted of people on site and remote team members who called in. Technology is allowing us to share screens and have productive remote employees. But what I did find, is that if you were on site, you were able to attend the “meeting after the meeting” where it turned out more was discussed after the call was ended. The people on site often had more information gained from these after-meetings, than those who were remote.

7) Working software is the primary measure of progress.
At one point we completed the sprint tasks, stories, and features. It was great, and then the product owner was upset that something he wanted completed, wasn’t. We had met all of the acceptance requirements, the feature he wanted wasn’t listed in any of the criteria. We were blindsided, and a bit demoralized. I remember after the meeting we all just kind of looked around and wondered what just happened. Turns out that the product owner wanted to see this feature live in the application. The acceptance criteria just wanted completed queries. If we had adhered to principle #4, we might have caught it, but if we adhered to principle #7 no one would have even known this was wanted, it just would have been delivered.

8) Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Story Points. I felt these weren’t exactly well defined, or maybe the definition changed. With this client, 13 story points represented a full sprint. To me, that meant 60 hours for 1 developer. With 3 developers on the team, we should have been close to 39 story points for each sprint. When I joined the team, they were completing 28, and when I left the team they were scheduling 48 and missing the deadline. Another issue was that a story with 5 story points to me should have taken around a half a sprint, 1 week, or 30 hours. And it was often tasked with 4-5 tasks totaling 10 hours. There seemed to be a disconnect in what story points meant, or how tasked hours correlated to story points. Maybe this isn’t the case, but in hindsight, it probably should have been discussed further.

9) Continuous attention to technical excellence and good design enhances agility.
To me, this means writing code that is proficient and maximizes the available technology. This may be the weakest principle at this client. The quantity was valued higher than quality and we wrote some inefficient code just to get it done. On occasion this meant we had to go back and rewrite the code which meant double the work, just to say we completed it during the sprint.

10) Simplicity–the art of maximizing the amount of work not done–is essential.
There was a story that was on hold for 2 sprints due to an architecture change that was needed. There was a lot of debate in each planning session about this specific story, and finally, in the third sprint, it was determined that it wasn’t needed. Even though it was groomed out and ready to be added, the delay and continued discussion was necessary to actually remove the story and not waste the time and resources to complete a story that was unnecessary. In this case, we kept it simple, and it allowed us to add work that was needed.

11) The best architectures, requirements, and designs emerge from self-organizing teams.
On this project I found we struggled a lot in having recommended changes to architectures in both the client systems as well as the third-party software systems. This can, of course, be necessary as making changes to an architecture can lead to issues with other products. The only real issue is how hard it was to get anyone to buy into the changes recommended, it just made for a difficult “agile” environment.

12) At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
I think this was one of the most adhered to principles overall. The client and teams were very receptive to continually evaluating ourselves and making adjustments to hone our workload and deliverables.

I would hate to think that this evaluation of my first ever experience on an agile data project would reflect poorly on me, Talavant, or the client. I enjoyed the experience and it was fun to see the teams learn as we went along. It wouldn’t be very Agile of me to say we have to follow all these principles to the letter! Instead, it was great to learn with the client. This reflection is meant to adapt and change my expectations just as the teams had to adapt and change with reflection.
My final thought on the 12 Principles of Agile Development Methodology is that it takes time to hone estimates and build effective teamwork. While it is important to identify shortcomings, remember to celebrate the successes and obstacles that are overcome.

Ryan Desmond

Senior Consultant

Share This Article

At Talavant, our goal is to help you discover, ask and answer the right questions – so you get more value from your data to drive your business forward. You’ll gain a competitive edge with faster, more relevant analytics. It’s all part of our focused, holistic approach that goes beyond tools.

Ready to get started?

Contact Us