How to deal with loudest guy-driven decisions in tech: Part I
Technology and the tech era have brought many advantages, both to individuals and companies. But there is no one influenced more by technology and its trends than developers and engineers. Overall trends and hypes influence them and cause constant switchovers between technologies. Technology, especially new ones, can significantly improve an engineer’s life, but in some cases, it can also be his nemesis. The Loan For us, the story began while developing a data-driven application for one of our clients. The main aspects of the application are data visualizations, where the underlying data comes from a database designed specifically to serve that purpose. The existing database system was fairly expensive and did not cover all functionalities that were much needed. As the license was about to expire, it was decided to replace it with alternative technology. But, which one to choose? License expiration time defined a short deadline, and lack of experience in the domain did not help either. Some of the technical staff had limited experience in one distributed in-memory database called Apache Ignite. To prove suitability, the tech team conducted testing to prove that Ignite is the right choice for the current needs. At the time, the data-driven application was in the initial phases of development. As the requirements and scope of the application were not set yet, the tech team had a thankless job to test future-proof capabilities. So, in the end, after basic testing provided positive results, the decision was made and the existing database system was replaced with Apache Ignite. Looking back on the decision, one could ask a few questions. The first one could be about complexity. From an infrastructure and configuration standpoint, the distributed nature of Apache Ignite brings extra complexity. So it’s questionable if this complexity was worth it when thinking of our use case. Maybe some simpler alternative would have worked just fine. That brings us to the next question. Were any other alternatives tested and compared to Apache Ignite? It’s always beneficial to compare multiple options and decide which one suits you best. In the end, there is a question if the testing phase covered all important aspects. Knowing all the limitations the team had, it’s hard to judge them for lack of testing. These issues can be linked to two terms, technical debt and hype of new technologies. The technical debt occurred when a fast decision had to be made in order to prioritize client value and time constraints over technical aspects of the solution. It might not be an ideal choice, but the transition to Ignite was done fast and solved the issue the client had. In other words, the team consciously decided to take a risk, solve current needs, and take care of issues that might occur along the way later on. The hype is an issue when a certain technology or group of them becomes trendy. Advertising and media influence in combination with a lack of experience and maturity can lead to illusions that they will act as a silver bullet for all of our issues. This phenomenon is frequently compared to the Dunning-Kruger effect, which has its origins in psychology. One well-known example of its application in technology is Gartner Hype Cycle Research, which visually represents the maturity of technology and its suitability for solving business problems. This happened to Apache Ignite as well. At the time, it was at the peak of popularity, which influenced the technical staff to consider it. While preparing for this blog post, I found an interesting article that describes hype in the tech industry. It defines a term called “Loudest guy-driven decision”, meaning that the decision-making is left to the team member(s) that talk the most, frequently based on a limited experience gathered from talks or conferences. To mitigate the hype, bring maturity to the focus, and reduce bias it would be beneficial to involve more people in the decision-making process. Debt Collection Each debt needs to be collected and there is no difference with the technical one. Let’s see how ours accumulated over time, and what caused it to explode in the end. The starting usage period wasn’t bad per se. Configuration and infrastructure were done in the simplest way possible – we had two single-node clusters for each environment, where one of them acted as a backup replica, while the other one served the application. Hosting it on AWS cloud enabled us flexibility when talking about resource allocation. Resources like RAM, CPU, and disk were set based on estimations and adjusted based on monitoring results. As the application was in the early stage of development, each sprint brought many new features. One of these included advanced data filtering and aggregations. Without going into too many details, the core functionality of the feature was to let users choose their own subsets of data on which they could perform filtering and aggregations. To achieve efficiency and application performance, we had to introduce some kind of data redundancy. All at once, our tables stored many more records than before. As we said, the application is based on visualizations. All of them pull the data from Apache Ignite using SELECT statements. Increased number of records in combination with frequent querying led to increased load on our database system. Unfortunately, this was not taken fully into account, as the testing phase was pretty short, and was done in an environment where we did not have a representative subset of production data. Why We Don’t Deploy on Fridays We deployed the new feature on Friday morning. The monitoring did not show any issues during the initial data synchronizations and throughout the rest of the day. It seemed like everything went off without a hitch. Or so we thought. The first issue occurred in the late afternoon. Which wasn’t a surprise since most users are based in the US. Knowing the time difference, our afternoon was the start of the day for them. It all started with application slowdown, which was a reflection of