2022: the year of efficient data management and decentralized organizations

Data management should be high on organizations’ agenda for the coming year.

According to data published by IDC, 64.2 Zetabytes of data were created in 2020 alone, which is very close to the total installed storage capacity of 6.8 Zetabytes. However, of all this constantly growing volume of data, only 2% will have been saved and stored by 2021, the vast majority of which will either be lost or form part of what is known as “ephemeral data”, data that is only associated with the moment of consumption, requiring only temporary storage, or that is updated or overwritten with new data. 

In this context, what is clear is that companies should prepare their systems to capture more information. Data is and will be a key business value for organizations, allowing them to enter what is known as the virtuous circle of data: data capture generates valuable information, which, when analyzed and exploited, generates business insights to improve decision making and generate more sales, which will return to deliver more data that will generate new insights. 

Five will be the key strategies in data management in the coming year.

1. Privacy and security: In 2022, many will continue to analyze the best way to use their information, but they must first ensure the complete security of their data. A trend in this regard will be to implement global data management, which controls both the visibility of the different data repositories and authorized access to each piece of data, in a framework that applies clear criteria on a day-to-day basis.  This is possible with the creation of a data governance layer that offers control over access, data security and privacy, as well as a detailed description of its lineage and ownership, among others. A strategy that also provides a better user experience when creating, sharing and consuming data. 
2. Democratization of data: The main objective of organizations is to facilitate (controlled) access to data for all business units, so that everyone can access them and can take advantage of the work done by other departments. The work already done on the basis of data should be available and accessible so that it can be used by other areas, thus promoting a data-driven approach in the organization.
3. Cost control: When a data platform is available, potentially all the teams in the organization can consume it, and this consumption entails a cost. The next step is to solve the challenge of distributing the cost of the data and identify formulas to monetize it. The ultimate goal is to distribute the cost based on the consumption of the data (pay-per-use)
4. Efficiency and automation: One of the objectives recently pursued with special interest by organizations is to achieve a high degree of efficiency in their operational processes related to data. Automation is becoming the trend to pursue in processes such as cataloging, publication, privacy or access to data, which provide great value, contribute to increased profits and achieve results in a less complex and faster way.
5. Artificial Intelligence: Until now, the most common approach to the development of projects that use artificial intelligence to process data has been through Proofs of Concept. However, the confidence that this technology already generates makes that, more and more, use cases are proposed to achieve real results and value for business and for the customers of the organization. Artificial intelligence is being applied mainly in its machine learning variant, in the field of automation of administrative and operational processes, detection of anomalies and prediction of demands or in maintenance environments; all this, very oriented to the search for efficiency and time and cost savings.

2022 gives a promising outlook for data architectures on public clouds.

Data governance and privacy will become more and more relevant and build the basis for big data projects. Low code will abstract the development of PoCs, increase value delivery and finally enable new projects with traditional development to emerge. However, it is safe to say that out of all these forecasts data democratization will be the final goal for every data oriented company in 2022. This is a crucial requirement to compete in the market and will directly determine the business value creation for enterprises.

Data Warehouse vs Delta Lake vs Lakehouse approach

The amount of data generated every day makes it necessary to look for secure long-term storage solutions and tools that allow the data to be processed and analyzed with the lowest possible latency from the time the data is stored until it is used.

Concepts such as LakeHouse or DataLake seek to offer a cross-company solution that allows the establishment of federated and reusable storage, security and data exploitation policies. Similarly, technologies such as Delta Lake seek to fill the gap left by public cloud providers in terms of data lifecycle and evolution control over time.

During the last four months of 2021, we have seen exponential growth in the use of the term DataMesh. This concept does not incorporate any new technological component that we did not know or use until now, but aims to evolve the way in which we work with data and that allows scaling and specializing both the tools and the human team to be able to cover the large volume of data that is expected to be generated throughout 2022.

Automation and the effects of LowCode platforms

Intelligent and automated self-services such as LowCode allow enterprises to generate business value more quickly and with less effort. This enables projects to evolve to greater and more complex business scenarios which again introduces the need for traditional development of digital products. LowCode allows us to focus on building secure, resilient, complete and efficient data platforms and to establish with the client the links so that they can go deeper into the consumption of their information and generate better PoCs in less time.

Data Governance and Privacy

We see here a huge gap between customers’ expectations for cloud native data catalog and lineage services and what services like Google Data Catalog or AWS Glue can currently provide. In terms of data governance and privacy we expect hyperscalers to release new services and features in order to compete with popular third party tools like Collibra, similar to how Azure has recently released its overhault data governance service Azure Purview. 

We assume more services to include data de-identification features to encode PII (Personally Identifiable Information) and other sensitive information included in data assets. De-identified data can be securely shared and analysed. However, de-identification introduces additional overhead, where data needs to be de-identified in pipelines. We assume more cloud native features to integrate de-identification features natively without the need of moving the data first. Finally, the goal of data de-identification is data democratization with its benefit of accelerated business value creation.

Data Democratization

Most big organizations have uploaded terabytes of data to their distributed data lakes. The effort to democratize this data to different types of users with the proper security, scaling operations and without exposing data is going to be huge, so we expect new services, solutions and tools for this issue. Data democratization will be crucial for future business growth, since data is the most valuable resource for modern companies. Enabling the use of this data for the whole organisation will increase business value generation significantly.

What’s to come in AI is even more exciting and companies will need increasingly sophisticated products.

Data continues to be a trending topic in all economic sectors and extracting value from it is becoming an increasing priority. This has led to an exponential demand for training in soft and hard skills (functional and technical knowledge) in relation to concepts such as Big Data, Cloud platforms, architectures in Cloud projects, process automation, data science or BI, among others. 

Over the last few years, many companies have found it necessary to migrate their on-premise services to any of the current public cloud platforms. This has allowed them to accelerate the development of use cases based on insights extracted from their data.

The process of digitization of the industry is a reality, and with the implementation of multiple IoT devices that generate huge amounts of information, the real challenge is the transformation into actions that help prevent, alert or advise with a high degree of success. Some examples are seen in the early diagnosis of diseases or in the prototypes of self-piloted vehicles. 

The growing trend in cyber-attacks can affect the prestige of many companies and cause critical economic damage. In this scenario, investment in security is enabling the development of zero trust models, remote work protection or information leakage prevention.

Human-machine interaction has always been one of the biggest challenges in AI, and one of the most popular use cases continues to be conversational assistants. The clear evolution of these assistants is their collaboration, allowing different bots with expertise in certain areas of knowledge to cooperate with each other and obtain more versatile too

What about tomorrow? 

What lies ahead is even more exciting and is of great interest to the AI community and to companies that need increasingly sophisticated products.

An increasing interaction between robots and humans will require better NLP tools allowing more personalization. As a consequence, solutions will have to be increasingly transparent and based on fairness criteria that are aligned with ethical awareness in AI projects.

There will be a demand for greater autonomy of IoT devices which will come hand in hand with the development of TinyML technology.

Finally, the democratization of AI will be a reality with the standardization of No-coding Machine Learning allowing different technical and business profiles to develop complex applications.

Maintaining team spirit, employee engagement, organizational culture and the capacity for innovation has been and remains the challenge of teleworking.

We look at agile trends on the 20th anniversary of the Agile Manifesto. While in recent years the agile world has focused on promoting and raising awareness of its mindset, the big news of these two years of pandemic in the IT sector comes with the new challenges posed by the sudden increase in teleworking and the impact it has had on employees, organisations and their capacity for innovation. Additionally, there is a growing interest in renewing and strengthening the culture of organisations as agile development, value delivery and having a good business strategy are competitive factors.

As a result of the pandemic, many companies were forced to have their employees working from home from one day to the next, having to accelerate the adoption of new technologies and processes. Fortunately, most employees in the IT sector were already familiar with the use of remote communication technologies, but the impact of having all workers offshore remained to be seen. In general, there has been a good adaptation in this sector to remote working and to try to reduce the impact of physical distance without reducing productivity or the ability to innovate. Existing team techniques have been adapted and new ones have been learned to facilitate sessions and reduce the impact that physical distance can wreak on communication and the generation of collective intelligence. Tools such as Miro, which offers the possibility of making collaborative online boards, have been fundamental to this.

The main challenges have been and remain the preservation of the team feeling, employee engagement, organisational culture and capacity for innovation. In particular, relocation has affected new recruits, who have not been able to have that first physical contact with the company or meet their colleagues face to face, which makes it difficult to generate bonds and a sense of belonging. Health concerns and isolation have added difficulties that companies have tried to combat with specific actions, creating online meeting forums for employees to interact and get to know each other.

According to the annual report, 5th State of agile (2021), 22% of companies have started to practice agile in the last two years. The use of agile in software teams has increased from 37% in 2020 to 86% in 2021 and in non-IT areas it has doubled. More and more Finance, HR and Marketing teams are using agile practices and processes. In terms of methodologies and frameworks, Scrum remains the preferred methodology with 66% usage. ScrumBan and the Kanban method remain with a discreet 9% and 6%, respectively. 

New organisational models that already existed are becoming more widely known and there is increasing talk of Teal, Holacracy and Sociocracy 3.0. The application of these new paradigms in organisations that were not set up this way from the start is still a long way off. However, more and more companies are introducing changes in their culture, giving a voice to their employees and caring for their well-being as well as their professional development. The trend is towards purpose-driven, decentralised and delegated power structures, with a strong foundation in communication, collaboration and collective intelligence. 

The big question is: is telework here to stay after the pandemic? It is very likely to continue, altought with less intensity. At the very least, a hybrid model will be maintained, with face-to-face and remote days. According to McKinsey (2021), this puts the number at four to five times higher than before the pandemic and office space will be reduced by 30%.

All in all, it is easy to conclude that both the agile paradigm and employee- and customer-centric organisational cultures that leverage collective intelligence will continue to trend in the coming years.

Image: Rawpixel