Algorithmic contingency and the self-perpetuation of datafication
Organisations couple their digital transformation with the adoption of agile management concepts or methods (and vice versa) almost as a matter of course (Muster & Büchner, 2018). These two phenomena – according to organisational narratives – seem to be mutually dependent: agility only works in a digitalised environment, and the benefits of digitalisation are only accessible within agile organisational structures (Baecker & Elsholz, 2021). Both types of transformations – organisational as well as technological – are associated with aspirations to achieve rationalisation, less bureaucracy and increased flexibility (Büchner, 2018:333). Nevertheless, de-bureaucratisation is often more of a shift in the locus of formalisation than its abolition (Sua-Ngam-Iam & Kühl, 2021; Bull & Muster, 2021; Eckstein & Muster, 2021). In the case we examine, the central human resources department of an international corporation adopted what is known as the ‘Spotify model’ (Kniberg & Ivarsson, 2012), a project-based matrix that aims to break down silos and banish hierarchies from operational work. To support the management of personnel, they also introduced a digital platform. We argue that these transformations increase the complexity of personnel decisions, while the platform produces algorithmic contingency (Esposito, 2017) as a subsequent problem. The dynamics of this interplay of structural changes and subsequent problems is the subject of this article. We will show how the reorganisation increases internal complexity, handling it via a machine that produces algorithmic contingency, which is then addressed through interaction systems, where the uncertain but unique informational output of the platform is negotiated.
After a brief introduction to the systems theory perspective on digitalisation and the concepts of datafication and algorithmic contingency, we will reconstruct our empirical case, firstly by describing the corporation’s structure and its reorganisation, and secondly, by outlining the digital platform and its functions. Afterwards, we will show how the reorganisation and the introduction of the platform increase the complexity of personnel decisions and, subsequently, produces an algorithmic contingency. The systems theory perspective (Luhmann, 2000) highlights changes in the mode of operation as boundaries of social and technical systems (communication in formal and informal interactions and, by focusing on boundaries, this approach allows us to disentangle the complex constellations often described as socio-technical systems.
Building on interview material, we will show how formal and informal interaction systems emerge, and how they hinder the self-perpetuation of datafication concerning personnel decisions as they thwart the platform’s output. Finally, we summarise our argument and propose further research on these social barriers to the self-perpetuation of datafication.
The interplay of organisation and datafication: a social systems theory approach
Restructuring in formal organisations is a modification of their decision premises. In the case we examine, two of the already determined decision premises are of interest: decisions on personnel and those on the official channels of communication (Luhmann, 2000). The targeted Spotify model (Kniberg & Ivarsson, 2012) combines developmental and operations logics. 1 This reorganisation is staged as ‘agile’ and aims at flat hierarchies, project work, network-like structures and increased interactions; therefore, it can be called ‘post-bureaucratic’ (Heckscher, 1994, 2015; McSweeney, 2017; Annosi & Brunetta, 2017). From a systems theory perspective, these interactions are systems based on attendance and mutual attention and, in the case of organisations, constrained by membership (Kieserling, 1994).
Digital technologies are essentially a long chain of strictly linked binary signals, that have little tolerance for contingency and equivocality (Esposito, 2014:235; Husted & Plesner, 2020:5). 2 Therefore, we conceptualise digital technologies, such as the management platform under discussion, as technical systems in which operations are strictly (causally) coupled (Luhmann, 2000:370), whereas social systems such as organisations and interactions, operate via communication (Luhmann, 1984:191). Since double contingency – the uncertainty about whether the relevant counterpart will accept or reject communication – is the problem addressed by system formation (Luhmann, 1984:156) 3 , it is debated whether human-machine interactions qualify as social systems. Esposito (2017:258) argues that machines produce virtual contingency, as users project their uncertainty onto the machine whilst the machine can also – as in the case of smart algorithms – produce a contingency based on its consideration of different observers. As a machine learning algorithm, such as the one under discussion, draws information from different sources – combining multiple observer perspectives – it becomes a source of uncertainty within the organisation. Nevertheless, this algorithmic contingency is still ‘a reflected perspective, because the algorithm inevitably does not know contingency’ (Esposito, 2017:258). 4 Büchner and Dosdall (2021) note that algorithms, understood as observation schemes, only become ‘actionable’ if they are embedded in organisational decision architectures. Using an input-output model akin to the one we apply in this article, they distinguish between the loose and strict coupling of informational output and subsequent communication. Whilst strict coupling predetermines what follows an algorithmic output, a loose embedding aims to explore data to arrive at better decisions. In the case under discussion, we differentiate between formal and informal interactions that follow an algorithmic output.
Following this train of thought, we draw on the concept of datafication (Muster & Büchner, 2018) to conceptualise self-perpetuation. Datafication refers to organisational initiatives to produce and use data. This links technical and organisational logic as it points to the formal implications of these data – its index of formality (Muster & Büchner, 2018:271). Organisational and informational formalisation are interdependent (Mormann, 2013). As algorithms presort data for subsequent decisions (Kitchin, 2017), they are subject to departmental logic and micropolitical strategies (Ortmann et al., 1990; Constantiou & Kallinikos, 2015; Alaimo & Kallinikos, 2020), as well as informal expectations (Büchner, 2018; Funken & Schulz-Schaeffer, 2008; Faraj et al., 2018). Organisational datafication creates new transparency regimes (Hempel, Krasmann & Bröckling, 2011), and offers opportunities for control, but also leads to increased efforts to conceal informalities.
The concept of datafication shifts attention to the dynamics of the relationship between technical and social systems without blurring their distinction. Whilst digital technologies do not know contingency (Esposito, 2017:258), they produce contingency for social systems, hence algorithmic contingency, which raises questions of how organisations deal with this contingency and how this affects the self-perpetuation of datafication.
The case: the tale of a reorganisation with a twist
In this section, we will introduce our methodological approach and explain our method for data collection and analysis. Then we will reconstruct the empirical case by describing the agile reorganisation and the digital management platform. These introductions provide the basis for the subsequent analysis of the increased complexity of personnel decisions, and the formal and informal interaction systems that deal with algorithmic contingency.
Methodological approach
This case is part of an in-depth case study on the effects of digitalisation on organisational structures. The research project is methodologically and theoretically located within the theory of social systems and functional analysis as presented by Luhmann (1964). When a problem that serves as a reference is localised, the functional equivalency of different structural changes can be observed and subsequent problems assessed.
In this case study, a structural analysis of social systems was used to map out these structural changes (Besio & Pronzini, 2011:22 ff.). As we reconstructed a chain of operations (decisions), information on the formal and informal structures of the organisation had to be gathered, which required the use of qualitative interviews, and a triangulation with data deriving from the analysis of documents (Flick, 2011; Kuckartz & Rädiker, 2022:41).
Data collection and analysis
For our in-depth study of post-bureaucratic structures and their interplay with digital technology, we selected an organisation that had a long (bureaucratic) history and operates in a highly digitalised field. We used an exploratory qualitative research design. Ten semi-guided expert interviews were conducted. Employees were interviewed as experts on the organisation’s structure (Bogner & Menz, 2009). 5 The interviews were protocolled simultaneously, recorded and fully transcribed. 6 Additionally, we triangulated the results of the interviews with findings from document analysis regarding the formal structure of the organisation. Interviewees were selected according to their function, to ensure that they held different organisational roles. Therefore, members at the border points of the organisation, as well as those accountable for the rollout of the platform and those who deal with its outputs in their everyday work life were of interest to us. 7
The data gathered were analysed using content-structuring qualitative content analysis (Kuckartz & Rädiker 2022:129−156) with the aid of QDA software, and categories were assigned: we focused on thematical, theoretical and – when statements were particularly concise – in vivo categories. Whilst thematical and in vivo categories were coded inductively based on the data, the theoretical categories were derived deductively from the theory of social systems. The categories were then arranged within a hierarchical system and main-and sub-categories were identified. 8 Since the following case reconstruction is the combined result of our document analysis, the interview analysis and our observations, we will refrain from directly quoting statements in the following section and, instead, use them in the subsequent section to support our analysis.
Reconstruction of the reorganisation
An international corporation reorganised its central HR department of about 600 employees. This was deemed to be an ‘agile transformation’ with the organisational structure of the streaming service Spotify used as a model. At roughly the same time, a digital platform was introduced to support personnel management within these new structures. We will now look at the details of this reorganisation and the platform in order to then analyse the functions and follow up problems of both.
Agile reorganisation
Looking at the formalised net of contacts, the new structure is a complex matrix organisation focused on project-based work in ‘squads’. There are five ‘tribes’ (former divisions) that report to the board. Each tribe has a chairman (tribe lead), who combines functional and disciplinary authority. Within tribes, there are chapters, clusters and squads. Employees are assigned to chapters for disciplinary purposes but work operationally in project-based squads. Several squads whose topics are similar form a cluster. In other words, the chapter leads have disciplinary authority while cluster leads have functional responsibility. Therefore, chapter leads decide on personnel issues concerning their subordinates while squad leads have to voice personnel demands for projects.
The new structure was intended to enable squads to be staffed across tribe boundaries, break down silos and banish hierarchies from operational work. Chapters serve as a formal organisational home described as a ‘pool organisation’ (P2). One objective is to accelerate staffing decisions. Some chapter leads have a broad span of control and stick to personnel issues, while others still have functional responsibilities which may be their main task.
Agile reorganisation increases the complexity of communication channels and, therefore, the complexity of personnel decisions. This complexity is also increased by the (self-)attributed skills of employees’ profiles as we will show below.
Digital personnel management
The technology at hand is a SaaS (Software as a Service) solution 9 and described by the provider as a management platform utilising machine learning. The two functions used in the present case are: firstly, collecting and producing data on the skills of the personnel; and secondly, collecting data on employees’ workloads and open project positions and mapping this information. The results are automated proposals as to whose skills and spare capacities best fit the requirements.
Although it is supposed to be on a voluntary basis, each member of the department is formally required to create a profile in which they enter their skills and individual workloads. The initial profile is based on their job description, to which certain skills are already assigned. Additional skills are selected from a wide-ranging list of approximately 1,400 skills, or manually entered if a skill is not yet listed. Employees rate their competencies in a particular skill on a scale from one to four. They also enter the projects they are dispatched to and their workload in percentages into a calendar. The platform then calculates free capacities automatically.
In addition to the profiles, squad leads can feed personnel requirements for their projects into the platform. To do this, they specify which skills and capacities are in demand. The platform then checks whether there are profiles that match these requirements. If so, the squad lead and the employee receive a message so that they can exchange ideas. If the platform identifies multiple matching profiles, the squad lead receives a list sorted by fit and can view the profiles anonymously to decide who to contact. In addition, employees can proactively search for projects with open positions. If there is a match between the profile and the project, the employee’s chapter lead is engaged and must formally confirm the assignment. Personnel decisions are expected to be informed by the platform’s data and are negotiated between the chapter leads.
As mentioned above, we view datafication (Muster & Büchner, 2018) as two-step processes of producing and using data. Building on Luhmann (2000) and Esposito (2011; 2017), we treat the personnel management platform as an invisible machine that deals with the increased complexity of personnel decisions as a subsequent problem of the reorganisation and produces algorithmic contingency as it draws information from different organisational subsystems, as well as from individual members profiles. 10 The platform’s machine-learning algorithm considers the multiplicity of projects and their demand for specialised personnel as well as the individual skill sets and interests of employees. This in turn produces an algorithmic contingency at the output boundary, as the platform is a black box to the organisation.
The self-perpetuation of datafication – how organisations deal with increased complexity and algorithmic contingency
Using the ability of functional comparison methodology to enable us to attribute and compare problems and solutions in the context of social systems, we will now show how the organisation deals with the self-perpetuation of datafication through formalised and informal interactions. In this perspective, self-perpetuation is conceptualised as a form of organisational datafication to highlight two aspects: the increased complexity of personnel decisions at the input boundary and the algorithmic contingency at the output boundary of the management platform. Below we will focus on the input and output boundaries of the platform and describe how organisations increase complexity and deal with uncertainty.
Reactions to increased complexity at the input boundary
By staffing personnel project-wise across tribe boundaries while simultaneously implementing a digital platform that also comes up with its own requirements, new demands emerge, not only for executives but for all employees. 11 To deal with this increased complexity, the organisation establishes formal and informal ways of generating data.
After filling in the platform’s profiles, there is a formal and obligatory discussion between the chapter leads and the employees about their self-assessment. The influence of the chapter leads is limited. They cannot reject a profile they disagree with because of the urgency of due process. They can only set up a meeting to discuss the aspects they do not approve of:
[…] there is no option with the tool to reject this skill by saying, ‘not relevant or not right.’ […] I have to release it because if I don’t release it now, as a chapter lead, I’ll be put under pressure next week because it’s [my job] to make sure by tomorrow that all these conversations have taken place. (P10) 12
Still, it is the employee’s decision whether to take their chapter lead’s opinion into account. The chapter leads are controlled by the People & Planning Squad (PP squad) and the business partner and pushed by them to update their employees’ profiles (P8). 13 Only the skill management squad can reject a profile and send an email on this matter to employees (P9). 14
It is not only employees and chapter leads who are monitored when entering their data. Squad leads cannot enter staffing requirements without the tribe leads’ permission: ‘the need for a squad lead was approved, so it’s not like every squad lead can decide, I need 50 people now, but that has to be well-founded, that is, on the goals’ (P8).
The third new formal structure addresses the problem of low data quality. The PP squad checks for overbooking, that is, whether employees are booked at more than 100% capacity. 15 If this happens, the PP squad speaks to the employee’s chapter lead about it. At the same time, there is no institutionalised control system for correctly entered workloads. If a department member – intentionally or unintentionally – enters a higher workload, no one would take note.
There are informal workarounds at the input boundaries of the platform. When filling in their profiles, members are completely free to select and rank their skills regardless of whether they do so truthfully:
I evaluate, for example, my level of language. I evaluate my level of working with project management tools, which of course has its limitations. Especially for people just starting, it’s not easy to evaluate adequately the level of competencies. And then it’s difficult to differentiate between top experts and just experienced people. It’s not yet, I would say, working perfectly. (P1)
One member explained that he and some colleagues intentionally chose skills they don’t have, or invented some new skills, for example, ‘drinking coffee’ or ‘micropolitics’ (P9). Although the chapter leads check their employee’s profiles, they can only approve them. Consequently, employees can describe themselves however they want and choose as many skills as they like. Despite the request to select only ten skills, members could choose 100 or even 1,000 skills which all had to be ranked. Thus, they overwhelm the chapter leads who are not in a position to review the accuracy of even 50 skills per person in a unit with 30 employees:
I2: Did your boss then also release coffee drinking? B: Yes. Sure, because in the 800 things I gave him, he didn’t see all the things I hacked into it. (P9)
There are several more informal strategies, such as choosing skills that are currently in demand 16 or entering a higher workload to avoid assignments: for example, employees can enter 100% capacity utilisation instead of 65% and, consequently, will not be listed as available (P3; P9). Some chapter leads have little interest in handing over their employees to other projects, since they are dependent on their performance (P9): ‘Do I want to give my best buddies to another tribe or does it make more sense to keep them closer to me?’ (P2).
Regardless of whether the data are accurate or flawed, the complexity of aligning a job and an employee increases. This complexity at the input boundary is transformed into algorithmic contingency at the output boundary since it is processed by a machine. The platform is a black box for the organisation whose operations cannot be observed and whose specific outputs are surprising and create opportunities for further decisions.
Processing contingency at the output boundary
In the case study, we were able to observe formally determined and informally emerging interaction systems that complement the platform and process the algorithmic contingency at the output boundary. These fill in the gaps left open by the platform as they allow us to create a new situation from which subsequent decisions must proceed. We will show how members add new information to personnel decisions as they attend different meetings to complement functions of the platform.
The first formal interaction adds new information not captured by the platform. It is a meeting of all chapter leads and the PP squad to first staff the prioritised squads called Big X squads:
[…] we have a commitment across all tribes that these Big X squads, these priorities, will be filled first. Because experience shows we have more work than people, we have to focus and prioritise. That means we have a list, let’s say ten new requirements, then we have to see which ones are prioritised and those will be filled first. (P8)
The second formal complementary process is the weekly staffing call. At this event, other ‘open project needs are discussed’ (P8), the necessity of which is explained by one chapter lead referring to the low quality of data (P5). They explain that they are notified of demands concerning their subordinates either by the platform, or during the staffing call, and then prioritise these against the backdrop of strategic considerations that are not mapped in the data. 17
If, besides these complementary interactions, there is still an unfulfilled need, there is a third formal interaction; ‘[…] a monthly meeting, called Prior Round, with representatives from each tribe who also have the mandate to prioritise to be able to decide’ (P8). In this third formal meeting, the decision is moved to the top to provide a final decision, so it can be escalated formally.
The fourth process is a discussion between the chapter lead and their employee before the employee is assigned to a new squad. A member of the PP squad explained that this is the desired process because of the company’s values. They said that it would be strange if a button reminded a chapter lead of an assignment given to their employee and they accepted it without speaking to the employee first. It would also be strange if a chapter lead proposed an employee for another project and the employee accepted it without speaking to the chapter lead first (P8).
In addition to these formally decided complementary structures, an informal structure emerged: every time a project and a profile are matched, the squad lead and the matched person meet in person (or via a web conferencing tool), drink coffee together and see whether they are compatible (P8). Other interviewees also told us that instead of using the platform, they would just talk to each other: ‘from my point of view, […] talking to each other dominates in practice’ (P10; The same argument was made in the interviews P1, P3, P5, P8 and P9).
There is one more formally-decided complementary structure which concerns the supply function of the platform. Once a month, the department organises a ‘pitch call’ where vacancies in squads are presented to employees interested in a new assignment: ‘we can take part in a meeting and tell what we have to do, what people we need. And then somebody can just volunteer to be in our squad’(P1). This interaction system mirrors the platform’s marketplace where squad leads can enter vacancies and receive suggestions for candidates who meet their requirements, whilst candidates get an impression of the people who work on the project (P8). However, one chapter lead only recalled two instances where his subordinates proactively used that event to communicate their desire for a new assignment. According to his assessment, about 95% of job changes stay within one chapter and the same tribe: ‘we have this cocoon formation, […] which we actually wanted to dissolve with the agile organisation’ (P10).
Hence, the organisation deals with the algorithmic contingency produced by the platform by introducing formally decided and informally emerging interaction systems that bridge the gaps left open by the platform and complement datafication.
Discussion and further research
To begin with, we raised the question about the relationship and interplay of social and technical systems, with regard to an agile reorganisation and a digital management platform. We outlined the dynamics that come into play when organisations introduce technologies to base their decision-making on data.
We described how personnel decisions are informed by the platform’s algorithm, and subsequently negotiated in complementary interaction systems. Building on the concepts of organisational datafication and algorithmic contingency, we did not argue that technology perpetuates itself, but rather that the interplay of agile reorganisation and technology produces new complexities and contingencies at the interfaces of the technical and social systems. At the input boundary of the platform, the equivocality of communication is transformed into computable uniqueness. The increased complexity of personnel decisions (project-based work and more detailed information on employee skills) is processed by the algorithm. The uniqueness of the platform’s output is then transformed back into equivocal communication as system boundaries are crossed. The output is negotiated in interaction systems. Therefore, formal and informal interactions constitute a barrier to the self-perpetuation of datafication as they make it possible to complement platform data. We thus identified an interplay of loosely and strictly coupled operations at the input and output boundaries of the technical system, reflecting its organisational embeddedness.
The systems theory perspective enables a fine differentiation of the interplay of digital technology and organisational reorganisations. It differentiates between social and technical systems and sheds light on the change in the mode of operation, as processes cross system boundaries by using an input-output model. It also considers the index of formality of organisationally produced data on the platform, as well as the organisational embeddedness of the formal and informal interactions that emerge at the boundaries of the platform.
This perspective also makes it possible to examine the relevance of the technological potential of the platform. Irrespective of how intelligent a technical system might be, drawing from this analysis it can be assumed that technology in organisations encounters both formal and informal reactions. We have indicated how those barriers to the self-perpetuation of datafication are in themselves characterised by contingency because they are social systems.
This train of thought illustrates the limitations of this case study, as well as the scope for further research. Since it is a qualitative case study, the findings of this analysis cannot be generalised, since it is likely that formal and informal expectations concerning the platform will continue to emerge. It would be interesting to look at other departments of the same organisation to determine whether any other types of logic (for example, those related to professions, the market situation, labour law or micropolitical strategies) also shape the emergence of formal and informal structures at the technical system’s boundaries. Finally, there is room for comparative research on other types of organisations that use the same technology.
Within the scope of this study, we have shown that the reactions of organisations to the introduction of technical systems are contingent and cannot be determined in advance. We hope to see more in-depth studies of technical systems from an organisational sociological perspective in the future to complement our present findings.
© Lene Baumgart, Pauline Boos and Bernd Eckstein, 2023.