PART 1: Bicycles and pilots (no, not airplane pilots)
A few years ago, the City of Johannesburg begun building bicycle lanes in various neighbourhoods of the city. The idea was ostensibly intended to diversify the city’s transportation options, and enable transport via bicycle to grow into a new, valuable commuter option. In the area I live, a mixed suburb with a fairly large student component, the bicycle lane project turned into a significant construction project: the roads where bicycle lanes were build were entirely resurfaced, the pavements completely overhauled (from tarred pavements to brick paving), bicycle lanes installed on both sides on the road through the use of “physical bump” yellow lane demarcations (not the painted version), and poles with camera monitoring facilities mounted on the corner of the neighbourhood blocks. It struck me at the time as a massively expensive exercise, and I wondered whether this project had been thought through properly (was it really necessary, for example, to re-tar the roads?)
In the years that have followed the construction of these lanes, there has been very little use of the lanes. At least by cyclists (the informal recyclers and their trolleys use them a lot more!). You might see the odd guy on a bicycle, but most days no one uses them for their intended purpose1.
The project has also faced criticism in other quarters. One of main political opposition parties, the EFF, attacked the project hard, claiming it was a solution not designed for the poor, and that “SA does not have a cycling culture.” My reservations about the design and execution of this project have, as a result, now grown: “Is there a bike sharing programme to encourage use of these lanes? Are the security concerns of potential riders truly addressed? Are there cultural barriers that prevent the wide-spread use of bicycles?” etc.
Mostly though, I’ve been thinking about whether this project could have benefited from a pilot version of the system? Or if there was a pilot version before this, could the pilot have been better designed and implemented? Or, if this is the pilot version (expensive pilot!), is it designed correctly to inform the full implementation? I should be clear that I have not studied this project in a lot of detail, and so it’s quite possible that many of my concerns and thoughts were addressed during the design phase. Nonetheless, it serves as a good anecdote to introduce a broader concept I’ve been thinking about for a while.
This concept is the idea that the design of pilot projects requires specific (and unique) attention, as compared to the design of the broader/final project.
Pilot projects are a popular concept, designed (in my observation) to address uncertainties and manage risks. In many companies and industries, a new idea is often put to test via a pilot version, usually designed to be much smaller in scale than a “full roll-out,” and designed (in theory at least) to limit the impact of its failure on the broader system/company/project. Pilot projects are run in all sorts of settings: from nation-wide policy projects like South Africa’s National Health Insurance, to first-of-a-kind (FOAK) engineering projects like the research renewable energy projects that got the industry started some decades ago, to “beta” software products rolled out by tech companies like Google to a limited customer base, to new packaging for detergents trialled by manufacturers.
Pilot projects are also called vastly different things in different industries. Test, trial, beta, FOAK, demonstration, Minimum Viable Product (MVP), Version 0, experiment, are all terms used to describe pilot projects. Piloting an idea is to test it before a full implementation, and is therefore a natural choice for many industries and management teams, albeit using different branding.
I’ve seen the “pilot project” concept act as a default response by management to new, risky ideas. As a mechanism to manage uncertainty and risk, I think it’s fantastic, provided it’s value is made clear (not all ideas, even risky ones, justify a pilot) and that the pilot project itself is designed well.
PART 2: Complex socio-technical systems
My suggestion that the design of pilot projects requires specific and unique attention, dovetail nicely with another concept I’ve had an interest in for a long while: complex socio-technical systems (CSTSs). Before I explain how this link between the two concepts works, I want to describe the concept of complex socio-technical systems in a bit more detail. To do this, I’m going to use a talk by Professor Joseph M. Sussman on the idea of CSTSs, and his rationale for considering this concept to be a field of study. This talk was delivered as the 2012 edition of “The Annual Charles L. Miller Lecture” at MIT. Although 5 years old now, I think this is a great talk, outlining many important ideas about CSTSs, and setting up a number of important follow up questions (the design of pilot projects being one of them, in my view).
The video can be found on Youtube here. A similar presentation (using the same ideas) by Professor Sussman as part of the MIT SDM Systems thinking webinar series is embedded below.
The full talk is really worth watching. Below I’m going to summarise my understanding of Prof Sussman’s presentation, and in the process build up to my explanation of the linkage between pilot project design, and complex socio-technical systems.
At the start of his talk, Sussman gives an explanation of what CSTSs are. He parses the phrase, and begins by giving a fairly generic explanation of systems: a system is a concept describing the collection of inter-connected components and sub-systems with various types of relationships between them, and with the elements outside the boundaries of the system. He describes a few of the properties of a system, particular more involved systems: they are typically non-linear, have feedback, and are often stochastic. In the footnotes below2, I’ve included some definitions of systems that I have found useful, which are also products of the MIT System Engineering focus (it’s fairly clear that my research has so far had an MIT bias, which is possibly due to them having an incredible competence in the area of System Engineering).
Sussman goes on to outline the idea that the “technical” part in socio-technical describes the idea that the system under consideration contains technology (which I personally understand as elements that are design and built by people). The “socio” phrase refers to the same system having significant social, political, psychological, and economic relevance and impact. To describe the complex characteristic of CSTSs, Sussman lists and describes a number of different types of complexity, each of which give rise to the system being difficult to understand, and more importantly from my perspective, difficult to predict.
After describing CSTSs and what the idea represents, Sussman goes on to make the case for why “complex socio-technical systems” should be considered a field of study in its own right (I’m summarising the talk, so I’ve skipped over a number of other good points Sussman makes in his presentation). In his case, he presents two (in my assessment, not necessarily his) “proofs.” The first of these proofs is that CSTSs as a field passes an important question: Are there principles and core underlying concepts for creating an integrated approach across disciplines that allow us to avoid solving CSTS problems each time from the ground-up? Put differently, and more succinctly, is there “transferable residue” from studying one CSTS that can be used to analyse and design another CSTS. As an example, he outlines the case of solving (or trying to design improvements to) the case of Mexico city’s transportation challenges. If there are ideas and methods that emerge from studying the complex socio-technical Mexico City transportation system that can be used to study other CSTS (say, the transportation problem for another city, but also perhaps the healthcare system of country), then there is “residual knowledge” available from this exercise, and therefore a field of study.
Sussman argues that there are indeed core principles and underlying ideas that can be used to approach CSTS in general (across many domains), and that these therefore form the basis of this field of study. Some of these that he lists are: the macro/micro question, life-cycle properties of a system, flexibility and resilience, feasibility (satisficing, not necessarily optimising), system architecting as an organising principle, and a few others. He describes some of these in more detail, and illustrates how these ideas cut-across different domains but yet all assist in understanding and designing better CSTSs.
These core principles and underlying ideas are of particular importance to this post, since I hope to argue later (in part 3) that the “design of pilot projects” is one of them.
The second proof that Sussman describes (although he does not present it as a proof as such, but to my mind it answers important questions that a field of study must answer) is that CSTS, as a field of study, has both increasing relevance (the demand side) and also is increasingly enabled by the tools and technology that exist today (the supply side). In making this point, he points out that studying CSTS is both more necessary and more possible than ever, and as a result, should be pursued as an independent field of study.
After this, Sussman makes a range of important points, but I want to conclude my summary by focusing on just one: he argues that that the concept of “engineering systems” is equivalent to the concept of “complex socio-technical systems.” He suggests that they are one and the same for all intends and purposes. For my part, I think that this is true (there is no engineering system without a social impact), but there is an important “branding” issue which must be addressed before we can safely declare an equivalence between them3.
PART 3 – Linking “Complex socio-technical systems” and “Designing for pilot projects”
At the start of this post, in part 1, I described my view that pilot projects are primarily a risk and uncertainty management mechanism. They are used, by and large, to test new ideas whose outcomes and performance are unclear. Risk and uncertainty are, incidentally, two of the features of CSTSs. In complex socio-technical systems, any new system design is fraught with uncertainty related to technical performance and its impact on people. Complexity, by definition, gives rise to uncertainty, and as a result pilot projects are often implemented in complex socio-technical systems (CSTSs). CSTSs provide a natural “habitat” for the implementation of pilot projects.
This presents a particularly convenient solution for bringing together my interests in CSTSs and “designing pilot projects”: the type of environments that most benefit from well-designed pilot projects are complex socio-technical systems. Further, and perhaps more importantly, when viewed from the perspective of the CSTS field of study, “designing for pilot projects” can be thought of as one of the core principles or underlying ideas that form the basis of the field. Like other core principles that Sussman presents as being the basis of the field of study, I suggest that the knowledge and methods embedded in the concept of “designing pilot projects” is applicable across different domains in which we find CSTSs. A pilot project to test blockchain usage on an electrical distribution system by a utility, and another pilot project to test a radically new hospital design, have a number of similarities, despite being in different domains entirely.
The field of CSTSs then, provides an extremely useful framework for the idea that designing pilot projects requires specific study and attention. Framing “designing pilot projects” as one of the core principles of the CSTS field of study allows us to take advantage of the other principles of the field when building this particular core principle, like for example the idea of using flexibility to manage uncertainty. The field of CSTS, and more broadly the field of “engineering systems” provides a convenient “intellectual home” for the research work into how designers across different domains might design better and more useful pilot projects.
PART 4: Principles of “Designing pilot projects”
In part 1 of this post I introduced the idea of pilot projects, and suggested that the design thereof requires specific attention. In part 2 I employed a talk by Prof Sussman to lay out the concept of complex socio-technical systems. In part 3, I (hopefully), demonstrated the relationship between the idea that designing pilot projects requires specific attention, and the field of CSTS.
In this last part of this post, I want to set out a very brief and draft idea of what I suggest is entailed in the idea of “designing pilot projects.” I’m going to do this by suggesting a number of principles that I propose underpin the design of pilot projects.
Increased flexibility in pilot projects
The idea that building flexibility into CSTS enables a system to maximise value over time is not a new one, and has been explored at length by many authors. The book by Richard de Neufville and Stefan Scholtes4, entitled “Flexibility in Engineering Design” provides a comprehensive overview of the subject.
So, while it is clear that flexibility should be a clear part of the design of any CSTS, I suggest that the flexibility included in a pilot project should be even greater than the flexibility built into the final/full-scale version of the project. This is premised on the idea that during the pilot project, it would be preferable to be able to test many different configurations of the system, and also be able to respond to variables that were uncertain during the design (and are now better understood). The greater the flexibility designed into the pilot project, the easier it is to test a large range of different system designs, and therefore the more valuable the pilot project becomes.
A high level of flexibility in the pilot project may also give better insights into which types of flexibility should be included in the final/full-scale version of the project.
Using the bicycle lane example I introduced in Part 1, a pilot version of that project may include the ability to change the actual bicycle lane routes (by, for example, designing the physical lane demarcations to be easily removable and installable on another road/route), while the full-scale version of the project may not include such a high degree of flexibility (but maybe flexible in other ways besides changing routes). Further pilot project flexibility in this example may include enabling the video monitoring system (mounted on poles on the street corners) to be able to communicate in many ways with bicycle riders (say, via a screen capable of colour video), while the full-scale version may include a more restricted number of possible communication options (say, a simple LED screen).
I want to suggest therefore, that one of the key principles of designing pilot systems is to include high levels of flexibility in the design (higher than the full-scale version), which allows for a broader and more comprehensive “test” to be conducted by the pilot system.
Measurability of pilot projects
Another key principle of designing pilot systems is building comprehensive measurement systems into the pilot project. Since pilot systems exist to test ideas, its essential that these systems are designed with the necessary data collection capability that enables evaluation of its performance. This covers both the measurement of technical data (for example, how many bicycles crossed a certain point on the bicycle lane route, or how many KWhs a customer bought under a pilot tariff system, or how many x-rays per patient did the new health insurance system use), through to the “measurement” of the more social aspects of a system (for example, “how positive were people to have bicycle lanes in their neighbourhoods?” or “how satisfied were patients with the level of service in the new health insurance system?”).
The idea behind comprehensive data collection in pilot systems, both in the technical and social spheres, is one of the key the tools we need to comprehensively test and monitor the performance of a pilot system. The data collection infrastructure in pilot projects should be more comprehensive (and therefore probably more expensive) than the data-collection system designed into the full-scale system.
Practically this may mean many more physical sensors, increased data processing ability (as an example, in our bicycle lane case, counting the number of users of the lanes could potentially be done by using machine learning techniques to scan video for bicycle riders, processed “in the cloud”), the inclusion of ongoing sociological and psychological assessments and many other types of data collection being built into the system.
Value Engineering and Minimum Viability of Pilot Projects
A further potential principle in designing pilot systems is related to the ideas of “Minimum Viable Product (MVP)” and “value engineering.” It asks the question: which part of the pilot project really needs to be built, and if it does, to what extent? Put differently, and borrowing the MVP idea from the lean start-up world (particularly Eris Ries’s book), which parts of the pilot system are the “minimum” components without which our pilot test would not be “viable?”5 The value engineering contribution focuses attention on costs and function, and asks “could these functions (in the case of pilot projects, functional largely means “testing”) be achieved at lower cost without sacrificing function?
Linking back to the bicycle lanes example, this principle should assist with whether or not the roads really needed to be resurfaced for a pilot version of the project (and if it did, could just the bicycle lane be resurfaced, and not the entire road?). And further, perhaps the pilot project could have been used to determine whether the resurfacing of bicycle lanes would be necessary even in the full-scale version? (perhaps some sort of A/B testing could have been done in a pilot version, where part of the cycle lane was resurfaced and another part not, to test its impact).
Scale and architecture of pilot projects, with respect to the “full-scale” version
Since pilot projects are very often smaller in scale than their final version/full-scale counter parts, another principle of designing pilot projects relates to the relationship between the architecture of the pilot system and the final system, in relation to scale. The concept of system architecture used here is the one proposed by Edward Crawley, Bruce Cameron, and Daniel Selva, in their book6 “Systems Architecture: Strategy and Product Development for Complex Systems.”
An example is instructive here: using the bicycle lane example developed throughout this post, the pilot system may be designed for a single route and a very small number of cyclists. However, the full version may (probably) have the intention to handle a large number of cyclists over multiple routes. Since the width of the bicycle lane has a strong bearing on the overall system performance, it is an architectural decision7. (the wider the lane, the more cyclists can be accommodated in the lane). The width of the lane in the final/full-scale version will probably need to be fairly large, to accommodate the large number of intended cyclists, while the width of the same lane in the pilot system need not be very large. However, building a narrow bicycle lane into the pilot will not allow for a true “test” of the final system, and pilot project lane width will need to be larger than technically necessary. Therefore the architecture of the pilot project is conditioned strongly by the architecture of the final system, particularly with regards to the dimension of scale.
The resulting principle is, I suggest, that the architecture of pilot projects should be evaluated in terms of the architecture of the intended final design, with careful attention paid to the impact of scale.
There are likely more principles that are applicable to the design of pilot systems, and which can be used across domains (health, energy, manufacturing etc.) in the design of complex socio-technical systems. For now, this seems like an instructive juncture at which to invite feedback on the ideas presented in this post.
|1||A low capacity factor in the early years of the infrastructure's lifespan is not necessarily a terrible outcome in my view, but it does need to be part of a larger plan|
A good definition that I like comes from the excellent book on Systems Architecture by Crawley et al :
"A system is a set of entities and their relationships, whose functionality is greater than the sum of the individual entities. A complex system has many elements or entities that are highly interrelated, interconnected, or interwoven."
Another useful definition is from the book on Engineering Systems by de Weck et al :
"We define engineering systems as A class of systems characterized by a high degree of technical complexity, social intricacy, and elaborate processes, aimed at fulfilling important functions in society."
 Crawley, Edward; Cameron, Bruce; Selva, Daniel. Systems Architecture: Strategy and Product Development for Complex Systems (Page 9). Pearson Education. Kindle Edition.
 Olivier L. de Weck;Daniel Roos;Christopher L. Magee. Engineering Systems: Meeting Human Needs in a Complex Technological World (Kindle Locations 480-481). Kindle Edition.
|3||There remains the idea in some engineering circles, particularly among older engineers, that an “engineering system” can be defined by drawing the boundary of such a system tightly around the purely technical components. I’ve seen this happen in the case of power stations, where the system is removed from the broader economic and social context, and considered to be a “big machine” required to produce power. Since the machine does have many components (turbine, boiler, generator, fuel supply etc.), it is considered a system, but only in the technical sense.|
|4||de Neufville, Richard; Scholtes, Stefan. Flexibility in Engineering Design. The MIT Press. Kindle Edition.|
|5||Eric Reis, in his book “The Lead Startup” notes that “A minimum viable product (MVP) helps entrepreneurs start the process of learning as quickly as possible.” While this is in the context of consumer products (and mostly web based ideas), this quote drives home the idea that the MVP concept is used to understand what works and what does not. Another quote, “The minimum viable product lacks many features that may prove essential later on,” illustrates that not all components of the pilot system need be built.|
|6||Crawley, Edward; Cameron, Bruce; Selva, Daniel. Systems Architecture: Strategy and Product Development for Complex Systems (Page 312). Pearson Education. Kindle Edition.|
|7||Using the notion of system architecture as a decision making process, as suggested by Crawley et al , those decisions (that) “significantly influence the metrics by which the architecture is evaluated”, are architectural in nature. Following this logic, the width of a bicycle lane is part of the system architecture.  Crawley, Edward; Cameron, Bruce; Selva, Daniel. Systems Architecture: Strategy and Product Development for Complex Systems (Page 313). Pearson Education. Kindle Edition.|