When I tried them for the first time, I realized the first two principles are easy to follow. The third one needs a bit of pondering and the last two are a bit tricky and takes a while to grasp and apply. Watching my work mates in different workplaces/countries, I would say they’ve experienced the similar learning curve. Maybe that is the reason for putting those principles from easy to hard order: S.O.L.I.D
The question is, if we are dealing with a legacy code and we are asked to fix the current issues, should we follow the same order?
Case study: The Novopay
Talent2 is an Australian HR company. They spend 7 years to implement a web-based payroll system for schools in New Zealand. The New Zealand Ministry of Education paid $182 million and waited total 10 years to receive a nightmare called: Novopay.
If you search Talent2, you will find that they are trying so hard to bury their past via re-branding and run away from their collateral damages. The fact that still troubling me today is why Talent2? The NZ government couldn’t find the right talent inside the country huh? I mean there are visionary IT people inside NZ (People who created Xero, TradeMe, SilverStripe, etc) and yet a payroll project was outsourced to Talent2, completely ignoring local talents.
Now if you’ve spent $182 M and waited that long you expect something meaningful in return right? Wrong! Since day one, serious problems popped out from every corner of Novopay. About 8,000 teachers received no pay, over pay, under pay and in some cases negative pay. We could call it occasional defect if it was a handful of schools, but when %90 of NZ schools nationwide experiencing the same thing, it means the system is not even in the alpha version. It is not even a system. It is an expensive joke.
But Prime Minister John Key had a different opinion. He was informed there were 150 software defects yet he said:
“Most big systems of this magnitude will have some warning bells attached to them so of itself it’s not unusual”.
Cute! Isn’t it? Perhap everyone should consider a system with 100k users, a MAGNITUDE!
The other day I was talking to Mikey McLellan (Tech Manager at Neighbourly and I need to add I really like their tech ideas and the way they bring communities together.)
[Commercials]: If you are a senior developer looking for bright people to work with, send your CV to Neighbourly.
Mikey mentioned that they have about half a million registered users in their system so far. Well according to PM. John Key, neighborly can be measured as a 5 MAGNITUDE system 🙂
This is a test. That’s an apple. Those are birds. Questions?
Ask any mid-level developer and they will tell you there is a reason to write tests. You write tests to catch and fix any defects before releasing the final product. However Education Minister Hekia Parata, Finance Minister Bill English and Associate Education Minister Craig Foss all signed off on the Novopay, despite 5913 payslip errors during testing.
How possibly you ignore 5913 errors? As a developer probably you won’t, but Leanne Gibson (Ministry of Education chief information officer) said they were not “showstoppers”.
I think all sloppy IT companies should follow Talent2 approach. They should hire politicians to cover up their faulty systems. A match made in heaven.
To make it sound even worse, Novopay was assessed by PWC, Ministry of Social Development, Ministry for Primary Industries and NZ Transport Agency and they’ve showed green lights. All of them!
- On the side note: I was wondering, why nobody questioned these independent advisers after Novopay disaster? They use tax payers money to stay in the office, then get paid again by tax payers to advise on quality of a product. So when things go south, why they should be immune from investigation?
- Question number two: How on earth a ministry could be considered as an independent adviser for another ministry?
- Final note: PWC! you put another big black mark on your reputation. Again!
Let the show begins
Despite of these alarming numbers, everyone in the NZ government, did some BS justification and in August 2012 they let the monster out. Within three months of its release, education staff were owed around $12 million. On top of that 255 invoices were sent to the ministry for extra costs regarding Novopay payment errors administration. Total value of those extra invoices: $1.197 million.
Then the hell broke loose. People started asking questions. Who is Talent2? Where is that contract? Who signed it? Why it took so long? (originally it was $30M contract and the deadline was 2 years). What are the extra costs? and who authorized them?
With a little search you can find individuals like ‘Alex Harris’ who’ve practiced their social rights and demanded to see that dodgy contract.
Read the whole conversation here and see how the government tried to jump around and do all sort of magic tricks hoping he will give up. And check out at the end how they provided a link saying: you can find the contract here: http://www.minedu.govt.nz/theMinistry/NovopayProject/ProjectInitiation.aspx
But when you click on the link, you don’t see the contract, instead you will be redirected to the Ministry of education home page. Surprize!!!
“Winston Smith: Does Big Brother exist?
O’Brien: Of course he exists.
Winston Smith: Does he exist like you or me?
O’Brien: You do not exist.”
– George Orwell, 1984
The ‘Drunk Devs’
The good thing about meetups is you find like minded people who like to hang out with. I used to go to a couple of them in Auckland where I found a couple of talented developers who shared the same interests and passions apart from the tech subjects. We started to go out for coffee, hangout during the weekends and go to quiz nights in the Viaduct area.
If you happened to be in Viaduct bars (and are interested in Pub-Quiz), You might’ve heard about a group called ‘Drunk Devs’. That was us.
After winning a few times in a row, quiz nights became so predictable and boring so we set new rules.
- Rule number one: Four of us should get drunk before the quiz, and are allowed to join only if the breatholyzer shows 0.13 BAC at least. One remains sober to drive the bunch to the battle zone and back to their homes after the quiz.
- Rule number two: The driver is not allowed to answer the questions. He is in charge of filming us during the session. (Oh man, sh!t that we’ve said/done those nights. Those are by far the funniest and the most embarrassing clips of my life.)
- Rule number three: There is no glory in the winning but all of us will be punished (including the sober one) if we were NOT in the top three at the end of quiz night.
- The punishment: for one week you will code in the programming language that you hate the most! If your work doesn’t allow that, then you join one of the HackerRank competitions or coding boot-camps related to your undesirable language and earn the completion/winning badge at the end. You are not welcome to the group unless you finish your punishment.
Then one night we didn’t go to the quiz. Instead one of us asked a 2 million dollar question: What do you think about Novopay?
Novopay the ‘Drunk Dev’ version
There was a pause in the room and he continued: “Look the Novopay problem has been around for a while now and since the teachers are not happy with the result, there is a chance that people like us might get a shot to fix the system.” Then he added: “Hypotatically, if I knew someone, who knows someone, who can connect us with this project, my question is are you up for it?”
Then all of us started to talk/ask questions simulteounously. When initial excitement passed, the concerns boilded down to three main questions:
- How big is the problem?
- How much do they pay?
- How much time do we have?
The answer to all three was “I don’t know”. However he said he can show us the ERD (Entity Relation Diagram) if we sign a zillion pages NDA first.
We all did, and then he showed us a bunch of A2 papers containing Novopay ERD. My first impression? I’m out!
But the energy in room was building up and everyone was talking how they can tackle it easily and deliver the project in no time. At the end they measured it as a project that can be fixed by five of us, for 2 million Dollors, in 6 months.
You see that is the problem with wishful thinking: the unreal numbers tend to shutdown the rational thinking process… every time!
This is equal to the same feeling when you think you love someone. When your brain releases Adrenaline, Dopamine and Serotonine in your blood stream, you feel in love with the IDEA of a concept, not the concept itself. (replace the concept with a person, a pet, a toy, a project, whatever. The mechanics works the same).
You can see that easily by watching how they phased out from the problem itself and slide into the dreamland and how each one started to dream about spending his share of $400k after 6 months. By the way that 6 months work equal to the salary of a Senior developer for four years, right? It all makes sense when you have an employee mindset.
Iran Ministry of Agriculture
Before that my life journey led me to New Zealand, I did a 2 years contract with Iran Ministry of Agriculture. I wrote a system to evaluate plans for Hydroponic Greenhouses to: a) find out if they are doable and economically feasable, b) predict if they are still profitable 3 years after hitting the break even point.
Ignore the big words here, the bottom line was that project skinned me alive so many times. The beurocracy hell aside, even thinking about the technical challenges that I’ve faced during that two years makes me feel nervous today. The result was far from ideal. I admit that I could have done a better job if I had more time. But I did keep my end of bargen and delivered a functional system. They were happy with the result and thrilled to see how a usual manual paperwork process – which used to take up to 4 months- is done in seconds now with an accuracy far beyond their expectations.
Just for the records: No! I did NOT used any ML or AI algorithms for that project. I didn’t know anything about Machine Learning at that time. I just streamlined a bunch of mathematical formulas which has been created by their analysts.
A few weeks later, Iran Ministry of Indistry, Mine and Trade asked me if I can implement the similar solution for their own plans. But the political climate changed dramatically after the coupe in 2009 Iran presidency election. The sparks of the Green Movement born in Iran, later turned to a raging fire called Arab Spring and caused the fall of several dictatorships in the region (Egypt, Tunisia, Libya, Syria, Yemen, …). At the end, I (we) got paid with bullets (long story).
The point is, when it is about government projects and the related challenges, I believe I know a thing or two. So that night in Auckland when everyone was happy and excited about the – possibility – of getting a great project done, I was back in Iran, living a simulated ordeal in my head. At that night, clearly I was the last person in the room who people wanted to listen to. So I decided to keep my mouth shut and talk about it when everyone is a bit calmer.
Luckily the NZ Ministry of Education saved me from that trouble when a few days later, they announced they are going to pay another $45M to someone who has the potential to fix the problem. And that someone is [drum roll please]: Talent2 again! Yey!
Naturally I can attack this decision from a million angles but lets save the energy and mention only one fact:
I’m sure there were so many dev teams in NZ who cared about the whole situation (definitely not as cool as Drunk Devs though) and truly believed that they could do a better job than Talent2.
The fact is, these people were willing to stand up for their national tech problems and they were more than happy to offer solutions for a problem with that MAGNITUDE (Mr. Key!). These are the people who don’t blame the world for their problems, instead they do what it takes to make things better for everyone. As a leader, when you insist to make rookie mistakes like that, I don’t see any reason why people should support you in the next election.
In An Ideal World
Sometimes I ask myself if we had a shot to offer a solution and if I was in charge, how would I handle that. In other words, imagine that they asked us to fix the Novopay. How would I plan and execute a reliable/maintainable solution for teachers payroll system?
Well for the starter I would shake the numbers noticeably.
This not a $2M job. It is at least a $10M one.
It can not be done in 6 months, 1 year is the minimum.
5 Developers are not enough to get it done. We need 21 developers and 7 designer.
The logic behind those numbers can be summarized in one sentence: We can not fix the system, we need to rebuild it from scratch. However we can use the functioning bits and pieces and migrate them to a Service Oriented Architecture (SOA) solution.
My first impression when I saw the ERD was that I’m looking at another monolithic application. (there is One database, One API, One/Multiple clients who expecting everything from One server). Bottom line we are about to face one giant spagetty code. That’s why it is not fixable. That is the very reason they got into trouble at the first place. The original solution was designed by the wrong Solution Architect/DevOp (if there was any).
So we can’t afford to continue the same mistake, instead we should create our own MicroServices and (if there is any) well written/well documented code we can migrate them from the old code to its new home inside a microservice.
Reordering the SOLID Principals
Looking at the ERD, I realized this project can be broken down into 7 smaller pieces and we can put each piece inside a new micro-service. That means we need to hire two more senior developer to make sure there is one senior dev in charge of each micro-service. It should clarify that technically we are implementing 7 projects using SOLID principals and each of them has its own echo system and at the end they communicate with each other to serve a higher goal: Processing Payrolls.
For each micro-service, we need another 3 members (designer, front-end dev, mid back-end dev) and that’s how they should work:
weeks 1 & 2 (7 devs): All senior devs check the code and find all useful portable classes from the legacy code
weeks 3 & 4 (7 devs): All senior devs agree on the services for each micro-service and create the blueprints (abstract classes and interfaces) for their own micro-services.
week 1 (7 devs & 7 designers): designers for each team join in and create the wire-frames based on the agreed blueprints
week 2 (14 devs & 7 designers): front-end devs join in after a week design and (while designers continue their work) they start creating failing UATs (user acceptance tests) + initial templates
week 3 (21 devs & 7 designers): mid back-end devs join in the third week and their job is to create all failing functional + unit tests and behaviors based on the provided wire-frames and interfaces
week 4 (21 devs & 2 designers): by now all initial UIs should be finalized so I would keep the top 2 designers (for UI updates) and let the rest go. By the end of this week all templates and UAT tests should be finalized as well. Senior and Mid back-end developers continue on their BDD and TDD tasks.
3rd, 4th and 5th months(21 devs & 2 designers):
From 3rd months onward the road map should be clear and the whole work force should be spend on implementing the logic. All classes (services) for each section should be created and verfied against the tests that have been already in place. That means from 3rd month onward we can show progressive results to the client (Ministry of Education) on a daily basis. So if there is anything wrong we can deal with it on the spot (no need for 10 years wait). If a change request showed up, or if there was a wrong UI, logic, userflow, etc in the code they can be caught and fixed immediately. Since there are still two designers on the project, even changing/updating the UI shouldn’t be a problem.
6th, 7th and 8th months (9 devs & 1 designer):
From the 6th month onward, we already have a good implementation pace and don’t need the whole team. That means apart from 7 senior developers, only one designer, one front-end dev and one back-end dev stay and the rest can go.
9th month onward (5 devs):
By the end of last month the whole project should be finalized. However project manager always recommend considering an extra %15 resources for errors. I ususally consider %30 room for unexpected situations. But since this is a government project I would add another 10 and consider %40 extra resources. That means by 9th Month the core team (Drunk Devs) stays and the rest should go. If there was any problem by then the core team have a comfortable 4 months to handle it in an easy and relaxed manner.
With that plan we need to reorder the SOLID principals. Since we are beginning by migrating the legacy code, the ‘D’ pricipal comes first.
Dependency inversion principle: one should “depend upon abstractions, [not] concretions.”
We are creating Service Oriented Architecture, by migrating bits and pieces of original code. That means we need to abstract the foundation of each service as much as we can. In other words before copying and pasting the old parts to the new system, we need to create all abstract classes and interfaces which describe the duty of each tiny service inside the microservice architecture. That is where the ‘I’ principal will be introduced.
Interface segregation principle: “many client-specific interfaces are better than one general-purpose interface.”
To be more precise, in this project the ‘D’ and ‘I’ principals would be specifically a senior dev’ jobs and they are in charge of creating the foundation and orchestrating their team (designer, front-end, mid back-end) toward their microservice implementation. But ‘S’, ‘O’ and ‘L’ principals is for all devs (senior, front-end and back-end).
The moral of the story is this: in order to migrate/renovate/maintain a legacy code it is ok to reorder SOLID principals into DI[SOL]. (‘D’ and ‘I’ come first and the ‘S’, ‘O’ and ‘L’ can be placed with any order that satisfies our needs.). Having that said, the SOLID principals are not written on the stone. In other words you might need a different order of SOLID depend on your project.
Thanks for reading.