SomethingSomethingSomethingDarkSide

I was in the U.S. for the Black Friday sales, a traditional day of discounted sales and shopping frenzy in the American Thanskgiving holiday period. We bought some real bargains over the weekend and hit the restaurants to celebrate with a lavish meal on Monday. Walking into a swish eatery to find it deserted I was informed in hushed tones… “it’s Cyber Monday. Everyone’s at home looking for bargains on the internet”.

Cyber Monday this year broke all records. American shoppers have been conditioned to expect deep discounting online on the Monday after Black Friday and in a virtuous circle retailers have learned that if they provide discounts on the day they can expect large volumes of online purchases. But beyond all that, it has created an engine that is rapidly teaching consumers that shopping online is good… after the frenzy has died down we’re left with the base figures: according to comScore a 26% year-on-year increase in online sales revenue for the weekend (effortlessly breaking the $1 billion mark), and over 50 million Americans shopping online (up 35% on the previous year). There is no reason to expect that this is anything other than a fundamental and irreversible change in customer behaviour.

Three players in particular stand to make the most of this new world and are busy laying out their strategy in something akin to the heady days of the first internet boom. Each is a seasoned survivor of those early days and each is coming at the ecommerce marketplace from different angles.

Firstly, there’s Google. They understand the power of search and have built a business on the back of monetising eyeballs through their ad service. They need to be at the start of every purchasing path to ensure those eyeballs don’t end up somewhere else. Google Shopping is their key play here; its main aim is to acquire and aggregate rich product data from as many stores as possible to facilitate the widest search. It’s interesting that they have also just announced that they will be releasing a next-day delivery service for retailers to tap into, leveraging their relationships with courier and express shipping companies. A third plank is Google Wallet which, integrated with the Android mobile platform, has the potential to turn every mobile into both a shopping device and a wallet. However, it’s difficult to see quite how these threads weave together for retailers and shoppers, even leaving aside the question of which elements are U.S.-centric. The killer app here will be whether Google succeeds in getting a rich product feed with inventory accurate to store level across a wide enough range of retailers. Turning up in store with a Google Checkout coupon on your mobile phone to take home there and then the product you already pre-purchased would be compelling, certainly.

The next player is Amazon, who have signalled intentions to encroach ever further into the traditional realm of bricks and mortar retailing. The purchase of Zappos demonstrates a clear direction towards moving more of the high street spending online; their feed integration infrastructure and APIs are also heavily geared towards letting retailers come onboard the Amazon platform as fulfilment partners, integrated through the Amazon checkout process (this is also the model that Westfield, broadly speaking, employs). This model lives or dies on the quality of its retailer interfaces since there is a heavy reliance on getting good product feeds, accurate inventory and timely fulfilment. This can be a hard nut to crack.

Then there is EBay. In retrospect, it looks like they have been heading steadily in one direction for quite some time but it was only with the recent XCommerce launch that their pitch for the future face of ecommerce became readily apparent. They have an urgent need to be known as more than just an auction house for garage sale items, an image that will be holding many retailers (for whom brand positioning is their prime asset) back from integrating with EBay; without a feed of quality, new items from respected brands EBay will have a hard time shaking their current image and grabbing a share of the e-tailing pie.

But let’s look at what they now have under their roof. Firstly, Red Laser to scan barcodes and launch a product page on the shopper’s mobile phone right there in the store. Then there’s PayPal (and its mobile-optimised express checkout) to collect the money. But that’s not really enough… they have brought Magento (an open source ecommerce site engine, poised to do for shopping websites what MySQL did for databases a decade previously) and GSI into the fold. This offers the retailers the prospect of transferring onto ecommerce platforms that run shopping sites, support checkout including PayPal, handle inventory management and other back-of-house functions with a small cost footprint. This provides the kind of cradle to grave shopping path that has Google worried, but their approach goes further with the introduction of the Fabric, an internet-scale service bus that allows all the potential players in ecommerce to tie together to provide pluggable services to retailers such as fraud management modules, warehouse management systems, payment gateways, etc. It goes without saying that PayPal, Magento, etc. are part of the fabric from day one and thus already providing sufficient Lego blocks to build a viable ecommerce business upon. If they succeed in building up an interoperating ecosystem of ecommerce modules, EBay could quite possibly become to online shopping what Facebook’s app universe did for social networking.

So, we have the underlying barrier to be overcome: all the product feeds, smart searches, payment systems, slick checkouts and value added services in the world are useless without the retailers’ ability and willingness to fulfil. Amazon has the edge here, being an integrated retailer already with deep experience across the entire supply chain. But the model can only scale so far before it meets the likes of Walmart and Tesco coming the other way with their deep, world-scale logistics experience, all battling for top spot in global consumer sales.

The bigger (but tougher) pie is all about clipping the ticket on purchases made with other retailers rather than seeking to act as the retailer directly. Google has many of the required elements and a track record of bucking conventional wisdom to achieve the apparently impossible, but creating an integrated supply-chain-as-a-service could be a stretch even for them.

This leaves EBay. Their apparent strategy of supporting systems and integration points to help the various players in the ecommerce space come together to service the customer means less revenue for them per sale. It has potential to be a lot more sales though. Out of the three, they may be just nibbling around the edges of the pie.

But that particular pie is vast.

Back when I was a lad, good things were Ace. Later they came to be described as Wicked instead. Then Bad, then Sick, then (apparently to differentiate from merely sick) Fully Sick. I stopped tracking the phenomenon at that point, so I have no idea what the kids use now, though I suspect it’s constructed completely of consonants in capital letters.  It’s as if each new teen wave felt the need to re-imagine language again, and there are I’m sure reams of literature in the psychology press about tribal identities and group-inclusion.

The same is true of software development.  Following a tweet stream on physical architecture (you know, buildings and stuff) suddenly illustrated to me what software development most likely looks like from the outside: it’s a terrain of impenetrable jargon, presided over by tribal units with their own dialects.  No wonder your 10 o’clock product review meeting with the sales team went off the rails; they probably had a hard time understanding what you were talking about.

There’s no doubt to me that this is how some of the tribes like it, and it is definitely what makes Agile more difficult to adopt than it should be for business people.  Imagine you have been posted to a foreign country as their ambassador, just got to grips with their language, and then all of a sudden they change the dictionary on you.  Having mastered terms like project phases and use cases, you’re suddenly dealing with sprints and stories.  Add to this the various flavours of Agile (XP? Scrum? Kanban? Huh?) and it can feel like you’re back to square one, so you have to really want to do it to make it successful. This is also not helped by contingents who use Agile as a cover for another methodology: JFDI. Agile and cowboy programming are poles apart, but under the cover of the jargon the latter has passed for the former on more occasions than are healthy for the former’s reputation.

My first reaction to Agile was, “why have they made new names for things?”, and it felt like when I realised that “Ace” wasn’t cool anymore (dammit).  The reason for the new vocabulary is to avoid bringing in a set of old assumptions from Waterfall and other places – it’s an attempt at a clean slate with no preconceptions. But it also means there’s a steeper learning curve because it purposely veers away from the possibility of homely similarity to what you knew before.  I’m not about to suggest a new set of names for parts of the Agile process… people that are tempted to do that should first bask in the impromptu wisdom of XKCD’s view on standards, but I am saying we need to recognise the workload we impose on newcomers.

In the end, it’s in all our interests to bring everyone along on the journey and put some distance between Agile and JFDI. But the next time you’re explaining to Customer Support about their request needing to be estimated using fibonacci-based points poker to work out velocity for the backlog, just take a deep breath and maybe go over it again.

Scrum as a fully-fledged methodology has been around since the Nineties, as an idea since the Eighties, and has sired many offspring in that time. In some organisations these offspring are given a test straight out of ancient Sparta, where legend has it that newborns were left on a roof to fend for themselves overnight to ensure that only the strong survived. Fast forward two thousand years and a lot of Scrums are also left to fend for themselves and die within a few sprints.

The Happy Birthday goes to the Westfield scrum process. I note that with sprint 42 starting shortly we are exactly two years since sprint zero. It’s fair to say that our scrum process has had to evolve quite a bit since sprint zero, and our organisation with it, so I think it might be worth taking a snapshot at the end of its second year of life.

At the very top level we have a bunch of people coming up with good ideas all the time. Some of these are very large, such as rolling out the platform to the US, some are much smaller, like ways to boost product search indexing. These ideas can come from anyone: the head of the company, a developer, a help desk person, etc. Each is discussed on its merits and if enough people agree (we have a weekly review meeting) it finds its way to the start of the product funnel. An idea’s progress through the funnel can be best described as Darwinian: only the stronger ideas push through to the next stage where they form a cross-disciplinary feature team, where strength is measured in the number of people who care about the idea enough to see it progress.

Once a feature has been created, it is assigned to a cell and stacked in priority order to decide what that cell will work on next, with the possibility that a strong new feature can come to the front of the stack and displace other features. A cell is the group of developers, BAs and QAs who will turn the feature into a set of Epics and break these down into Stories, and then schedule these into the product backlog assigned tentatively to a future sprint.

The way we currently work is that we have four cells of usually around four people, which is looking like a good number: intra-cell communication is pretty efficient and the team is still large enough to deliver a substantial new increment of the feature every sprint. We have settled on a three week sprint since this has yielded higher throughput than two weeks (less time spent on sprint set up and tear down ceremonies, plus it’s long enough to tackle a big vertical stripe – which becomes more important if you have an application architecture of more than a few interconnected services).

The cell concept is also scalable. Individuals can and do move between cells to promote intermixing of ideas, which makes it easier to spawn a new cell if the development pipeline needs to be widened: the new cell can be picked from existing members of other cells, backfilling with new hires, or even outsourced. The crucial thing is that the new cell adheres to the same communications processes as the existing cells so everyone still has an idea of what’s going on across the entire team.

However, scaling cells leads to the standard problem of how to keep everyone informed: inter-cellular communication is a challenge. We came up with two ways of solving this, having discounted the traditional Scrum of Scrums concept since this only involves a subset of the team (we still find it useful to do a SoS that pulls in representatives from the development team alongside customer service, retailer management, etc. but by its nature the SoS only dives into the salient highlights). One idea was to have a single scrum with everyone saying their piece, which only works up until a certain size after which individuals tend to zone out and miss details. The approach we finally settled on was to form an inner circle of cell representatives (an honour that rotates through all cell members) to give their cell’s status, surrounded by an outer ring of everyone else in the team. This lets the entire team dig into particular issues if needed without the stand-up taking fifteen or twenty minutes.

The backbone of the process is JIRA, which has evolved from a bug management tool into a highly flexible story manager through the Greenhopper plug-in. Part of the problem in the early days was that story and sprint planning was entirely divorced from story management, the former being done in an Excel spreadsheet, the latter in a bug tracking tool, leading to the potential for endless confusion and miscommunication. We brought in JIRA and customised the workflows to add in the approval steps our organisation needed and in short time had the business people talking the same language as the development team, planning sprints as easily as dragging and dropping stories from one sprint to the next. Sure, the software costs money, but next time you’re in the middle of a crisis meeting on who committed to what, when, just jot down the collective cost of having all those people in the room for the hour. It breaks even pretty quickly.

As I said previously on Westfield’s Agile approach, it’s not necessarily the steps of Scrum that matter (though there are fundamentals that you need to keep), but it’s Scrum’s introspective ability to evolve itself to fit the organisation. I would now add its ability to radiate information effectively as a feature of successful Scrum. After all, what’s the point of being in a team and being in the dark?

Last week, I got into a robust discussion at work, some might even classify it as an argument, and looking back it was ultimately my fault.  I broke a cardinal rule of successful innovation: I was making a change without having justified it.  You tend to see this happen from time to time, and it goes by a variety of names such as skunkworks, 20% time, secret squirrel projects, etc. and looks to those outside the project that innovation is all about sneaking round processes and people to get an outcome that ultimately few people outside the project are bought into.

But… if you’re doing skunkworks in an organisation you’re throwing away one of the principal benefits of being in an organisation, which is the ability to draw on a wide range of resources to help you achieve your goal.  If you don’t utilise these resources, you’re effectively working as an n-person group (where n is typically a very small integer) in a virtual garage somewhere within the organisation.  It’s true to say x% of garage startups (where x is a large integer approaching 100) die because they aren’t properly supported, either financially or through specialised expertise (how do you expect to build the world’s next Great Widget when you have to do the accounting yourself?).  The same happens with most skunkworks projects, except the run rate is even worse because an outside influence (your management) can quite legitimately make you go and do something else with your day.

Skunkworks works when you get someone outside the team to care about it enough to resource it (going back to the garage analogy, the idea survives because it manages to attract seed funding).  This is what I mean by Doing It Wrong, since you’re hastily retrofitting the steps you should have taken at the start (e.g. securing backing) to allow the project to continue.   To do this right I think you need the following three skills in the team if you’re going to innovate successfully:

• talented people
• a good, well-defined idea or vision
• change management

The first two are self-explanatory, but the third is, I think, the key to being successful at innovation within the organisation: innovation is change therefore you need to understand how to make changes in the organisation.  You have to identify stakeholders who you will need to gather resources from (people, hardware, time, etc.) and bring them along for the ride, from the first blue-sky ideas session to the point where the project is accepted into the organisation’s mainstream activities.  You need to make allies of these people because they are the ones who are most likely to stall or kill the idea (they may still do that, in which case you need to be objective: are their reasons justified? Was there actually something flawed in your idea?).

There are a bunch of change management models out there on the interweb.  One particular model I’ve used successfully is ADKAR. The steps to successful change can be summarised in terms of innovation as:

1. Awareness: make stakeholders aware of the idea. Socialise it.
2. Desire: let everyone know why it’s a good idea. Sell it. Tell them why they should want to do it
3. Knowledge: let people know how to make the change. How do they get to There from Here?
4. Ability: tool up. Make sure you have the necessary people on the job to make it a success. Train them if necessary, recruit them if you have to
5. Reinforcement: embed the change into the organisation, make it part of the everyday operation

This is where I have seen innovation projects fall down.  You may be all over #3 but if you haven’t got #1 and #2 bedded down you don’t have #4 and certainly will find it very difficult to then achieve #5.  The gotcha is thinking you have #4 nailed because you are thinking in terms of the immediate team, whereas you’re nowhere near achieving #4 because you don’t have the other necessary people on the project to make it a success in the long term.

So, going back to to the work argument, what did I do wrong? I had an idea, socialised it less widely than I should have, and tried to kick off the project.  The result was a number of previously unidentified stakeholders scrambling to understand the idea in a compressed timeframe and therefore not buying in, and a rush to get through steps #1, #2 and #3 above in time to kick off the project on schedule.  This is not optimal… innovation is (by definition) not the business-as-usual work that people are used to seeing; it’s going to take longer to understand and needs more mental effort.  Innovation is hard enough without adding this extra pressure.

Innovation is all about a passion for new ideas.  Sometimes this manifests as a breakneck desire to drive change through regardless, but this is doing it wrong.  Similarly, all that you achieve in hiding innovation in special innovation groups with special dispensation is singling out innovation as a foreign body in the system.  In the end you’re highly likely to end up strangling your own idea and getting frustrated in the process.  On the other hand, if you lay the groundwork carefully the innovative idea looks just like a normal project and earns the right to the resources it needs.  If you can stop talking about seeding innovation and start talking instead about laying out a company strategy, you’re doing it right.

By the time you read this it will be out of date. Traffic to Westfield sites coming from mobile devices is sitting on 20% at the end of July 2011. If you trawl the net for similar stories, this figure isn’t remarkable. What is remarkable is that at the start of the year this figure was 10%, by Easter 15% and we have no mobile web offering. We are not serving properly 1 in 5 of our audience at this point in time, and while extrapolation is a tricky business the trend’s pretty clear… it looks like the mobile wave has finally broken.

This is one wave that’s been rolling in for a long time. I was building app stores for T-Mobile back in 2002, back when WAP was the markup standard (for any random definition of ‘standard’) and the main game was J2ME on Nokia. We were sure the wave was less than 18 months away.  By 2004 I was building mobile news sites for smh.com.au and theage.com.au at Fairfax, attempting to play nicely within the walled gardens of the four major telcos in Australia, and the wave was just a year away. I was back at Fairfax in 2007 building the mcommerce site for rsvp.com.au, attempting to navigate the maze of per-carrier premium SMS payment options. With the latter, at one point we reached 5% of all mobile internet traffic in Australia, and the wave was just about any day now.

But still the wave didn’t break. Why…? These are what I think the reasons were. In the early days it wasn’t because WAP was crap necessarily, it was more to do with other factors. Handsets were waiting for Moore’s Law to bring them up to speed, sure, and elements such as screen display weren’t great, but a news site or a Twitter client would have run fine. It wasn’t the god-awful rigmarole of setting your handset up for the interweb either… that was a UI and provisioning problem that could have been solved if enough people thought it mattered. It wasn’t really even about connection speeds that sometimes compared unfavourably to simply placing the handset directly on the server and waiting for the data to migrate via osmosis. For me it came down to two things: the blizzard of handset types and the fear of the cost of surfing. If every manufacturer and even every OS level has a different way to get onto the web, that’s quite a burden of education you’re imposing on your customers. But the root I think was the pricing of data. If it costs a dollar a kilobyte then the only defence against bill shock is to surf as little as possible. Why bother reworking the UI or building compelling content if no-one has a deep enough wallet to use it? I’d contend that the iPhone didn’t kickstart the mobile web so much as 3G plan prices kickstarted the iPhone. Breaking out of the costly data plans has also killed off the walled garden since there’s now little reason to stay in-garden to receive the luxury of unmetered downloads. We’re finally free of the telco standover man, and as the market grows a lot of the other problems with mobile are finally worth getting solved.

So the wave’s finally here… but what does looking back tell us looking forward? Here are my key learnings:

• Data still has to get cheaper; telling customers to stop using their mobiles through captive-market pricing is a massive brake on market growth. This has taken a decade to learn and it’s not over yet. As an example, international roaming data rates would make Al Capone blush and are killing a mobile heartland – you don’t take your desktop computer on a tour of Italy but you do take your phone. Who would surf the Lonely Planet guide at GPRS prices though?

• Handset proliferation will increase. You’d have to be a pretty hardcore Apple fan to argue that guys like Android aren’t going to dominate our future given the stats coming out of the US. Remember Nokia’s position in the 90’s? They brought mobile telecoms out of the dark ages, and pretty much transformed the planet in doing so. But they also gave a new breed of manufacturers a powerful incentive to get smart and try and cut in on the pie.  There’s nothing to indicate that Apple this time are any different; the mobile landscape will be heterogeneous. One size does not fit all.

• Related to the point above, apps may be the reality now but they are not the future. For a mobile product provider the number of app versions to test can be worked out via this formula: platforms * OS versions * countries = variants to test. When the iPhone launched, this was 1*1*1=1; for Westfield this would currently be 3*3*4=36 combinations (and this calculation doesn’t take into account Android platform differences between manufacturers). Add to this the logistics of deploying 36 app variants and keeping them up to date on customers’ handsets and you soon come to the conclusion that there are some combos you’re willing to skip testing for. At this point you have implicitly decided you are okay to move away from seeking the perfection of the app experience; so you may as well embrace the equivalent imperfections of a web experience but with a much smaller testing and deployment burden.
• Mobile is core. There is a shrinking distinction between web and mobile for consumers: checking a Twitter feed on the phone, browsing the news on a laptop, watching YouTube on the iPad, all these activities and devices are becoming increasingly interchangeable.

The last point is important for those of us who are seeking to provide a service online.  It’s time to stop talking about our mobile site and our website as if they are two separate things.  They are now two aspects of the same product and consumers increasingly expect a level of service on the mobile pages comparable to their experience on your website – the days of outsourcing the production of a 5-page mobile site to a 3rd party specialist dev shop are over.  It’s time to tool up and talk about mobile as a first-level part of your product offering.

Firstly, let’s get this out of the way: I like Waterfall. It’s definitely got a place in the toolbox. As a methodology, it’s been around for a very long time… I’m pretty sure they didn’t use Agile to build the Pyramids. It’s a great methodology if you have to make complex but atomic changes, such as cutting over a road traffic management system, or doing a PR launch: there are many situations where iteration isn’t possible, or the iteration length would be months (at which point, you don’t have an Agile process, you have a series of waterfall projects running in consecutive phases, regardless of what you may think is actually happening…). But here’s the thing: Waterfall predates the creation of the Corporation by thousands of years, so has become part of the invisible fabric permeating the enterprise, like Payroll services or Legal departments. It’s the default position and like any tool it shapes the wielder’s perspective: if you have a hammer, all problems tend to look like nails. You can tend to deliver a project or create a product using Waterfall with a known and understood success rate, which makes it a favoured option for the enterprise since there’s a lot of emphasis on reliability of outcome.

There’s a higher level question though. If you didn’t have Waterfall, would you find new ways to do things? Would they actually be more appropriate? This leads to the question of whether doing things differently increases the success rate enough to offset the risk and cost of retooling the enterprise to a new methodology.

Hardcore Agilistas tend to insist that if everything’s done to the letter of the manifesto, impediments will be overcome, management structures will melt away, everyone will be empowered and the people will get what they want. To me (and leaving aside the echoes of Karl Marx) this misses the point of Agile, in that it is a process that introspects itself. This is its superpower: where Waterfall has shaped the entire enterprise over time to look like it, Agile is capable of adapting to its environment as well as shaping it. You don’t have to be a Prussian Field Marshall to understand that no battle plan survives contact with the enemy; a methodology that is able to adapt will survive.

Westfield is a large organisation, and specialises in digging very big holes in the ground, pouring in concrete and earning rental income for the next two decades.  Fair to say that it does this very well in that they’re able to repeat the process successfully and have most of the processes down to a fine art. It’s Waterfall’s home ground.  But how does a property management company used to dropping a billion dollars to rebuild a sizeable chunk of the East End of London address a clear and present danger to their core business with the seismic changes due to the rise of retail ecommerce?

Let’s look at the ingredients of the problem: lack of deep knowledge of the domain (online retail), needing to create an ecommerce offering in a rapidly-evolving landscape, while building a new business from the ground up.  By the time that a traditional Waterfall project had gathered the business requirements, they would already be out of date.  This was why we went for an Agile approach, focussing intently on building and releasing in tight iterations, always delivering something new into the production environment, looking at our direction versus the landscape and modifying accordingly. It’s fair to say that the feature list we started with was not the one we ended up delivering; by deploying something to the market we were able to get an early insight into what was and what wasn’t working and act accordingly.  Another important factor is accommodating a rapid growth in team size and team functions; roles were defined and redefined several times as operational pain-points became clearer.  Often it seemed like whack-a-mole with new problems arising in one area as previous problems were resolved in another.  Here, choosing a particular Agile methodology such as Scrum is not as important as its underlying principle of continuous introspection and improvement through use of the end-of-sprint retrospective to air pain-points and a commitment to re-engineer processes to remove them.

So for us, was the cost of retooling for Agile outweighed by the benefits?  It’s certainly reshaped (and is still reshaping) the way we do things. In retrospect, there really wasn’t another option.