Application Platform Strategies Blog

Java Modularity - Time to Set Sail

Blogger: Kirk Knoernschild

About 18 months ago, I published OSGi in the Enterprise, with help from my esteemed colleagues, of course. In the time that's passed, a lot has changed. Alex Blewitt has already provided an excellent overview of many of the interesting events that have occurred over the past several months in his InfoQ articles, Bundle.update, the Current State of OSGi, Bundle.update, OSGi in Java EE, JSR 291 Marked Inactive, and Bundle.update: The Year of Java Modularity.

In addition to these great overviews explaining where OSGi stands today, Eclipse has announced two very notable projects worth following: Project Gemini, which aims to provide implementations of the standards developed by the OSGi Enterprise Expert Group, and Project Virgo, which was made possible through the donation of SpringSource dm Server to the Eclipse Foundation. IBM and JBoss are also making great strides in exposing the virtues of OSGi to their enterprise customers, while Paremus continues to innovate with cool products like Nimble. And let's not forget about Apache Aries and iPojo and Sun's Glassfish. Of course, the notable list goes on.

The Early Adopters

With all the progress we've made over the past 18 months, we're left with two nagging questions:

• Is OSGi ready for the enterprise?
• Is the enterprise ready for OSGi?

Of course, the answer to each is, "It depends!" Let me elaborate through reference to the technology adoption lifecycle. When OSGi in the Enterprise was published roughly 18 months ago, I have no doubt that we were in the Innovators phase. The hype surrounding OSGi was magnificent and vendors were working feverishly to leverage OSGi. Yet, without subjecting themselves to a significant burden due to dearth of tooling, lack of third party modules, and scarcity of available resources, leveraging OSGi in the enterprise was simply not feasible. This is why, 18 months ago, OSGi was not a viable enterprise technology. Yet, it was easy to recognize the potential, as well as the momentum, which is why I made the following recommendations:

• Include OSGi in the Infrastructure Roadmap
• Monitor Product Evolution and Market Penetration
• Design for Modularity Now

Today, I have no doubt that we've crossed over to the Early Adopters phase. I am starting to hear many more success stories from developers who are leveraging OSGi. This is because platforms expose the capabilities of OSGi to developers, tools have emerged that ease development, and many third party frameworks have been "osgi-ified". Each of these advancements makes it easier and more viable to develop applications that leverage OSGi. More than ever, we must heed the recommendations above. OSGi continues to make its mark. If you're an early adopter, OSGi is ready and waiting for you.

Crossing the Chasm

Yet, the most significant hurdle looms - crossing the chasm. There are a few different routes for OSGi. Let's examine a few scenarios. First, some background.

JSR 294 is the standard that will define the module system for Java SE 7. Project Jigsaw leverages JSR 294 as part of the OpenJDK project to create a simple module system for JDK 7. Officially, Jigsaw is not part of Java SE 7. However, Jigsaw is going to be the reference implementation (RI) for Java SE 7, implying Jigsaw is a logical choice to become the JSR 294 RI.  The OSGi Alliance does have members on JSR 294 (ie. Peter Kriens) to help ensure compatibility of JSR 294 with OSGi.  If you're interested, here's a perspective from the middle of last year following JavaOne. Now, at this point, a few things potentially happen that affect the future of OSGi and Jigsaw.

• Recently, JSR 294 291 was marked as inactive. Alex Buckley commented that this was only because of a process flaw in the JCP. A JSR is marked inactive if it hasn’t produced an early draft for 18 months.  He assured the community that JSR 294 is alive and well. Yet, possibly it’s a sign that Oracle isn’t behind the project, especially since they have leveraged OSGi in some of their products, though they have been quiet about OSGi since they announced their acquisition plans. With the death of JSR 294 goes the death of Jigsaw. So, Jigsaw dies, and Oracle throws their weight behind OSGi. OSGi becomes the runtime module system on the Java platform.
  
• JSR 294 inactive status truly is a snafu in the system. The JCP delivers JSR 294, for which Jigsaw is the RI. Momentum builds around Jigsaw and it becomes part of Java SE 7 implementations. OSGi is relegated to niche markets (Eclipse RCP, home appliances).
  
• The second scenario plays out, yet there are flaws in the specification and Jigsaw’s implementation that prevent its adoption. Neil highlights a few of these. In the end, OSGi is proven to address these shortcomings, and OSGi emerges as the de facto standard module system.
• JSR 294 does in fact ensure compatibility between Jigsaw and OSGi. They can be used interchangeably. Everyone goes home happy. Doubtful!

Try playing out your own scenario. Either way, all paths lead to a module system on the Java platform. And certainly OSGi has a significant headstart.

But What If...

Wait a minute, though. Imagine that Java 7 never gets launched as a JSR, is instead a proprietary implementation from Oracle/Sun who tries to monetize Java 7. The other vendors decide to boycott Java 7 and the world sticks with Java 6. Let fragmentation of the Java platform commence. A dire scenario indeed.

Yet a plausible scenario, as well. There is no JSR for Java SE 7, meaning there is no specification nor technology compatibility kit (TCK). A recent article in SD Times points out that there have been no new JSR submissions since Oracle announced their decision to buy Sun. Likewise, no JSRs have entered early draft review. So what exactly are Oracle's plans? Well, it's possible we're about to find out. On January 27th, Oracle is hosting a webcast where they intend to announce their plans. And as Mitch points out, Ellison certainly doesn't want any distractions hanging around as the big race approaches.

So it may come down to this. The future of OSGi, Jigsaw, and modularity on the Java platform comes down to the direction that Oracle intends to take Java technology going forward. Of course, their decision will have a tremendous impact on the entire Java ecosystem. We should have direction sometime soon. Time to set sail!

The Fit Client

Blogger: Kirk Knoernschild

Soon, we'll be publishing our newest research on presentation technologies, where we'll be spending a good share of time talking about the fit client. The fit client offers a blend of benefits from previous and current generation presentation technologies, including tighter integration with the underlying operating system, ability to execute when not connected, and a very rich user experience (UXP). These capabilities are illustrated in the diagram below.

Obvious examples of fit client technologies include solutions from entrenched vendors such as Adobe (Air), Microsoft (Silverlight), and Sun (Java FX). But looming on the horizon is a major update to the HTML specification that could pose a serious threat to vendor technologies.

Based on open web technologies, HTML 5 will offer many capabilities that rival those of the commercial fit client. Since the release of the HTML 5 draft specification in 2008, many browser vendors are already supporting some of its capabilities. However, full support won't be complete for years. While this offers proprietary technologies such as Air and Silverlight a short-term advantage in terms of capabilities, they still lack the deep market penetration and ubiquity of the browser.

Without question, the presentation technology space is a volatile hotbed of innovation. As the products and technologies continue to evolve, applications will undergo amazing transformations. Vendors are forging ahead in providing products with unprecedented capabilities, while evolving open web technologies aim to supplant the vendor solutions before they ever gain widespread adoption. The race is on to provide multi-channel, branded, and context aware experiences filled with rich media, and each are on a collision course that is going to define the future of presentation technologies. It's going to be a wild and exciting ride.

Burton Group Institute - Improving the Development Process

Blogger: Kirk Knoernschild

At this year's Catalyst conference, I led a workshop on Improving the Software Development Process. The workshop was recorded, and is now available through the Burton Group Institute. I've captured some excerpts from the three and a half hour session, which can be viewed in the video below. We covered a broad range of topics in the workshop, just a few of which I highlight below:

• An optimal team structure for software development
• The impact that organizational structure and dynamics have on a software development team.
• Debunking myths and talking about benefits of agile and iterative development. We also talked about the waterfall mistake, and briefly discussed some lean principles such as building quality in, eliminating waste, and delivering fast.
• The SDLC infrastructure, which included a demo illustrating how we can automate various tasks by leveraging this infrastructure.
• Requirements management, the difficulty in stabilizing requirements early, and emergent requirements.
• Continuous integration, tightening the feedback loop, project visibility and transparency, and the value in having a system that always works.

Enjoy the excerpt!

News Flash: Insourcing Fosters Relationship

Blogger: Kirk Knoernschild

A recent article on TechTarget claims that IT insourcing is on the rise. In other words, organizations are beginning to bring their recently outsourced IT functions back in-house. The article cites a few motivating forces for this change. Organizations have more control, can respond more quickly to business change, and are leery of outsourcing going forward due to unpleasant past experiences. The articles cites on of the reasons for outsourcing - it was cheaper - may not actually be true. While outsourcing may be cheaper short-term, the long-term cost may be more. Not an epiphany, for sure.

But what really caught my eye was the following quote from the article:

There's a sense of intimacy with the business users that's hard to achieve with outsourcing.

That's a rather profound statement. It took outsourcing for us to realize that collaboration with business users is a critical success factor on an IT initiative? Wow.

IT Labor

Since we're on the topic of outsourcing.... Another popular reason why organizations outsource is due to the IT labor shortage. Many claim there is a lack of qualified IT candidates. This reason is also the foundation of the H-1B program. Not so fast. I've never actually believed this was entirely true. My good friend Randy once said, "There's also a shortage of $2.00 per six pack micro-brews." If you're interested in IT labor, this Lou Dobb's video might interest you.

Conspiracy Theory

I take my car to a certified GM dealership, even though it's a bit more expensive than going to the hometown mechanic. Why? Because I know they won't use secondhand or refurbished parts. I also know they have the specialized equipment and expertise to work on my model vehicle. But if the GM dealership charged a 50% premium for the same level of service I could get at my local auto shop, I'd have no motivation to take my vehicle to the dealership. Now, apply this analogy to corporate IT.

If corporate IT charges a 50% premium over an outsourcing provider, but doesn't provide any strategic or competitive advantage to the organization, what's the motivation to use corporate IT? Instead, it's sound fiscal strategy to outsource. In other words, why should I pay $100 an hour for crap when I can get the same crap for $75?

For now, let's ignore the crap (that's a topic of discussion for another day), and instead focus on cost. $75 an hour may seem cheaper than $100 an hour, but only when you fail to factor in the number of hours it takes to complete a project. And the challenges inherent to distributed development, contract negotation with outsourcing providers, and more will impact delivery, which will increase time, which will drive up cost. So now, we're starting to realize that outsourcing isn't necessarily cheaper. Don't be surprised to see aggressive legislation on the table soon asking for an increase to the H-1B cap. Wouldn't it be ideal to have the best of both worlds? Cheap labor that's co-located! Really? Is it possible there's a causal relation between IT failure, motivation behind outsourcing, and the H-1B program?

Disclaimer: I'm not claiming that everything we deliver is crap. But there is no shortage of evidence illustrating some of the challenges we've had in delivering IT projects.

Disclaimer: I don't claim that we should abolish the H-1B program. However, there is evidence that H-1B visa holders are paid between $12,500 and $20,000 less than citizens. That's not the purpose of the program.

Increasing Your Agility

Blogger: Kirk Knoernschild

Some remark that agile development processes have crossed they chasm - agile is mainstream and adoption is widespread. I'm not convinced that's true. Instead, I find that waterfall development is still alive and prospering. Unfortunately, the teams leveraging waterfall methods are not prospering. In fact, I'd argue that less than half of all software development teams leverage agile practices. I have no hard data that supports my claim, only anecdotal evidence suggesting it's valid. Regardless, it's disconcerting. Why?

Given today's economic woes, the impetus is on IT to show and prove their value. Continuing with status quo will not suffice. We're being asked to do more with less - fewer resources, shorter timeframes, reduced budgets, and more. While organizations look for ways to reduce costs, IT has been given a window of opportunity. It's critical that we take advantage of it.

So what can should we do? Look at what we can do to improve the quality of the software we deliver. Bake quality into the product from the beginning instead of attempting to validate quality at the end of the lifecycle. Look at what we can do to improve the efficiency of our team. Eliminate practices that aren't directly contributing to customer value. Look at what we can do to improve the transparency of the development effort. Engage the customer and share the good and the bad surrounding the current state of the system. These are just a few of the things we need to be doing now, and each have a direct correlation to reducing cost.

Agile processes and practices can help in these, and many other areas, but only if applied correctly and pragmatically. Adopting Scrum, XP, or any other agile method won't guarantee success. It won't guarantee agility. Agility is not defined by process adoption, but by the ability of the team to deliver valued software in a timely fashion. At this years Catalyst conference, I'll be leading a workshop titled Improving the Software Development Process. In this workshop, I'll be talking about ways to increase agility and improve the development process. Surprisingly, little of the discussion will focus on agile processes. Instead, the discussion focuses on techniques to improve software quality, maximize team efficiency, increase transparency, and more. I argue this is the essence of increased agility; not process adoption. We'll discuss actionable items that teams can use immediately.

If you're attending Catalyst, and are interested in improving your software development process, I encourage you to attend the workshop. If you do not plan to attend Catalyst, but are a Burton Group client, schedule a dialog now and we can discuss the issues, how they interrelate, and what you can be doing now to improve your software development efforts.

Scaling Agile

Blogger: Kirk Knoernschild

There seems to be a common myth that agile practices work well for small teams, but don't scale for large enterprise development efforts. Experience has shown me the opposite.  Certainly agile methods are a natural fit for smaller teams, who don't have the unique communication challenges and project management difficulties present with larger teams. In fact, it's these unique challenges that makes agile practices such a perfect fit for large teams. It's time to debunk the myth that agile practices don't scale.

I've written two pieces recently about using agile practices on big teams. The first talks about a macro process for making the practices work. The second clarifies a few items that may raise questions and cause for concern. I've also created a couple Burton Group resources that can help a team increase their agility. The first is a paper that talks about Rightsizing the Software Development Process, and the second explores Continuous Integration and it's many benefits. There is also other relevant and valuable information in our research library, including Agile Software Development: Not For Lightweights and SDLC Infrastructure: Supporting the Development Process.

This material underscores some fundamental elements that explain why agile practices are so important for any team. Strive to always have a functional system. Leverage the functional system to increase collaboration and garner important feedback from the customer. The more feedback you can get earlier in the development lifecycle, the better you'll know whether you're on track to deliver a system the customer values. As customer feedback and collaboration increase, so to does project transparency. Everyone - the development team and the customer team - understands the current state of the development effort. In fact, as the system evolves, you might even find that you no longer have a development team and a customer team. Instead, you've got "a team".

Metrics and measurements

Burton Group recently published two popular reports on IT metrics, measuring IT’s business value, and justifying IT's value.  The new reports target Data Management and the Software Development Life cycle perspectives. 

I found Kirk Knoernschild's coverage of 'metriculation' a refreshing and healthy departure from the customary 'metrics bashing' or 'metric as a panacea' perspective espoused by others. A few key quotes:

"Many IT organizations are leery of using metrics. In some cases, such caution may be warranted, especially when metrics have been used deconstructively. Metriculation refers to the improper use of metrics. More succinctly, metriculation is when metrics lie."

"Although software teams may already capture specific metrics, it’s imperative that IT evaluates how well those metrics align with business objectives. It must be careful to avoid “metriculation,” where metrics convey misinformation."

"Internal IT metrics are a valuable tool for helping the IT organization improve its internal processes, but they aren’t much help when it comes to justifying investments. For that, the IT group must define metrics that measure its business value (i.e., the value that IT contributes to the business)."

For more information on metriculation and measuring value, read Kirk's blog entry and research report.

APS on Twitter

Blogger: Kirk Knoernschild

Some of the APS Analysts have been experiencing the Twitter phenomenon lately. I'll admit, at first I was a tad skeptical. But I've quickly grown fond of the service as a way to monitor all that's happening in the tech world, in real-time, right from the comfort of my laptop. It's also a great way to send broadcast messages to a large group of people. Here's are the tweeting APS Analysts and their Twitter usernames.

Follow along, and happy tweeting.

Anne Thomas Manes - atmanes
Chris Haddad - cobiacomm
Richard Watson - richwatson
Kirk Knoernschild - pragkirk

Metriculation - Metrics that Lie

Blogger: Kirk Knoernschild

Metrics can be a valuable asset to software development teams in helping to improve their efficiences and measure their effectiveness. Unfortunately, metrics don't always provide the intended value. Sometimes, metrics lie.

Metriculation is a term I use to describe how metrics are made to lie. It's derived from statisticulation, a term introduced by Darrell Huff in his book, "How to Lie with Statistics", first published way back in 1954. There are many forms of metriculation, and it's a significant factor that affects the successful use of software development metrics.

Metriculation is not the same as using metrics ineffectively, or measuring the wrong thing. For instance, using LOC (lines of code) was once thought an effective way to assess programmer productivity. LOC has nothing to do with productivity. Code generation wizards can be used to create a component with thousands of lines of code, but it might take days to correct a nasty bug that requires changing only a single LOC. LOC does have it's value, however. Using LOC to help calculate the percentage of code under test, or to help with software estimation, are examples. Using LOC to gauge developer productivity is ineffective, but it's not metriculation.

Instead, metriculation involves using metrics to mislead. Downright deceive. Lie! For instance, in an effort to undermine the value of unit testing, a developer may claim that 75% of the system defects are found in code with unit tests. This form of metriculation, known as semi-attachment, is done by finding a way to prove what you want to prove by drawing false conclusions. Based on only this information, one might be drawn to the false conclusion that unit testing does not contribute to higher quality software. Unfortunately, what the developer may not have shared is that 95% of the code is under test, meaning that the remaining 25% of defects reside in 5% of the code. Given this additional information, we could have arrived at an entirely different conclusion - that unit testing does contribute to higher code quality. In this example, the metriculation was intentional, but that's not always the case. Often times, it's due to oversight. Regardless, metriculation is dangerous.

In addition to semi-attachment, there are many other forms of metriculation, all with numerous real world examples of how they've been used to push an agenda. These include selective use, biased samples, well-chosen average, fudging the numbers, overgeneralization, faulty assumptions, whiz bang graphics, and creative charts. Upcoming Burton Group research will discuss in more detail metriculation, and how we can deal with. As that material matures, I'll continue to share interesting snippets.

Palm's Nova

Blogger: Kirk Knoernschild

I purchased my first Palm in the mid-90’s. I upgraded to a Sony Clie (Palm OS) in the late 90’s, and bought a Treo 700p in 2007. Over that span of about 13 years, the only noticeable change to their OS was that it went from black & white monochrome to decent color. Very few new productivity features were added, and the existing applications  were never improved. Some 3rd party apps emerged that worked...kind of.

I wrote a review of the Palm Treo after I had used it for a while. Since that time, I've upgraded to an iPhone (note: "I upgraded"). Since that review and my subsequent iPhone purchase, I’ve changed my mind on few items regarding Palm, their OS, and applications. Regardless, Palm squandered a golden opportunity. It's only real competitor in the mid-90's was an Apple Newton. Remeber those? Then Apple was out of the game for about a decade before introducing the iPhone in 2007. During those years, Palm made little news. But this isn't about Palm yesterday, it's about Palm tomorrow. Where they are going, and what they are doing.

Palm claims to have something big up their sleeves, and they'll unveil it to the world at CES in early 2009. Is it really possible that after letting RIM (BlackBerry) and Apple (iPhone) steal their market, they can catch up? It's difficult to believe they can pull it off, but if you dig a little deeper, a hint of possibility does exist. Rumor has it they've been working on their new Nova OS for over four years. They just received an additional $100 milllion in funding. They've recently launched the Palm Software Store. And Palm's leadership certainly has the experience to make it happen. We'll see in early 2009.