Who are you working for anyway?

One of the main reasons that I've felt a little uncomfortable with working for big corporates has been the increasingly onerous Terms and Conditions of employment that I've had to endure. I've always felt that out of the hours of 9.00 to 5.30 my time was my own, I was free to express myself in any way that I wanted.
Now, the only freedom I really wanted was that of being able to go home and work on my own projects; the things that burnt inside of me, the things that needed to be expressed. Increasingly nowadays, the big corporates are assuming that you work for them 24 hours a day and anything you do is either (i) Theirs entirely, IP and all and (ii) Theirs to censor. In other words they believe that they own you lock stock and barrel.
It doesn't really work that way, we'll all expressing ourselves in Blogs, contributing to Open Source, and generally interacting with other people and organisations on a commercial and semi-commercial basis all the time out there on the Internet.
I've got to be able to say that I'm doing it for someone else, or I'm simply doing it for myself.
To be fair, there are many big corporates that are so frightened of their public image, or worried about compliance, that they have to assume that whenever their employees communicate, they communicate on behalf of that corporate. And it seems it's not enough that we use private email addresses.
We are fast moving into an age where we will be working for many more companies simultaneously, and sharing our time, interests, tools and materials. Many companies nowadays don't have a permanent workforce at all, they have outsourced, sub-contracted or employed the big consultancies on projects. How do they ensure seperation of concerns and Intellectual Property? And how do we maintain the right identity?

Bringing it all together

How many databases does it take to change a lightbulb? and Databases and ear discomfort and Bringing it all together
And you may remember that, along with some helpful comments, we arrived at the idea that it was closely related to the number of business processes that a company runs......
Well, we never really closed on it, never got to the position where we could mathematically say whether we had the right number in a particular company or the wrong number.
Why do I want to know, why do I get so hung up on this kind of stuff? Well, I guess I'm always trying to get at our inefficiencies, our lack of quality, and would like to be able to say that we have too many systems or too many technologies. You have to be able to say what the optimum number is in the first place.
It's all very well to go into a company as a consultant and say "Gee, 3467 databases does seem quite a lot, I think you may have a problem here...". I'd like to be able to authoritively say "Based in your current business Model, and the number of High-Level Business processes that you carry out, your optimum number should be 42, so you are over-deployed by a factor of 82.55. Based on current metrics and industry practice, you could achieve over-deployment factors of 13.67, saving you 2893 databases and $10M every year in Support, Maintenance and Licensing."
Ha!
But of course, we haven't got this. We can only stand there wringing our hands, talking to the floor and saying "Weeeellll, I kinda think we've got a problem" and "I think we could kinda save some money here" and "I kinda think that this is contributing to our high levels of operational risk"
What is the optimum number of databases for any company, in any industry?
Well, I'm going to stick my neck out here and say that it's more than 10 but less than 20, but I do have a high-level list of suggested business processes to back this up...
What's your take?

Reasons, reasons, reasons

In the last Blog entitled "The number is 10" I tried to re-ignite earlier blogs on optimising the database space, and came to the conclusion that a company should have 10, possibly 20, databases. I had some helpful comments as well (thank you Vincent). I'd like to continue this theme.....
The reasons why we exceed the optimum number of databases are many and varied:
1. We use Vendor products that don't provide perfect coverage of the data space for the business function they cover. This coverage can be more or less of that optimum data space, but either way, additional, external, data coverage is required to facilitate usage of that Vendor product. There is further rationale available around this....
2. We use many Vendor Products and Systems to cover the same business function. Enterprise Architects spend a large amount of their time reviewing IT Projects promoting re-use and rationalising the Application space in the sometimes vain attempt to stop this kind of proliferation.
3. The Systems using these databases may be planned to be replaced at some time (presumably and hopefully moving towards a more perfect model). If you have hundreds or even thousands of business applications, this could be quite a waiting queue.
4. Some of the databases are merely Integration points for data concentration and transformation. We do this, a lot, particularly to concentrate data for reporting, but mainly for "Impedance Matching" between Systems.
5. Reporting databases proliferate freely. The data is never in the format you want it. This is more about the feeling that Data WareHousing is difficult and costly and the tools difficult to use. Many, many databases are used in Back Offices everywhere to massage data for Reporting in differing ways and to facilitate investigations.
6. Personal Productivity. During the Year 2000 efforts, I worked at a company where we inventorised all Access Databases on file servers and, for a department of 120 people, found 8500 databases. Further investigation showed that some of them were tracking lottery numbers, but many of them were merely copies of existing data in a more manageable and accessible form produced to make the owner more efficient. Of course, this introduces a bigger problem than technology proliferation, it introduces Operational Risk. Much more could be said here....
There may be other reasons here that I'm sure you can help me with....

Pass it on.....

I'd like to keep to this theme about database proliferation offering you another little story from my travels through the magical world of Megabank.
I was taking a look in an emerging business area; we were creating a new investment product for a particular market. As is usual in investment banking, sometimes with these more esoteric products you can price them and produce them using manual procedures when there are very low numbers involved. This group had started to handle the product semi-automatically using Microsoft Access and Excel to handle tabulating the deals and to do some of the processing, but things were moving too quickly, business was booming.
One of the days that I visited the area I was most amused (secretly) to see 50 people, all with spreadsheets and databases of various forms open, shouting across the floor at one another saying "Who's got the Order Sheet open?, I can't get it" and "OK, finished James, it's all yours". The entire group were using Excel, Access and shouting to process the business.
One of the reasons I was there was to look at audit problems, and particularly the very high number of fails and exceptions in the process. Investigation showed that it was plainly the handling of the Excel and Access files that was the problem; they were being updated and copied by many people at the same time, with multiple versions in existence.
And, on pointing out the obvious problems to them, some of these people even said that their data was just harmless copies! Even though they went on to make business decisions and carry out other processes based on that data.
Bad data governance, no identifiable safe update process, and no sense of "gold copy". Nightmare. I was really surprised that these people hadn't understood that data is the lifeblood of a company, and that it needs treating with care and reverance.
It was crying out for a system to be developed, and a single source of truth that everyone could work to, a single, gold copy database managed inside a single defined process.
Over the following months as a tactical measure, we helped them build the gold copy database based on Microsoft's SQL Server, which worked very well, reducing the number of fails and exceptions dramatically. Until one evening when the office cleaner pulled out the plug on the server................but that's another story!

At last a number.........

I'd like to come to a close on this stream of consciousness around my perfect number of databases by summarising where I think we've got to.
By databases, I mean actual Operational Data Stores that contain the data that support the business process. I don't mean convenience copies (although I don't believe they should exist at all), and I dont mean Integration databases (which are largely transitory), and I don't mean databases that exist in Vendor Products that don't fit the whole of the business function for which they supposedly cover (this is probably a big cop-out).
The list of Systems supporting the major Business Functions that use a database looks like this:
1. CRM - Supporting the Sales Force with Prospect and Current Customer details for the purposes of new and repeat sales.
2. CRM - Service - The Helpdesk for existing Customer service.
3. Sales Order System (Possibly by major Product Lines - I believe there could be multiples)
4. Manufacturing System, Stores (Posisbly by major Product Lines)
5. Despatch/Delivery System (Possibly by major Product Lines)
6. Assets - Asset register
7. Finance (Receivables, Payables, Ledger)
8. Data Warehouse (One please)
9. Reporting - Customer, Management, Regulatory (Data Marts connected to Warehouse)
10. Reference - The single initiation point for Customer (Gold Copy)
11. Reference - External External market "temperature" prices, rates, indicators. (Gold Copy for company)
12. Reference - Other (Country, Currency, Legal Entity, Reporting Entity, Management Entity) (Gold copy)
This makes 6 Common Business Systems, 3 dependent on the number of Major Product Lines, and 3 Reference
Perfect number of databases = 9 + (3 * Major Product Lines). It doesn't look as impressive as E=Mc2 or anything like that, but simple is good.
We don't care that nobody (of any substantial size) will ever achieve that number since it is only a reference point, indeed I expect that the vast majority of companies will always exceed that number by a significant amount. What matters is that we have a reference point from which to test the health of companies Systems and their IT.
So, Megabank had around 11 major Business Products making their "perfect" Database number 42, which we all know is the answer to life, the universe and everything. They actually had around 2500 Applications each with a database, so that gives Megabank a database score of 2500/45 = 55.5. We're not going to get rid of 2444.5 databases, no way, - thats not the point. The point is that SuperBank (in the same industry) for the same number of major Business Products had 1500, making us look very inefficient since we are carrying enormous costs maintaining the extra 1000 databases and making ourselves less competetive.
And at this point I'm going to let this subject rest, since we'll all probably getting bored with it, but I would like to explore at some point how to address some of the problems of overdeployment, regardless of how it is measured!

Another Paradigm-shift?

The ever-increasing clock speeds of Processors seem to be coming to an end now; it seems we have come to the limit of the capabilities of our current technology. We had clock speeds of 2MHZ in the late 70s and early 80s, and have increased more than a thousand-fold to the GHZ clock speeds of today. But it does seem like we have reached a plateau at 4GHZ that we're having problems getting past.

The answer seems to be that instead of making processors go faster, we're actually just using more processors to solve our processing problems. So, we've moved from single processors to multiple processors (which really just increased the number of streams of work we could process) to the present mulitple core idea (which is just like multiple processor idea except we put the processors on one chip).

But are we really taking best advantage of these multiple-core processors? At some point, for personal computing at least, we don't have enough streams of work to take advantage, and the applications we use are constructed to support single streams of processing.

And I don't think the answer is just multi-threading programs like we've been doing for the last few years. Multi-threading is all about intimate knowledge of the work that needs processing and the mechanisms to allow it to happen safely.

I feel like we need here is another one of those paradigm-shifts of the kind that we engineered during the 90s with Object-Orientation (at least when OO came into the mainstream). We need to start thinking about little pieces of work, units of work applied to Objects perhaps. Something around the idea that all methods on Objects are naturally threads and then some mechanisms to synchronise the Objects readiness to interact?

While we are there, perhaps we can start viewing Grid as just an extention of Multi-Core, in that processors are either near or far away, the only difference is latency?

Anyway, in terms of my earlier blogs on What's the next language, I think that perhaps this might be a good direction..... I think we should name the next language "Q".....there! we're now well under way....

Are you leading or following?

A reader commented on an old blog of mine and the gist of his comment was around questioning why we change our technologies so much when it's patently clear that the old technologies provided good value.
When these new technologies are presented to us, they come packaged with all sort of promise about productivity, flexibility and cost-effectiveness but, I wonder, do we ever look back and assess whether they actually met their objectives? The short answer, of course, is no. We're all too bust looking ahead to the next technology step, and no-one looks back and analyses where we, perhaps, went wrong with the last one.
The reasons for this are probably numerous, but sometimes the most simple reason suffices here. - we're just too busy keeping up!
I've sensed this for a number of years, particularly around the development fraternity; the march of technology means that we're trying to stay current, keep up the pace, make sure our CVs are spot on the market. There's no money in philosophising, no demand for it, only a demand for technology solutions.
So, who's driving this? It's obvious we're all just too busy keeping up here, so it's not us.
It's the tools market, the products market. They need to stay in business, they need to keep you buying; and to do that they need to make the product attractive with new features. They invent the requirement for new technology; it doesn't exist without them.
The Internet is to blame too. It's all too easy to communicate now, it's all too easy to market ideas, it's all too easy to create the impression that there's a lot of activity around something which is essentially niche. And you fall for it hook, line and sinker! And your reaction propagates and re-enforces the message that there is a need, that this need is being met with the new Acme-Duffer widget set, and you need to get reading about it, training about it and using it!
Self-fulfilling prophesies.
When was the last time you lead the market for new technologies? Or are you just following?