Architectural Qualities – Integrity Part 3

And onwards through my Architectural Qualities. Today I'd like to introduce Reconciliation. Well, how do you know what went in stayed in, and what was in came out?
Integrity: Reconciliation - That the passing of data between Components can be reconciled as transactionally correct.
Well, I guess that you didn't expect it to be put that way, but that's the way I like to view it. I like to think about everything in Architecture as qualities of Components and interactions between Components. This isn't any different since what's happening is that data is passed from one Component to another, and we'd like to make sure that the receiving Component has got it.
We don't actually have to do reconciliation in this way; often in a large system we only reconcile from the front Component right to the back Component and don't bother with reconciling the Components in between.
Reconciliation is most often done by an automatic process that checks for differences in counts and contents between the different Component data sources. The actual fixing is done by one or more staff in Back Office Operations, they spend many a happy hour figuring out what the correct data should be and correcting in the appropriate Component. I've seen Operations departments deal with hundreds of reconciliation errors daily and employ tens of people dealing with them, a very expensive process.
So why do we need to do this, aren't our systems perfect? Don't we use guaranteed delivery messaging; don't we use non-stop state-of-the-art platforms? Well yes, of course we do, but there's always a hole, an opportunity, however small, that a piece of data will be lost, duplicated or corrupted, and heavy traffic discovers it more easily than low traffic.
You may remember that I talked in an earlier blog about an "acceptable error rate", well this is all about how you detect and fix the errors, and believe me, they are there.

Architectural Qualities – Integrity Part 2

Here's the next part in my Architectural Qualities where I'm dealing with various Integrity attributes, this one being about Transfer Integrity. You know you do it, but do you do it well?
Integrity:Data Transfer
Data Transfer Integrity - Data transfer is complete, unchanged by intermediaries and repeatable.
In this distributed world the quality of transfer is very important, my own firm has hundreds of Applications that exchange data in one form or another mainly in the form of file transfers numbering high up in the thousands, and I know there are very big firms out there that transfer ten or even a hundred times as many.
What kinds of solutions are good here?
Well, you could buy your way out of trouble by obtaining top-of-the-ladder guaranteed, restartable file transfer utlities that handle all the integrity issues for you, and you could get cute with custom programming, doing your own headers and trailers and checksums and file renames when complete and stuff like that.
Another aspect particularly important ensuring that the file came from a know sender and hasn't been interfered with, this is the area of using asymmetrical keys and encryption that achieves what is known as non-repudiation. If you can decrypt and make sense of it, then it came from the only possessor of the private key, and as a bonus has not been interfered with.
Although I've talked about files here, it's obvious that many much of this applies to synchronous real-time single data item transfers through Remote Procedure Calls, Message Oriented Middleware, or just plain http web access, but I'll leave this and the really interesting aspects on guaranteed delivery until another blog.

What is an acceptable failure rate?

Yesterday I was asked what non-functional requirements should we be asking our Businesses for in an upcoming Requirements exercise. Actually they asked me for Architectural requirements, but they amount to the same thing.
It came as a little of a surprise to be asked since I had begun to give up trying to raise awareness of the subject some time ago, but better late than never.
I considered for a moment my "usual suspects" and looked for more. These usual suspects are (i) What you want to do (ii) How fast you want it to be done (iii) How many of them you want to do (iv) What time of the day do you want to do them.
A brief pause, and I added the most contentious of them - (v) How many failures are acceptable? They haven't replied to the email yet, perhaps they haven't read down that far, or perhaps they are just sitting there open-mouthed thinking that I've gone off the deep end.
We have to accept that we don't build perfect systems, it's usually either impossible or uneconomic to design and build the perfect system. When we design our systems we often make trade-offs, opting to write custom code to handle protocols ourselves rather than buy vendor solutions that are tried and tested and proved in the field. We often believe it is cheaper than buying more licenses.
We don't buy non-stop systems generally, and we don't double-up on piece of technology as a rule, so we know we should expect problems. At some point the function being performed will fail, and it will fail because it is under-engineered by design, making a conscious decision to make a trade-off sometime back.
Ok, how do we mitigate against failure of our business functions? Generally we build in checks, we build reconciliation utilities, comparisons of what was put into the system against what came out of the back. We do this. We do it because we know we will have failures. We assign people to perform the reconciliation work and we assign people to fix the failures when they occur, and all this takes time and money.
So, when we ask the business for an "acceptable" failure rate, we're asking them what their capabilities are in their Operational Processes or we're preparing them for the inevitable staffing-up to cope.
Of course, they weren't expecting it, they thought that the new system would require less staffing, they told the CEO so, it was the basis of their declared return on investment for the project, there were to be a number of job savings and this represented the saving. The very idea of retaining some of the staff for failure processing just isn't acceptable. Don't build the system unless you know the true costs of it including the running costs, and remember that the test of a projects success is it's return on investment, and not whether it was delivered on time.
So ask them, what is the acceptable failure rate? Go on, ask them.

Architectural Qualities – Integrity Part 1

Here's the next part in my continuing foray into Architectural Qualities; this is the part that finds me wandering into the realm of Integrity and its various facets. Architectural Integrity is the application of appropriate technology and/or methods to ensure that data is secured against loss, duplication or corruption.
Integrity:Transactional
Transactions are all about the safe storage of our business data, this safeness is around four specific areas known throughout the IT industry as the ACID properties.
A is for Atomic - All data is updated together or none at all.
C is for Consistent - All data is consistent with each other e.g. Referentiality is maintained.
I is for Isolation - The result of data update is the same whenever the transaction is executed, now or e.g. two minutes later.
D is for Durable - The result of data update persists beyond the transaction.
I've had a enormous number of conversations about co-ordinating updates to two or more different data sources without a Transaction Manager. Transaction Managers sit outside of the data sources and commit or rollback as appropriate all the updates or none at all. Most of my conversations are either around inappropriate configurations or developers trying to program their way around the problem.
The developer with a little knowledge is the most dangerous, he goes about re-inventing the wheel based on a weak idea and total confidence in his ability to solve problems that have challenged computer scientists for years. Schemes are created attempting to do error recovery by reversing out the previous transaction, protocols are created using elaborate timeout schemes, and all we end up at the end is a solution that has inherent weakness and opportunity for Operational Risk.
If there is a chance that a message will be dropped or timed-out or an update uncertain in any way, that chance will turn into a real event at some time. It might not be today, it might not be next week, but it will happen. When it happens it could affect a large cash flow, it could affect a customer, and it could involve compensation and/or bad feeling and this, ultimately is a risk to your business.

So did we do well?

This one happens all the time, and we were discussing it again today. I maintain that the project can't be declared a success unless it has delivered to its requirements, and that these requirements include Architectural Qualities. If we ignore them, we're ignoring the fact that there is a direct correlation between these qualities and Operational Risk inherent in the delivered system.
All too often we find that Project Managers are driven by getting the project delivered on time and on budget, and nothing else. Well what else is there? We review the project, we find technical problems in there, and none of them are show-stoppers so the Project moves on since delivery is King.
So, here's a simple one; I've found a system that has static passwords, has no checks for the usual guidelines; it does not check for password that is the same as the logon name, it does not check whether the password has been used before, and it does not ask for mixed letters/numbers. The result is, of course, weak authentication.
So, what are the chances that some unauthorised person will break into the system? I really haven't a clue, but the important thing is that I know that the chance exists, no matter how small, or big. When they break in what damage will they do? I really haven't a clue again, perhaps they'll take money out of our accounts, perhaps they'll steal our customer list or perhaps passwords and account details, who knows?
Now the big question, how much will it cost us after a hacker has breached the security? Well, we can start at the millions in lost revenue from potential customers who think we're not safe and then there's the millions from existing customers who thought we were safe and then there's compensation to those that were affected.
It would be good to put a value on this identified lack of quality in the delivered system and it goes something along the lines of the well-known Risk Management calculations such as -
Chance of occurrence * Cost of occurrence = Risk in money terms.
All the compliance and regulatory activities that are occurring to us today can actually help here since we're being asked to show records of unauthorised access to our systems and we're being asked to show records of financial loss due to it.
So, finally, I may be able to tell the Project Manager that the lack of Architectural Quality we're discussing right now is going to cost us $5,000,000, with a likelihood of 1% so we've got $50,000 to play with here. Get the vendor to fix it.

Architectural Qualities - Confidentiality

As I have referred to before in a previous blog, I use a set of qualities to determine how good a solution is when I review them. These qualities can be described as "non-functional" requirements that a business can demand in support of its functional requirements; in other words they describe the How, the When, and How Secure, for example, rather than the What. This blog is on the subject on Confidentiality and its various facets.
Confidentiality: Authentication
The system and its data are only available to those persons that can be uniquely identified by some means. This has been one of the most challenging aspects of technology over the past few years, particularly since the internet provided ready access to the worlds systems. Many years ago it was the ability to enter the building past security guards that authenticated us when we accessed our systems by means of the green screens, plus the fact, of course, that the man-in-the-street didn't have a clue what to do when faced with such a screen!
Today, our solutions vary as we fight a never-ending battle against unauthorised access coming up with solutions around dynamic password devices, Public Key Infrastructure, and Biometrics (I once made a verbal slip and said "Rectal Scan" rather than "Retinal Scan" that created much hilarity - I guess the former gains you access to photocopiers from what I've seen of Office parties).
Confidentiality: Authorisation
Facilities that perform functions or allow changing of the systems's data are only available to those persons suitably empowered.
Once you've managed to identify the person, all that rests is specifying what they're allowed to do, bearing in mind that the person that provides the authorisation must also be authorised to do it as a double-check (Maker/Checker).
Again, as a challenge, since the User Interface is rarely the only means of access to data, we also have direct access to our databases and access to our interfaces to worry about. We need to ensure that our overall Authentication and Authorisation is consistent throughout the System, and wherever an interface is presented - none of those "functional" identifiers with imbedded passwords in our code.
Confidentiality: Usage
Usage means that although you may be identified, and although you may be authorised to view data, you're not allowed to do what you want with it, only valid business usage is implied by the authority. Although it's obvious that you shouldn't allow persons who are not authenticated and authorised to see and perhaps use the data, you, personally cannot use it in unauthorised ways e.g. use customer data to mass-market get-rich-quick schemes through your part-time business.

Sex, lies and backup tapes

Security is our big thing, in fact it's everybody's big thing and every bank in town is working on it. We're trying to safeguard our systems, we're trying to protect our customers, we're authenticating, we're authorising, and we're encrypting. So how come we backup all our valuable data and give it to the first person that knocks at our door?
With all the panic about hacking or unauthorised access to our data, we seem to have forgotten the most basic principles in protecting our data, wherever it is.
A long, long time ago I use to work on a big IBM Mainframe at a company that sold holidays, I say big here, but of course it had the power of that PDA in your pocket right now, despite the fact that it support 120 terminals.
Anyway, every night we backed-up all our reservations data onto those 9-track tapes, put them into a cardboard, and dutifully walked down the road in the dark to a rented garage down the road to store them "offsite". The garage was damp and cold and thoroughly unsuitable for storing data tapes, but they never seemed to suffer from it. It wasn't very secure either, the door was one of those up-and-over types and the key was as unsophisticated as of those old desk keys that you're probably using today.
The point was that it was still quite secure in that if anyone got into the garage, they'd be disappointed to find something as worthless as the tapes, and if they realised that the data might be interesting they didn't have anything available to read it on (IBM hadn't made anything any more personal than a Mainframe at the time). Add to that the fact that the damp had rotted the tape coupled with the time that we realised we'd been writing zeroes onto the tapes every night for a month (that's another story…), what data that was there was probably very secure indeed.
Well, here we are 20 years later, concentrating very hard on protecting our Systems and data, writing gigabytes onto CDs or Tapes without any form of encryption, trying to get our data offsite for Recovery and Safety reasons, and giving it to some guy who calls by the Data Centre every night saying that he's our truck driver. This truck driver who's got enough computer equipment and bandwidth at home to read, disseminate, distribute and re-write it and sell it to every spammer down the road.
So, where's your data right now? … are you sure?

The mechanics of Enterprise Architecture

One of the things I have to do in Enterprise Architecture, and have been called to do a number of times, is set up a process for Architecture in an organisation.

Generally, in large corporations, EA has much more to do with how you do things rather on what you do. It's not good to be good at determining Strategy, or good at J2EE Architectures, or Security; it's very good if you know how to go about doing things if you want to get along in this game. If you want to be the Chief Architect or Head of Architecture or other head honcho, you've primarily got to know how to get organised.

EA processes touch many areas in IT in the Systems Development LifeCycle, in IT Strategy setting and monitoring, in Investment Reviews, and in Audit and Compliance reviews.

And they introduce some of their own around defining standard components, standard technologies, and design reviews.....

The thing is, these kinds of processes standardise EA itself, they are the processes that define EA.

Gone are the times when "Technical" Architects stirred up trouble, got under your skin, got in the way of development and generally make a nuisance of themselves. Today we are much more process-aligned, coming in at the right time, making our points, and shrinking back into the background until review time.

Over the last few years we've been developing custom processes to support these activities and been reasonably successful. Today, though, I've noticed with such things as TOGAF 8, these processes are becoming more prescriptive, standardised and mature. It's good stuff, but you can take as much or as little as you like. It doesn't tell you about diagrams, and it doesn't tell you about technology; it tells you what to do.

So go on, read the book, become an Enterprise Architect of today; you won't learn Strategy or Technology, but you will learn how to get organised, and how to rise to the heights....

Who are you working for anyway?

One of the main reasons that I've felt a little uncomfortable with working for big corporates has been the increasingly onerous Terms and Conditions of employment that I've had to endure. I've always felt that out of the hours of 9.00 to 5.30 my time was my own, I was free to express myself in any way that I wanted.
Now, the only freedom I really wanted was that of being able to go home and work on my own projects; the things that burnt inside of me, the things that needed to be expressed. Increasingly nowadays, the big corporates are assuming that you work for them 24 hours a day and anything you do is either (i) Theirs entirely, IP and all and (ii) Theirs to censor. In other words they believe that they own you lock stock and barrel.
It doesn't really work that way, we'll all expressing ourselves in Blogs, contributing to Open Source, and generally interacting with other people and organisations on a commercial and semi-commercial basis all the time out there on the Internet.
I've got to be able to say that I'm doing it for someone else, or I'm simply doing it for myself.
To be fair, there are many big corporates that are so frightened of their public image, or worried about compliance, that they have to assume that whenever their employees communicate, they communicate on behalf of that corporate. And it seems it's not enough that we use private email addresses.
We are fast moving into an age where we will be working for many more companies simultaneously, and sharing our time, interests, tools and materials. Many companies nowadays don't have a permanent workforce at all, they have outsourced, sub-contracted or employed the big consultancies on projects. How do they ensure seperation of concerns and Intellectual Property? And how do we maintain the right identity?