Information is at the heart of all architecture disciplines & why Conceptual data modeling is essential

of 33
7 views
PDF
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Document Description
Information is at the heart of all of the architecture disciplines such as Business Architecture, Applications Architecture and Conceptual Data Modelling helps this. Also, data modelling which helps inform this has been wrongly taught as being just for Database design in many Universities. chris.bradley@dmadvisors.co.uk
Document Share
Document Transcript
  • 1. ENTERPRISE ARCHITECTS WHITEPAPER INFORMATION IS AT THE HEART OF ALL ARCHITECTURE DISCIPLINES AND THERE’S MORE TO DATA MODELLING THAN YOU EVER THOUGHT
  • 2. 2 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED INFORMATION IS AT THE HEART OF ALL ARCHITECTURE DISCIPLINES DOC DATE MARCH 2014 PREPARED BY: CHRISTOPHER BRADLEY CHIEF INFORMATION ARCHITECT & ENTERPRISE SERVICES DIRECTOR VERSION: V1.0.0 ENTERPRISE ARCHITECTS ©2014, all rights reserved
  • 3. 3 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED CONTENTS DATA MODELLING IS A CRITICAL TECHNIQUE AND AT THE HEART OF ALL ARCHITECTURE DISCIPLINES 5   DATA MODELLING INTRODUCTION 6   BACKGROUND & HISTORY 7   DIFFERENT TYPES OF MODELS FOR DIFFERENT PURPOSES AND AUDIENCES 9   DATA MODELLING FOR DBMS DEVELOPMENT 10   DATA MODELLING INCORRECTLY TAUGHT AT UNIVERSITY 14   WHAT NEEDS TO CHANGE? 15   MODELLING FOR THE “NEW” TECHNOLOGIES 16   DEMONSTRATING BENEFITS 28   THE GREATEST CHANGE REQUIRED 29   WHAT NEEDS TO STAY THE SAME? 31  
  • 4. 4 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED DATA MODELLING IS… A CRITICAL TECHNIQUE AND AT THE HEART OF ALL ARCHITECTURE DISCIPLINES Many years ago people believed the world was flat and if they sailed over the horizon, then they would fall off the edge. They also believed that the Earth was at the centre of the heavens, and that all other planets orbited around it. But they were wrong. People who believe Data Modelling is just for DBMS design are just as misinformed. Data Modelling, particularly Conceptual Data Modelling is an absolutely critical technique and is at the heart of all architecture disciplines. Here’s why: Since data has to be understood to be managed, it stands to reason that gaining agreement on the meaning and definition of concepts will be a key component. That is precisely what a data model provides. But just what do I mean when I state that Data Modelling is at the heart of all architecture disciplines? FIGURE 1: Data Modelling is at the heart of all architecture disciplines At its heart, the Data Model provides the unifying language, lingua franca, the common vocabulary upon which everything else is based. Other modelling techniques within the complimentary architecture disciplines will interact with each other, forming a supportive; cross- checked, integrated and validated set of techniques. It’s not just (sometime it’s never) about technical DBMS design.
  • 5. 5 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED SO TO ILLUSTRATE THE CASE WITH A FEW SIMPLE EXAMPLES, WE SEE IN: Every type of model references the entities of significance in the conceptual data model, showing why conceptual data modelling is such a vital technique. The Business Architecture Domain: A Project Charter documents the rationale, the objectives, the business scope, and measures the success of the project. It uses the language of a high level data model to describe the business concepts. The Process Architecture Domain: A Workflow Model describes the sequence of steps carried out by the actors involved in the process The Application & Systems Architecture Domain: A Use Case describes how an actor completes a step in the process, by interacting with a system to obtain a service. A Service Specification describes some form of business service that is initiated to complete a business event. The Information Architecture Domain: A Data Model depicts the critical data items, and the attributes or facts about them. This is important data that the organisation wishes to know or store information on, and is the stuff that the processes and systems act on. Getting agreement on the language and definition of the data concepts always must always occur first; once established detail about processes can be added: › To begin we discover the Nouns: i.e. the items of interest to the organisation , e.g. “Product” “Customer” “Location” › Next we discover “Verb – Noun” pairs: These are activities that must be performed, such as process and sub-process, in order for the organisation to operate, e.g. “Design Product” “Ship Order” › Lastly we discover “Actor – Verb – Noun “ combinations: These form the Use Cases or steps within a business process, e.g. “Lead Architect Designs New Product” At this high level, we are seeking to gain an understanding and agreement on terms and vocabulary for the data concepts. We do not want to get bogged down in the level of excruciating detail that a detailed logical model would take us into. Thus high level conceptual models (often called Business Data Models) are the appropriate vehicle to use here. It can be loosely argued that they provide some of the features of an “ontology” i.e. business concepts and their relationships, although a Conceptual Data Model with its metadata extensions provides much more.
  • 6. 6 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED DATA MODELLING INTRODUCTION The problem for many Data Architects is that “Data Modelling” has, in far too many companies received a lot of bad press. Have you heard any of these? › “It just gets in the way”, › “It takes too much time”, › “What’s the point of it”; › “It’s not relevant in today’s systems landscape”. › “I don’t need to do modelling, the package has it all” Yet when Data Modelling first came onto the radar in the mid 1970’s the potential was enormous: We were told we would realise benefits of: › "a single consistent definition of data" › "master data records of reference" › “reduced development time” › “improved data quality” › “impact analysis” › to name but a few. Do organisations today want to reap these benefits? You bet, it’s a no- brainer. So then, why is it that now, here we are, 30+ years on and we see in many organisations that the benefits of Data Modelling still need to be “sold” and in others the big benefits simply fail to be delivered? What’s happened? What needs to change? As with most things a look back into the past is a good place to start.
  • 7. 7 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED BACKGROUND & HISTORY Looking back into the history of data management; we see a number of key eras. 1950’s – 70’s: Information Technology (at that time often called Automated Data Processing (ADP)) was starting to enter the mainstream world of commerce. During this period we saw the introduction of the first database management systems such as DL1, IMS, IDMS and TOTAL. Who can remember a DBMS that could be implemented entirely on tapes? ** At that time the cost of disc storage was exceptionally high, and the notion of exchangeable disc packs was just coming into the data centre. The concept of “database” operations came into play and the early mentions of “corporate central databases” appeared. ** It was IMS HISAM if you really want to know. 1970 – 1990: Data was “discovered”. Early mentions of managing data “as an asset” were seen and the concepts of Data Requirements Analysis and Data Modelling were introduced. 1990 – 2000: The “Enterprise” became flavour of the decade. We saw Enterprise Data Management Coordination, Enterprise Data Integration, Enterprise Data Stewardship and Enterprise Data Use. An important change began to happen in this period, there was a dawning realisation that “technology” alone wasn’t the answer to many of the information issues, and we started to see Data Governance being talked about seriously. 2000 and beyond: Data Quality, Data as a Service, Data Security & Compliance, Data Virtualisation, Services Oriented Architecture (SOA), governance and alignment with the business were (and still are) the data management challenges of this period. All of this needs to be undertaken in these rapidly changing times when we have a “new” view of information: Web 2.0, Blogs, Mash-ups, Data Virtualisation. It seems anyone can create data! At the same time we have a greater dependence on “packaged” or COTS (Commercial off the shelf) applications such as the major ERPs. There is also more and more use of SOA, XML, business intelligence and less reliance on traditional “bespoke” development.
  • 8. 8 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED NOTICE I SNEAKED IN “MASH-UPS” (OR WEB APPLICATION HYBRID) THERE? See the Wiki article for more on mash-ups. There are many powerful facilities available now that enable you to create your own mash-ups. Make no mistake, these are now becoming the new “Shadow IT” of this decade. Remember the home grown departmental Excel macros of the 90’s and onwards that became “critical” to parts of the business? Now mash-ups are doing the same thing. But just who is looking at the data definitions, the data standards, applicability etc.? Certainly not the data management group – because frequently they don’t even know that these functions are being built in departmental silos, and anyway the “data team” is pigeon holed as being only involved in DBMS development. So that leads us on to examine the belief that many people still have (too many unfortunately) that Data Modelling is only for DBMS development. So why is that? Firstly we’ll look at Data Modelling for use in DBMS development.
  • 9. 9 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED DIFFERENT TYPES OF MODELS FOR DIFFERENT PURPOSES AND AUDIENCES In its early days Data Modelling was mostly (almost exclusively) what we now call Logical and/or Physical Data Modelling and it was primarily aimed at DBMS development. However, there are many different levels of “Data Models” that can be developed, and they each have a different purpose and audience: FIGURE 2: Levels of Data Models From Figure 2 above, we see there are many different levels of “Data Models”. The higher up the pyramid we go, the more “communication” focused the models are. Whereas the further down the pyramid we go the more “implementation focused the models are. Frequently, a higher level model is created with the sole purpose of improving communication and understanding. FIGURE 3: Purpose of data model levels
  • 10. 10 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED DATA MODELLING FOR DBMS DEVELOPMENT In its early days data modelling was primarily aimed at DBMS development. We’ll have a look at the two main techniques in a moment. Just to illustrate this we can look at 4 typical roles that may be considered as “customers” of the data modelling output: The Enterprise data customer: This might be at Director or CxO level. The accuracy of data is critical, they are reports users, and the data “products” that data professionals produce are key to serving the needs of this level of user. The Data Architect: This person knows the business and its rules. He/she manages knowledge about the data and defines the conceptual direction and requirements for capturing of data. The DBA: This person is production oriented, manages data storage and the performance of databases. He also plans and manages data movement strategies and plays a major part in data architecture by working with architects to help optimise and implement their designs in databases. The developer DBA: This role works closely with the development teams and is focused on DBMS development. They frequently move and transform data often writing scripts and ETL to accomplish this. Data models (more accurately the metadata) were (and are) seen as the glue or the lingua franca for integrating IT roles through the DBMS development lifecycle. All of the roles above depend on metadata from at least one of the other roles. What then are the steps for developing a DBMS and utilising Data models? What then are the steps for developing a DBMS and utilising Data Models? Firstly a word of warning; this could be the subject of a huge paper in its own right, but I’ll try and summarise it simply here:
  • 11. 11 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED THERE ARE TWO “MAIN” APPROACHES TO CREATING DBMS’S FROM MODELS There are two “main” approaches to creating DBMS’s from models: One is the “top down” or “to-be” approach and the other is termed the “bottom-up” or “as-is” approach. TOP DOWN (TO-BE) APPROACH STEP 1: When speaking with business representatives, discover and document the business requirements, before agreeing on a high-level scope. The output is typically some form of Business Requirements Document (BRD). This will give an understanding at a high level, of the concept where the data is used by business processes, and vice versa. STEP 2: Create a more detailed business requirement document with subscriber data requirements, business process and business rules. STEP 3: Understand and document the business keys, attributes and definitions from business subject matter experts. From this create and continually refine a logical data model. Determine what the master entities are and what is common to other business areas. STEP 4: Verify the logical data model with the stakeholders. Walk a number of major business use cases through and refine the model. Apply the technical design rules with knowledge of the technical environment that you are going to implement the solution on, use known volumetric and performance criteria and create a first cut physical data model. Remember the same logical model could be implemented in different ways upon varying technology platforms. STEP 5: Generate the Data Definition Language (DDL) from the physical model. Refine the physical design with DBA support and implement the DBMS using the refined physical model. This top down approach has an advantage that the “New” or “To-Be” business and data requirements are the main priority. In the early days there were not many “existing systems” to consider, a good job because the approach doesn’t take into account any of the hidden nuances & rules that may be deep down within the existing systems.
  • 12. 12 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED The primary purpose of the Bottom Up (or As-Is) Approach is to create a model of an existing system into which the new requirements can be added. Frequently, the bottom-up approach is used because a model of the current system simply doesn’t exist. Often because it has evolved and/or the original design staff have retired, died, or moved on and the documentation has not been kept up to date. BOTTOM UP (AS-IS) APPROACH The main steps in the bottom-up approach are: STEP 1: Reverse engineer the database of file schema from the system that is already implemented. From this you will have the database catalog, table, column, index names etc. Of course these will all be in “tech” language without any business definitions. STEP 2: Profile the real data by browsing and analysing the data from the tables. Scan through the ETLs to find out any hidden relationships and constraints. Modern data profiling tools are invaluable here as they will allow you to gain real insight to the data, way beyond simply trying to understand from the column names. You did know that SpareField6 really has the alternative delivery location? STEP 3: Find out foreign key relationships between tables from IT subject matter experts, and verify the findings. The typical output here is a refined physical model. STEP 4: Document the meanings of columns and tables from IT subject matter experts STEP 5: Try to understand the business meaning of probable attributes and entities that may be candidates for logical data model. From here the result is a “near logical” model. The bottom up approach is great for capturing those hidden “gotchas” that are tucked away inside the current system. However it doesn’t give any serious attention to new requirements. Thus, a third way is a hybrid of these two approaches that is frequently called the “Middle Out” Approach. The Middle Out Approach employs the best parts of the Top-Down and Bottom-Up Approaches. This is the approach I favour when designing a new model, which is likely to have a better chance of ultimately being used for a technology solution.
  • 13. 13 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED
  • 14. 14 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED DATA MODELLING INCORRECTLY TAUGHT AT UNIVERSITY Over the past 10 years or so I have been taken aback at what I have observed regarding the way in which Data Modelling is portrayed on courses at many Universities in the UK and USA (and I suspect in other places too). As part of my DAMA-I education brief (and to be honest as a way of giving something back to the community) I am frequently asked to speak not just at conferences but with academic institutions. Here are a few snippets I have pulled from 5 separate universities recently regarding data modelling on the Computer Science Bachelors & Masters courses: › “The purpose of a Data model is to design a relational database system” › “An ER Model is used to specify design and document Database design” › “A Data model is a pictorial representation of the structure of a relational database system” › “… it is a description of the objects represented by a computer system together with their properties and relationships” › “ER Modelling is a Database design method” At one of these I dug deeper and examined several of the course assignments. One assignment asked students to prepare a model to represent an office environment and in part of the detailed description within the assignment brief it mentioned the “Rolodex” and “IBM Selectric” that were on the desks in this office. Now, I’m not talking here of reading an assignment paper set for a course in 1975, this was one I saw in 2013!! Now with all of these uses of Data Models that I have described so far, the history of Data Modelling, the way it’s still being taught in some Universities, and judging from much of the literature from the Data Modelling tool vendors themselves; it not surprising that many people are left with the impression that data modelling is just for DBMS’s. But this is wrong!
  • 15. 15 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED WHAT NEEDS TO CHANGE? See also “Data Modelling For The Business – A Handbook for aligning the business with IT using high-level data models”; Technics Publishing; ISBN 978-0-9771400-7-7; The use and benefit of Data modelling is considerably greater than its current “one trick pony” press would suggest. To make Data Modelling relevant for today’s systems landscape we must show that it’s relevant for the “new” technologies such as: › ERP packages; › SOA & XML › Business Intelligence › Data Lineage › Data Virtualisation Without forgetting that an appropriate level Data Model is an awesome communication tool so it can for used for communicating with the business. See also “Data Modelling For The Business – A Handbook for aligning the business with IT using high-level Data Models”; Technics Publishing; ISBN 978-0-9771400-7-7; We also need to break away from the “you must read my detailed Data Model” mentality and make the information available in a format users can readily understand. For example this means that Data Architects need to recognise the different motivations of their users and re-purpose the model for the audience: Don’t show a business user a Data Model! Information should be updated instantaneously, and we must make it easy for users to give feedback, after all you’ll achieve common definitions quicker that way. We need to recognise the real world commercial climate that we’re working in and break away from arcane academic arguments about notations methodologies and the like. If we want to have Data Modelling play a real part in our business then it’s up to us to demonstrate and communicate the genuine benefits that can be realised. Remember, Data Modelling isn’t a belief system, just because you “get it” don’t assume that the next person does.
  • 16. 16 | ENTERPRISE ARCHITECTS © 2014, ALL RIGHTS RESERVED MODELLING FOR THE “NEW” TECHNOLOGIES I feel I must make a confession here. The technologi
  • Search Related
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks
    SAVE OUR EARTH

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!

    x