Bernd%20Bohne

1
in a banking data center -from the
past to the future

• Bernd Bohne
• Sparda-Datenverarbeitung e.G.

SAG UG NA - Conference Philadelphia 10/2006
2
Sparda-Datenverarbeitung eG
3
Sparda-Datenverarbeitung eG oriented on the
German market offers its service to 28
Bank-Companies with 7900 Employees and a total
Balance of 66.2 Bn. 23,600,000 Accounts
5,000,000 Customers
4
ADABAS was introduced in 1987 at SDV

• A new core core-banking application was
  introduced
• Batch-timeframe was to small
• VSAM-Files needs to be closed in Online (CICS
  req.)
• Batch didnt fit in ...
• Weekends have been very short in those days...
• At this time there was only a single
  bank-corporation inthe data-center
• Early shortcomings have been ironed out and
  incremental progress was done to harden the
  application

5
The next years have been very busy...

• The nagging question remainsHow to deal with
  big bank companies?
• Some of the tension disappears by
  implementingADABAS.
• Thought the core-banking application was VSAM
  based?
• SAG offers a migration tool, called ADABAS VSAM
  bridge
• via SVC-screening requests have been converted to
  ADABAS-calls

6
ADABAS VSAM Bridge (AVB)
SVC 19 open SVC 20 closeat CICS-Startup
PLT and CICS-Shut Down
CICS
Batch
SVC-Screening
AVB Transparency Table VSAM-File
ADABAS-File
7
Results of this move in general?

• Batch could run in parallel to CICS
• Datamodel could remain unchanged
• Restart informations could be used (ET-Data)

8
What does it mean to SDV?

• Time to fix the real problems in
  batch-processing
• The new option was SQL available (native SQL)
• New candidates have been in the queue to join SDV
• 4 Bavarian banks joined the data-center
• not only hosting, it was a migration process at
  the bank-companies of the whole banking
  application
• The German reunion happened in those days
• one bank, covering the whole market of east
  germany of this banking-group, joined SDV
• Several batch-programs must be optimized -
  usingSQL was crucial

9
To put it in a nutshell There was no time to
take care of other things Performance was again
a big issue! The really important problems must
be solved,some problems seemed to be similar as
beforeAdabas has been implemented.Get, Put,
Update and Delete havnt been the real McCoy -
but thats what VSAM offers at all and also the
AVB (converting the VSAM-commands to L3, N1, A1
and E1).It was time to start digging deeper
Adabas native SQL seems to be the key to more
performance!
10
A new era Adabas native SQL

• L9 (histogram), S1 (find) etc. offered new,
  more efficient options of processing huge files
• on the other hand side There was always a
  data-accessing-layer in the application
  programs but with Adabas-SQL the statements have
  been implemented in the normal program-code
• the co-existence of the Adabas-VSAM-Bridge and
  SQL in a program (here in terms CICS-transation)
  could lead to side-effects
• ... but anyway, pressing business and besides
  there are new bank-companies in the queue to join
  SDV

11
Whats about Adabas transactions in CICS?
130
EXEC CICSSYNCPOINT
EXEC CICSSYNCPOINT ROLLBACK
Syncpointprocessingwas realized via AVB-Code!
(CICS) DFHSPP
(CICS) DFHSPP
AVB SPP
AVB SPP
ET
BT
Adabas-transaction 1
Adabas-transaction 2
There was a very close link between CICS and
Adabas using theAdabas VSAM Bridge!
12
Dawn of realization There are limits in VSE

• More bank bank-companies intends to join SDV
• Logically VSE and VM seemed to be no longer the
  right platform
• A migration project started to migrate to MVS
• There was no time to fix additional thingsThe
  migration was again an exhausting task!

13
The year of consolidation 1996...

• After such a long period of time, there was the
  year of consolidation announced...!
• This turned to be one of the most busiest years
  in history of the company
• 2 bank-companies asked to join SDV - who wants to
  say no!
• But anyway, theres still a light at the
  horizonLets do it next year!
• A complete new banking-group decided in favour of
  SDV...

14
Automatic Teller Machine technology requires to
consolidate files
Bank A
database A
Bank B
database B
Bank C
database C
Each bank-company was from the infrastructure
point sitting in its own isolated
environment! Advantage and disadvantage Sharing
no data!
15
ATM technology requires to consolidate files...
?
Bank A
DB A
Copy
?
ATMFile
Bank B
DB B
Copy
?
Copy
Bank C
DB C
To copy every day specific files, does this
really make sense?
16
ATM technology requires to consolidate files...
Bank A
DB A
Bank B
DB B
Bank C
DB C
Read-only!
ATMFile
DB X
Copy
17
By the way ATM means 24 by 7 Project Sysplex
2000

• A 2nd Data Center was implemented
• DASD mirroring (PPRC) established
• Network-components like IP have been separated in
  2 Net-LPars and two production LPars
  established
• The whole System architecture was re-engineered
• Only redundancy could meet the much higher
  requirements of our customers
• Also at these days The 1st gateways were
  established to the Internet.

18
The big picture
Internet
WebSphere (Blades/Linux)
CICS Transaction Gateway
Routing CICS
A weak pointSingle point of failure
CICS Bank A
CICS Bank B
CICS Bank C
DB A
DB B
DB C
DB X
19
A new ATM application was implemented

• ATM-front-end (CICS-/DB2-based) was implemented
• DB2 Isolated in a small LPar after changing to
  WLCSW-Pricing (saved approx. 10k per month)
• This step was also necessary to bring DB2 with
  datasharing in an affordable cost-range
• Front-end and core-banking-application have been
  linked via DPL

20
And the application? The MBS-new project
started!

• Core banking application modular banking system
  was redesigned
• Data access layer was new established (incl.
  native SQL)
• Data access was in server-modules file-related
  established
• XML-Interface was established
• CICS-getmain-shared removed (CICSplex...!!!)
• AVB removed (how do you synchronize?)
• New requirementWhy is it not possible to update
  more than one database?(Modification-requests
  have been rejected by user-exit atthe
  link-interface).

21
Synchronization and Two-phase-commit

• The synchronization of the AVB was replaced by
  using the ADABAS Resource Manager Interface (RMI)
  via Task Related UserExit (TRUE).
• To allow update on multiple ADABAS databases
  means you need two-phase-commit, and this means
  Adabas Transaction Manager (ATM)
• The Version 1 had typical 1st generation
  shortcomings, nowadays, Version 7.5, at SDV
  currently 7.5.1 is running stabil and with very
  good performance.
• But, step-by-step the picture became more
  sophisticated...

22
Overview ADABAS, AFP/COR and ATM
145
Appl.-LPar P82
Appl.-LPar P96
DataCenter 2
DataCenter 1
AFP82 / COR
AFP96 / COR
NE T W O R K82
NE T W O R K96
CICS82
ADA82
CICS96
ADA96
XCF
ATM P582ATM
ATM P596ATM
PPRC
) Daemon-Mode
23
Centralized databases
Appl.-LPar P82
Appl.-LPar P96
DataCenter 2
DataCenter 1
CICS
CICS
ADA
ADA
ADA
Currently unfortunatlya single point of failure
24
Clustered databases in general (next steps)
Appl.-LPar P82
Appl.-LPar P96
DataCenter 2
DataCenter 1
CICS
CICS
.
.
.
.
.
.
.
ADA
ADA
XCF
ADA
ADA
The future option to reduce risk by eleminating
the single point of failure.
25
Clustered databases in general and ATM (ATM751)
Appl.-LPar P82
Appl.-LPar P96
DataCenter 2
DataCenter 1
ATM82
ATM96
A new single point of failure
XCF
CICS
CICS
.
.
.
.
.
.
ADA
ADA
The ATM 751 is clearly a new single point of
failure There are two linked ATMs but each one
is unique because its LPar-bound.
26
Clustered databases in general and ATM
(V8/CE/future?)
Appl.-LPar P82
Appl.-LPar P96
DataCenter 2
DataCenter 1
Clustered ATM
ATM
ATM
XCF
CICS
CICS
.
.
.
.
.
.
.
ADA
ADA
The next step in the evolution should be to
create an ATM-cluster in full datasharing mode
with automatic failover in case of an outage of
one of the ATM-tasks.
27
Current Infrastructure (Mainframe)

• 2x z9 (IBM 2094-705) - each 64 GB
• 2x z890 (IBM 2086 / 2 Engines CF / each 16 GB)
• Coupling Links (XCF) ISC-D (Card) and ISC-3
  (Port)
• 2x HDS USP (USP600 USP100, each partitioned)
• 2x HDS 9970 (MF only)
• DASD Direct attached Ficon
• OS zOS 1.8 and zVM 5.2
• CF CC14
• Tape STK SL8500/9940B, STK 9310 (Powderhorn)/
  9840 both at each DataCenter, Titanium 10,000
  andFSC CentricStor

28
Current Infrastructure XCF/Coupling Links
DataCenter 1
DataCenter 2
IBM z890
IBM z890
ISC-D / ISC-3 (Fiber)
(8x)
(6x)
(6x)
(6x)
(6x)
1.6 km
IBM z9
IBM z9
29
Level and Configuration ADABAS and Add On

• ADABAS
• ADA743 LX08 (ATM)
• ADABAS Fastpath
• AFP742 (Global Buffer at both DataCenter)
• ADABAS Transaction Manager
• ATM751 (incl. CICS-RMI)
• ADABAS System Coordinator
• COR742 (Daemon Mode)

30
XCF - Request Time (Hardware)

• Started with shared CPUs 300 - 400 micSec.
• dedicated CPUs 20 - 80 micSec.
• Influence of ISC-D and ISC-3 (interface/port)

) Average over-all, not ADABAS DS-specific -
env. GRS Star, CICS TS 2.3, z/OS 1.6
31
Our current Status (ADABAS datasharing)

• Since end 06/2006 we have all ADABAS env. in
  test-sysplex in a datasharing configuration
• We started with minor-important environments in
  production in 07/2006 with ALS
• If no further obstacles appearWe will finish in
  2006 with the rollout in production,latest end
  2nd quarter 2007
• Currently we see no influence of ALS to AFP/COR,
  ATM (necessary configuration changes)

32
Stresstest CoreBanking Application with
datasharing

• Nov. 18th, 2005
• Response of file-requests behind expectations
  -reason shared CPUs (see XCF request time)
• CICS-Txn-Rate 4.000 - 5.000 Txns per min.
  -current max. is up to 3.000 Txns per min. each
  CICS
• workload driver could not generate needed
  peakNo outages or malfunctions have been
  detected in SAG components!

33
Experiences with ADABAS Cluster Services
200

• Frame-Work
• How to control two or more DBMS for one database
  (System Automation)
• You need more DASD-Space for the DBMS and
  PLog(2PC Work-Part-4)
• A minimum of 1 XCF is recommended(but 2 XCF
  were in our case cheaper)
• Technological backlevel of XCF is high
  likely(... or you modernize the Application-MF
  and XCF at the same time - costs/budget!?)

34
Experiences with ADABAS Cluster Services

• ADABAS
• dedicated XCF-CPUs are crucial for datasharing in
  general
• really needed Storage in the XCF is currently not
  clear
• Usage of the fastest available Coupling-Links
  recommended
• Changed PLog-Processing (PLog-Merge, PLog-Merge
  DS)
• detected several times specific changes
  (maintenance) ALS(currently several ADABAS LMODs
  in LX08-Load-Library)
• Currently no common view for Review, AOS, ASF and
  CLog(in case of CLog Merge necessary)

35
Modernization of the Application
CICS
XML-Interface
Core BankingApplication
Core BankingApplication
Web-Services
Data Access Layer (SQL)
36
Modernization of the Application Establish
CICSplex
Net.-LPar P94
Net.-LPar P96
IP
IP
TOR/CICS
TOR/CICS
CPSM
CPSM
DataCenter 2
DataCenter 1
Appl.-LPar P82
Appl.-LPar P96
ATM82
ATM96
AOR/CICS
AOR/CICS
XCF
.
.
.
.
.
.
ADA
ADA
37
Infrastructure Full Clustered Environment
Net.-LPar P94
Net.-LPar P96
IP
IP
TOR/CICS
TOR/CICS
CPSM
CPSM
DataCenter 2
DataCenter 1
Appl.-LPar P82
Appl.-LPar P96
ATM
ATM
XCF
AOR/CICS
AOR/CICS
.
.
.
.
.
.
ADA
ADA
38
CICSplex

• Remove application system affinities
• Share common data outside CICS CICS File Owning
  Region or shared database
• Establish dynamic routing CPSM CICSplex System
  Manager including Workload Balancing

39
Business Continuity Freeze and Hiperswap (GDPS)
)Geographically Dispersed Parallel Sysplex
Net.-LPar P94
Net.-LPar P96
IP
IP
TOR/CICS
TOR/CICS
CPSM
CPSM
DataCenter 2
DataCenter 1
Appl.-LPar P82
Appl.-LPar P96
ATM
ATM
1. Freeze (suspend I/Os)2. Hiperswap
(UCB Switch)
XCF
AOR/CICS
AOR/CICS
.
.
.
.
.
.
ADA
ADA
X
X
40
WebSphere / CTG / CICS
WebSphere Portal
Env. Linux on Blade
Sniffingimpossible ConnectedviaHipersocket
CTG(CICS Transaction Gateway)
Env. Linux on System z
RoutingCICS
Env. z/OS on System z
ApplicationCICS
Env. z/OS on System z
41
Linux in a high availability environment
215

• Started in 2004 to consolidate Linux Server on MF
• OS z/VM (currently z/VM 5.2)
• Hosting
• Tivoli Storage Manager (TSM)
• Database Server (DB2 UDB and Oracle)
• CICS Transaction Gateway
• Currently 2 x 2 IFL

Systemautomation for Multiplatform (SAfMP)
Contains Agent to plug z/VM including Linux in
GDPS (Freeze/Hiperswap)
42
Overview Linux on z

• Production 13 Server
• Integration - QA 9 Server
• Test and Education 5 Server

• Advantages
• Consolidation in general (Administration)
• Higher Usage of installed resources
• PPRC-Protection (Business Continuity)
• DR (CBU Capacity Backup and GDPS)
• significant savings regarding license-fee/SS

43
Our Goal As close as possible to 100
Availability
Application
Data
Desaster Recovery
CICSplex -dynamic Routing/Processing (DTR)
Datasharing
Freeze Hyperswap (GDPS)
) to deal with planned / unplanned outages and
flexible workloadbalancing
44
Questions?