OPS-12: Caring for an Ailing AppServer

1
OPS-12 Caring for an Ailing AppServer
Roy Ellis
Principal QA Engineer
2
Agenda

• Anatomy of the AppServer
• Diagnosing Problems
• Help Treating your AppServer

AppServer Broker
Cl ients
Agents
3
Anatomy of the AppServer
What is the OpenEdge AppServer?
4
Anatomy of the AppServer
Or What is the Progress AppServer?

• Broker
• JavaTM process
• Handles communication from clients to agents
• Manages client and agent pools
• Writes to ltbrokernamegt.broker.log

ltbrokernamegt.broker.log
5
Anatomy of the AppServer
Or What is the Progress AppServer?

• Agent
• C process
• Handles requests and replies to the clients
• Communicates STATUS to broker
• Writes to ltbrokernamegt.server.log

ltbrokernamegt.server.log
6
Anatomy of the AppServer
4 Operating Modes

• State-aware
• State-reset
• Stateless
• State-free

One to one
- Default
Many to few
7
Anatomy of the AppServer
2 Connection Models

• Client to Broker gets redirected to Agent
• State-aware
• State-reset
• Client to Agent through the Broker
• Stateless
• State-free

8
Anatomy of the AppServer
2 Connection Models State-aware State-reset
2 Connection Models Stateless State-free
Agent
Agent
Agent
Agent
AppServer Broker
client
Agent
9
Anatomy of the AppServer
AppServer Broker Thread Pools
Listening Thread L-3090
AppServer Broker
Client Threads C-0001 C-0002 C-0003 C-0004 C-0005
C-0006
Server Threads S-0001 S-0002 S-0003 S-0004 S-0005
10
Anatomy of the AppServer
Broker communication with Agents
Broker starts agent and waits for agent to respond, status set to STARTING
Once started the agent reports to broker and broker sets status to AVAILABLE
Client request agent, is redirected to agent port, broker sets status to CONNECTING
Client connects to agent, agent reports to broker, broker sets status to CONNECTED
Agent completes request, notifies broker, broker sets status to AVAILABLE
11
Anatomy of the AppServer
Reading Log Files
P-001268 T-009968 2 AS AS
08/04/04_at_114131.334-0400

• 08/04/04_at_114131.334-0400 P-001268 T-009968 2
  AS AS Application Server connected with
  connection id 172.30.1.47asbroker13090b0d80
  cae721191a17d1f0-7ffc. (8358)
• 08/04/04_at_114131.334-0400 P-001268 T-009968 2
  AS 4GLTRACE Run performCardAuth "John Doe
  1234567890 Visa -H localhost -S 6789" Main Block
  - authcard.p
• 08/04/04_at_114131.334-0400 P-001268 T-009968 2
  AS 4GLTRACE Run connectAuthSvc "-H localhost -S
  6789 " performCardAuth - authcard.p
• 08/04/04_at_114131.350-0400 P-001268 T-009968 2
  AS 4GLTRACE Run createAuthMsg "John Doe
  1234567890 Visa " performCardAuth - authcard.p
• 08/04/04_at_114131.350-0400 P-001268 T-009968 2
  AS 4GLTRACE Run sendAuthMsg "1000 NameJohn
  DoeNumber1234567890TypeVisa "
  performCardAuth - authcard.p
• 08/04/04_at_114207.397-0400 P-001268 T-009968 2
  AS 4GLTRACE Run disconnectAuthSvc "1000"
  performCardAuth - authcard.p
• 08/04/04_at_114207.428-0400 P-001268 T-009968 2
  AS AS Application Server disconnected. (8360)

Application Server connected with connection id
172.30.1.47 asbroker13090b0d80cae721191a17
d1f0-7ffc. (8358)
AS 4GLTRACE Run connectAuthSvc -H localhost -S
6789 performCardAuth - authcard.p
12
Anatomy of the AppServer
Supporting Processes

• AdminServer - Required
• JAVA process
• Manages all OpenEdge Brokers and Servers
• Writes to admserv.log
• NameServer - Optional
• JAVA process
• Traffic manager from clients to AppServers
• Writes to ltnameservernamegt.ns.log

13
Agenda

• Anatomy of the AppServer
• Diagnosing Problems
• Help Treating your AppServer

14
Diagnosing Problems
Startup Problems

• DLC/bin/asbman -name asbroker1 -start
• OpenEdge Release 10.1C01 as of Fri May 9
  215830 EDT 2008
• Connecting to Progress AdminServer using
  rmi//localhost20931/Chimera (8280)
• Searching for asbroker1 (8288)
• Connecting to asbroker1 (8276)
• Unable to start asbroker1 (8297)

15
Diagnosing Problems
Startup Problems - ltbrokernamegt.broker.log

• 
  
• 08/05/18_at_200525.933-0400 P-000283 T-Main 1
  --- --- /usr1/stat/progress/101c/wrk/asbroker1.bro
  ker.log opened.
• 08/05/18_at_200525.939-0400 P-000283 T-Main 1
  --- --- Logging level set to 2
• 08/05/18_at_200525.940-0400 P-000283 T-Main 1
  --- --- Log entry types activated
  UBroker.Basic,08/05/18_at_200526.020-0400
  P-000283 T-Main 2 UB Basic ubroker
  version v101c (07-May-08) (8038)
• 08/05/18_at_200526.314-0400 P-000283 T-L-9090 2
  UB Basic Started listenerthread L-9090
  (8043)
• 08/05/18_at_200526.394-0400 P-000283 T-L-9090 1
  UB ---------- Could not listen on port 9090
  (8044)
• 08/05/18_at_200526.394-0400 P-000283 T-L-9090 1
  UB ---------- java.net.BindException Address
  already in use (errno226)
• 08/05/18_at_200526.394-0400 P-000283 T-L-9090 1
  UB ---------- at java.net.PlainSocketImpl.socketB
  ind(Native Method)
• 08/05/18_at_200526.394-0400 P-000283 T-L-9090 1
  UB ---------- at java.net.PlainSocketImpl.bind(Pl
  ainSocketImpl.java359)
• 08/05/18_at_200526.394-0400 P-000283 T-L-9090 1
  UB ---------- at java.net.ServerSocket.bind(Serve
  rSocket.java325)
• 08/05/18_at_200526.394-0400 P-000283 T-L-9090 1
  UB ---------- at java.net.ServerSocket.ltinitgt(Ser
  verSocket.java186)

16
Diagnosing Problems
Startup Problems

• DLC/bin/asbman -name asbroker1 -start
• Connecting to Progress AdminServer using
  rmi//localhost20931/Chimera (8280)
• Searching for asbroker1 (8288)
• Connecting to asbroker1 (8276)
• Starting asbroker1. Check status. (8296)
• DLC/bin/asbman -name asbroker1 -query
• Connecting to Progress AdminServer using
  rmi//localhost20931/Chimera (8280)
• Searching for asbroker1 (8288)
• Connecting to asbroker1 (8276)
• Broker asbroker1 not running (8313)

17
Diagnosing Problems
Startup Problems - ltbrokernamegt.broker.log

• 08/05/18_at_201521.807-0400 P-000435 T-S-0004 2
  UB Basic Started server /usr1/stat/progress/
  101c/dlc/bin/_proapsv -logginglevel 2 -logfile
  /usr1/stat/progress/101c/wrk/asbroker1.server.log
  -ubpid 435 -Ms 1 -logname asbroker1
  -logentrytypes ASPlumbing,DB.Connects
  -logthreshold 0 -numlogfiles 3 -ASID 4
  ubpropfile /usr1/stat/progress/101c/dlc/propertie
  s/ubroker.properties -svrefresh -ipver IPv4 -db
  mydatabase (8108)
• 08/05/18_at_201522.210-0400 P-000435 T-S-0004 1
  UB ---------- Message from server Startup
  Procedure Failed (8113)
• 08/05/18_at_201522.241-0400 P-000435 T-S-0004 1
  UB ---------- IOException reading message from
  server java.io.EOFException pipe to server
  broken (8117)
• 08/05/18_at_201522.241-0400 P-000435 T-S-0004 2
  UB Basic ERROR cannot start server. (8100)

18
Diagnosing Problems
Startup Problems - ltbrokernamegt.server.log

• 08/05/18_at_202913.991-0400 P-000667 T-000000 1
  AS -- There is no server active for database
  mydatabase on -H cruz -S 6543. (704)
• 08/05/18_at_202913.991-0400 P-000667 T-000000 1
  AS -- Error initializing the application server.
  (5479)
• 08/05/18_at_202913.992-0400 P-000667 T-000000 2
  AS AS Application Server Shutdown. (5476)
• 08/05/18_at_202915.001-0400 P-000668 T-000000 1
  AS -- There is no server active for database
  mydatabase on -H cruz -S 6543. (704)
• 08/05/18_at_202915.001-0400 P-000668 T-000000 1
  AS -- Error initializing the application server.
  (5479)
• 08/05/18_at_202915.002-0400 P-000668 T-000000 2
  AS AS Application Server Shutdown. (5476)

19
Diagnosing Problems
Startup Problems - ltbrokernamegt.server.log

• 08/04/03_at_130333.470-0400 P-008824 T-004332 2
  AS AS Starting application server for asbroker1.
  (5560)
• 08/04/03_at_130333.501-0400 P-008824 T-004332 2
  AS AS Application Server Startup. (5473)
• 08/04/03_at_130333.814-0400 P-008824 T-004332 1
  AS -- C\tmp\ex08\roy\eg1\asstart.r Database
  sports2000 not connected. (1006)
• 08/04/03_at_130333.814-0400 P-008824 T-004332 1
  AS -- C\tmp\ex08\roy\eg1\asstart.r startup
  procedure ended with an ERROR condition. (8025)
• 08/04/03_at_130333.814-0400 P-008824 T-004332 2
  AS AS Application Server Shutdown. (5476)

20
Diagnosing Problems
Startup Problems

• All INITIAL agents must start or the AppServer
  will fail to start
• Set initial agents to start to zero
• initialSrvrInstance0 in ubroker.properties
• (IPv6 enabled on a machine can cause agents not
  to communicate with broker)
• Add Djvmstart.debug1 to the jvmargs in
  AdminServerPlugins.properties
• View WRKDIR/jvmStart.log file

21
Diagnosing Problems
Phantom Agents

• Broker Name asbroker1
• Operating Mode State-reset
• Broker Status ACTIVE
• Broker Port 3090
• Broker PID 25319
• Active Servers 5
• Busy Servers 0
• Locked Servers 0
• Available Servers 5
• Active Clients (now, peak) (0, 6)
• Client Queue Depth (cur, max) (0, 11)
• Total Requests 124
• Rq Wait (max, avg) (259 ms, 11 ms)
• Rq Duration (max, avg) (5543 ms, 24 ms)
• PID State Port nRq
  nRcvd nSent Started Last
  Change
• 25320 CONNECTED 02003 000020 000020 000020 May
  19, 2008 0628 May 19, 2008 0728
• 25321 AVAILABLE 02018 000031 000031 000031 May
  19, 2008 0628 May 19, 2008 0728
• 25323 AVAILABLE 02019 000014 000014 000014 May
  19, 2008 0628 May 19, 2008 0728

22
Diagnosing Problems
Phantom Agents

• ps -ef grep _proapsv
• statqa 25324 25319 0 062847 ? 024
  /usr1/stat/progress/101c/dlc/bin/_proapsv
  -logginglevel 2 -logf
• statqa 25323 25319 0 062846 ? 056
  /usr1/stat/progress/101c/dlc/bin/_proapsv
  -logginglevel 2 -logf
• statqa 25321 25319 0 062845 ? 055
  /usr1/stat/progress/101c/dlc/bin/_proapsv
  -logginglevel 2 log

protrace.25320 protrace.25322
23
Diagnosing Problems
Diagnosing Application problems

• Look at log files
• ltbrokernamegt.broker.log
• ltbrokernamegt.server.log
• Database log files where agent is connected
• Increase logging levels
• srvrLoggingLevel4 (extended)
• srvrLogEntryTypes4GLTrace
• UBNet - New for 10.1C

24
Diagnosing Problems
Hanging Agents

• Broker Name asbroker1
• Operating Mode Stateless
• Broker Status ACTIVE
• Broker Port 3090
• Broker PID 25319
• Active Servers 1
• Busy Servers 0
• Locked Servers 0
• Available Servers 1
• Active Clients (now, peak) (0, 1)
• Client Queue Depth (cur, max) (0, 1)
• Total Requests 11
• Rq Wait (max, avg) (11 ms, 11 ms)
• Rq Duration (max, avg) (24 ms, 11 ms)
• PID State Port nRq nRcvd
  nSent Started Last Change
  
• 25320 SENDING 02003 000001 000001 00001 Apr
  4, 2008 1140 Apr 4, 2008 1141

25
Diagnosing Problems
Diagnosing Agent problems

• 08/04/04_at_114131.334-0400 P-001268 T-009968 2
  AS AS Application Server connected with
  connection id 172.30.1.47asbroker13090b0d80
  cae721191a17d1f0-7ffc. (8358)
• 08/04/04_at_114131.334-0400 P-001268 T-009968 2
  AS 4GLTRACE Run performCardAuth "John Doe
  1234567890 Visa -H localhost -S 6789" Main Block
  - authcard.p
• 08/04/04_at_114131.334-0400 P-001268 T-009968 2
  AS 4GLTRACE Run connectAuthSvc "-H localhost -S
  6789 " performCardAuth - authcard.p
• 08/04/04_at_114131.350-0400 P-001268 T-009968 2
  AS 4GLTRACE Run createAuthMsg "John Doe
  1234567890 Visa " performCardAuth - authcard.p
• 08/04/04_at_114131.350-0400 P-001268 T-009968 2
  AS 4GLTRACE Run sendAuthMsg "1000 NameJohn
  DoeNumber1234567890TypeVisa "
  performCardAuth - authcard.p
• 08/04/04_at_114207.397-0400 P-001268 T-009968 2
  AS 4GLTRACE Run disconnectAuthSvc "1000"
  performCardAuth - authcard.p
• 08/04/04_at_114207.428-0400 P-001268 T-009968 2
  AS AS Application Server disconnected. (8360)

26
Agenda

• Anatomy of the AppServer
• Diagnosing Problems
• Help Treating your AppServer

27
Help Treating your AppServer
asbman

• -query
• Get status and history of AppServer broker and
  agents
• -kill
• Shutdown an AppServer with busy or hanging agents
• May need to run twice in a row

28
Help Treating your AppServer
LogRead

• Available on PSDN
• ABL source code available
• Read and Sort multiple log files
• Unsupported

29
Help Treating your AppServer
Database Connections

• Dont connect using the startup parameters
• Any database down to a single database will cause
  your AppServer agents to be unresponsive
• Connect using the CONNECT call in the
• srvrStartupProc
• srvrConnectProc
• srvrActivateProc
• Use IF CONNECTED ABL

30
Help Treating your AppServer
Lifespan

• New in 10.1B
• Described in ubroker.properties.README
• Read the CAUTION!
• Use only when directed by Tech Support
• Periodically force the shutdown of all agents and
  allow new agents to start when requested

31
Help Treating your AppServer
OpenEdge Management
32
Under Development

• The next slide includes information about
  potential future products and/or product
  enhancements.
• What I am going to say reflects our current
  thinking, but the information contained herein is
  preliminary and subject to change. Any future
  products we ultimately deliver may be materially
  different from what is described here.

33
Help Treating your AppServer
List AppServer Connections

• New in 10.2A
• Described in ubroker.properties.README
• List of all clients known to the AppServer,
  includes
• Host/IP
• Port
• Connected Agent
• More!

34
Help Treating your AppServer
Help from Progress

• Technical Support
• Knowledge Base (kbase)
• PSDN

35
In Summary

• Understand your AppServer
• Diagnose your Problem
• Care for your Ailing AppServer

AppServer Broker
Cl ients
Agents
36
Relevant Exchange Sessions

• OPS-15 What was Happening with my Database,
  AppServer, Operating System Yesterday, Last
  Month, Last Year?
• SOA-38 OpenEdge Middleware Road Map and Info
  Exchange
• OPS-19 What is IPv6 and Why Should I care?
• OPS-13 Building and Deploying a Highly Available
  Application

37
?
Questions
38
Thank You
39
(No Transcript)