Software Evaluation in AI

1
Software Evaluation inAI
2
The Selection Problem

• Recency of products
• Evolving nature of products
• Variety of products
• Lack of standards

3
Common Approach to Choice

• Select a few seemingly important or complicated
  functions within the products, and compare the
  products based on them.

4
Shortcomings

• It is evaluator-specific and unreliable.
• No product could be judged by only a few
  attributes.
• It excludes the relevant issues in a particular
  application.
• It lacks an evaluation measure, reflecting the
  strength of a product or its ratings compared
  with its competitors.

5
A Structured Approach

• Contain all of the important features of AI
  products
• Make it possible to compare different types of
  products
• Accommodate the special needs and requirements of
  AI project
• Summarize the results of evaluation into a
  quantitative or qualitative measure

6
Attribute Hierarchy of AI Tools

• Financial Aspects
• Producer Aspects
• Special Aspects
• Hardware Aspects
• Functional Aspects

7
Financial Aspects

• One-time costs purchasing cost
• Periodic costs licensing, training, and
  maintenance

8
Producer Aspects

• Reputation
• Length of time in business
• Product line compatibility issues
• Technical support

9
Social Aspects

• User group
• Number of users
• Compatible products technical leadership of a
  product

10
Hardware Aspects

• Platform OS, Networking, Parallel Processing
• I/O Devices
• Resource Requirements RAM, Disk Storage
• Efficiency Response Time

11
Functional Aspects

• Knowledge Representation
• Inference Engine
• Knowledge Management
• Outside Hooks

12
Knowledge Representation

• Logic-Based Representation Rule, Fuzzy Logic
• Object-Based Representation Frame, Semantic Net
• Uncertainty Representation Bayesian, Fuzzy
  Logic, Certainty Factors, User-Defined
• Meta-Knowledge Representation
• Mathematical Representation Math Operations,
  Math Functions, Variables

13
Inference Engine

• Chaining Forward, Backward, Mixed
• Induction Dec. Tree
• Object-Oriented Single vs. Multiple Inheritance
• Blackboard
• Conflict Resolution Recency, Antecedent Ordered,
  Consequent Ordered, Top-Down

14
Knowledge Mgt. Tools

• User-Interface Creation Graphics, Windows, NLP,
  Voice Input/Output, Help, Animation
• User Interface How, Why, Graphics, Uncertainty
  I/O, Help
• Debugging Tools Tracing, Error Messages
• Knowledge Maintenance Editor, Menu, Mouse,

15
Knowledge Mgt. Tools Continued

• System Security User Access, KB Control
• Integrated Tools Databases, Spreadsheets, Forms,
  Files, Prog. Languages
• Developer - User Assistance On-Line Documents,
  Off-Line Documents, Tutorials, Error-Message
  References

16
Outside Hooks Components

• Access Data Database, Files, Spreadsheets
• Text Access Reports, Forms, Word Processing
• Knowledge Base Access Multiple Access,
  Concurrent Access to Different KBs
• Language Access Access to Different Languages

17
Outside Hooks Components Continued

• Portability Exporting to Other Platforms,
  Generating Standard Files, like ASCII

18
Legal Issues

• Data Processing Services Inc. v. L. H. Smith Oil
  Corp. The Indiana Court of Appeals upheld a
  lower courts verdict that Data Processing
  Services was liable for professional malpractice.

19
Legal Issues

• Quad County Distributing Co. v. Burroughs Corp.
  This case is significant because the court held
  that a computer program is covered by the UCC
  provisions concerning the sale of products. This
  means that the injured party does not have to
  prove negligence on the part of manufacturer to
  recover damages.

20
Legal Issues

• Users
• Domain Experts
• Knowledge Engineers
• Seller Organizations

21
Users

• Who is responsible if a decision maker uses a
  defective expert system to make a decision that
  leads to damage? There are cases that the user
  could be held responsible even though he/she has
  been unaware of the fault. For example, when a
  software error caused a machine to dispense a
  lethal dose of radiation to a patient, the doctor
  was sued alongside the manufacturer of the
  machine and the institution where the machine was
  being used.

22
Users

• On the other hand, there could be consequences
  for NOT using an available an available system.
  Take the case of a nurse who does not have access
  to a doctor, and chooses not to use an expert
  system that could save a patients life. Is the
  nurse liable for negligence?
• Need for procedures?

23
Domain Experts

• Experts who do not have adequate expertise ti
  stand the test of a court challenge should
  altogether avoid getting involved in the
  development of the system.

24
Knowledge Engineer

• The knowledge engineer could damage the integrity
  of the knowledge base by his personal biases,
  negligence, and lack of understanding of the
  knowledge domain. The documentation process and
  paper trail of the knowledge engineering process
  would be of critical importance in auditing the
  quality of knowledge engineering.

25
Seller

• If Turbo Tax program is defective and gives
  wrong tax advice that leads to financial losses,
  is the company liable?

26
Seller

• Almost all software companies have limited
  warranties. As long as a software system is
  considered a product, such warranties would
  protect the seller. However, when the system is
  considered to be providing a service to
  customers, then such disclaimers could not
  prevent litigation.

27
Some Questions...

• Can management force employees to contribute
  their expertise?
• What is the value of an expert opinion in court
  when the expertise is encoded in a computer?
• Who is liable for wrong information provided by
  an ES?
• Who owns the knowledge in KB?