Skip to content

Lesson 5 of 5

Ways of testing software

Listen to an AI summary of Ways of testing software

0:00 / 0:34

Three things a test can look at

A test for weaknesses looks at one of three things:

  • The source code.
  • The running software, which is the program switched on and in use.
  • The libraries that the software uses.

What is a library

A library is code that other people wrote, which a program uses to save writing the same code again. In JavaScript, require('crypto') brings in a library that makes random numbers, among other jobs. A dependency is a library that a piece of software needs to work.

A library is like a shop-bought sauce in a home-cooked meal. The meal is the software and the sauce is the library. The cook did not make the sauce, and a fault in the sauce is a fault in the meal. A weakness in a library is a weakness in the software that uses the library.

The sign-in page of the sports shop

The JavaScript code below runs at sign-in to the sports shop. Look at the library on the first line and the password on the second line.

const passcheck = require('passcheck')    // a library that checks passwords
const DATABASE_PASSWORD = 'Football2026'  // the password for the shop's database

function signIn(username, password) {
  const account = findAccount(username, DATABASE_PASSWORD)
  if (passcheck.matches(account, password)) {
    return account                        // the user is signed in
  }
  return null                             // the sign-in fails
}

The function signIn uses the database password to find the account that has the username. The line that starts with if asks the library whether the user's password is right.

The shop closes the account of a customer who asks, and a closed account must not sign in. No line of signIn checks whether the account is closed.

A tool that reads the source code

Static analysis, also called SAST, is a tool reading the source code without running the software. The tool has a list of rules, and each rule describes code that can be a weakness. One rule describes a variable with a name such as PASSWORD that is set to a string.

Static analysis reports the second line of the sign-in code. Everyone who can read the source code can read the password.

Static analysis reads every line each time the code changes. The tool has no rule about closed accounts, so static analysis does not report the missing check.

A tool that searches for secrets

Secret scanning is a tool that searches the code for passwords and other values that must stay private. Secret scanning reports the second line of the sign-in code too. Secret scanning also searches every older version of the code. The tool can find a password deleted last month.

A tool that checks the dependencies

Dependency scanning, also called SCA, is a tool that checks each dependency against lists of known vulnerabilities and harmful packages, such as the CVE list.

Each update of a library has a version number, such as 1.2. A project keeps a list of its dependencies with the version number of each one. The shop uses version 1.2 of passcheck.

In this example, version 1.2 is on the CVE list and version 1.3 has the fix. Dependency scanning reports the library and names version 1.3. Dependency scanning does not read the shop's own code, so the tool misses the password and the missing check.

A tool that tests the running software

Dynamic testing, also called DAST, is a tool sending requests to the running software. The tool checks each response for signs of a weakness.

The tool sends a sign-in request to the shop over HTTP, and the shop accepts the password. HTTP sends the request as plain text, so anyone between the browser and the server can read the password. Dynamic testing reports the weakness.

The fix is HTTPS, which is switched on in the server settings and not in the code. The tools that read the code miss the weakness. Dynamic testing never reads the source code, so dynamic testing misses the password and the missing check.

A person who reads the source code

Manual code review is the work of a person, called a reviewer, who reads the source code to find weaknesses. A reviewer often starts from what the tools report. A reviewer who knows the rules of the shop sees that signIn never checks for a closed account. None of the four tools reports the missing check.

Manual code review is slow. A person cannot read a whole project every time the code changes.

A person who attacks the running software

Penetration testing is the work of a person, called a tester, who attacks the running software with permission. The tester finds weaknesses before a real attacker does.

A tester uses tools and the tester's own ideas, so a tester can try things that dynamic testing does not. A penetration test checks one version of the software. Code that changes after the test is not tested.

The ways side by side

  1. Dependencies

    dependency scanning

  2. are used by

    Source code

    static analysis, secret scanning, manual code review

  3. runs as

    Running software

    dynamic testing, penetration testing

Each way of testing looks at one of the three parts of the software.
Static analysis (SAST)
Looks at the source code. Finds the password in the code. Misses the missing check.
Secret scanning
Looks at the source code and older versions. Finds the password in the code. Misses every other kind of weakness.
Dependency scanning (SCA)
Looks at the dependencies. Finds the vulnerability in passcheck. Misses weaknesses in the shop's own code.
Dynamic testing (DAST)
Looks at the running software. Finds the password sent over HTTP. Misses the password in the code and the missing check.
Manual code review
Looks at the source code. Finds the missing check. Misses code that the reviewer has no time to read.
Penetration testing
Looks at the running software. Finds weaknesses that a person can reach by attacking. Misses code that changes after the test.

AI in software development

A large language model is an AI program that reads and writes text, including source code. An AI chatbot uses a large language model, called a model for short.

A programmer can use a model to write source code. A model that writes source code makes decisions, such as how to check a password. A reviewer checks each decision and anything the model left out.

AI-assisted code review is a model reading the source code and reporting the weaknesses the model finds. A model reads code more as a person does than as a tool with a list of rules. A model can find a weakness that static analysis misses.

Why a person checks the results of a model

A model can report a weakness that is not in the code. A model can also miss a weakness that is in the code. A model does not always know what the software is supposed to do, such as the rule about closed accounts. A person checks each result of a model.

How the ways work together

A team does not pick one way and leave the rest. The ways work together in one plan. The team chooses how much of each way to use from the risk of the software and the time the team has.

A reviewer can start from the lines that static analysis reports. A tester can start from what a review found. A model can help with a review and with a penetration test, and a person still checks each result.

Check your understanding

  1. Name the way of testing that checks each dependency against the public list of known vulnerabilities.

    Show the answer

    Dependency scanning, which is also called SCA.

  2. Which way of testing finds that a closed account can still sign in to the sports shop?

    Show the answer

    Manual code review. A reviewer who knows the rules of the shop sees that signIn never checks for a closed account.

  3. Give one reason why a person checks the results of a large language model.

    Show the answer

    A model can report a weakness that is not in the code. A model also does not always know what the software is supposed to do.