---
title: "AI Is Learning How to Hack. Are We Ready? AI and Cybersecurity"
description: "How AI is transforming cybercrime, exposing AI application vulnerabilities and reshaping cybersecurity. Explore real cases and an eight-layer defence framework."
url: https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence
---

Security & Risk · 7 November 2026· 19 min read

# AI Is Learning How to Hack. Are We Ready?

How Artificial Intelligence Is Transforming Cybercrime, Exposing Digital Vulnerabilities and Redefining the Future of Cybersecurity

By [Dr. Jovian Tan](https://joviantan.com/about) ·Published 7 November 2026

[![Illustration titled AI Is Learning How to Hack. Are We Ready? How AI is reshaping cybercrime and the future of cybersecurity. A glowing digital brain with a padlock chip, a globe and a row of security shields above a hooded figure facing screens of threat icons](https://joviantan.com/_img/images/insights/ai-learning-how-to-hack-cybersecurity-defence-1920.webp)

View the image full size](https://joviantan.com/images/insights/ai-learning-how-to-hack-cybersecurity-defence.png) 

Illustration, not a photograph.

**In this article**

> Artificial intelligence is not making every hacker a genius. It is changing the economics of cybercrime: helping attackers scale deception, accelerate reconnaissance and exploit the growing number of systems that now trust AI to take action.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#executive-summary)Executive summary

Artificial intelligence is changing cybersecurity in two directions at once. Criminals can use generative AI to produce more convincing social-engineering content, impersonate people, translate scams and accelerate parts of their operations. At the same time, AI applications have their own attack surface: prompts, connected tools, retrieval systems, data pipelines, model supply chains and permissions can all be manipulated or misconfigured.

The distinction matters. An AI-assisted phishing campaign is not the same thing as a prompt-injection attack against an AI assistant. The first uses AI as an attacker’s tool. The second targets the AI-enabled application itself. Organisations need to defend against both.

Real incidents already demonstrate the consequences. In 2024, [Hong Kong authorities described a case](https://www.info.gov.hk/gia/general/202406/26/P2024062600192.htm) in which an employee was deceived by a pre-recorded deepfake video conference and authorised transfers totalling about HK$200 million. Meanwhile, the [2025 Verizon Data Breach Investigations Report](https://www.verizon.com/business/resources/reports/2025-dbir-data-breach-investigations-report.pdf) found that credential abuse accounted for 22% of known initial access vectors in its relevant breach dataset, vulnerability exploitation 20%, and phishing 16%. Those figures are not AI-specific; they show why foundational controls still matter even as AI changes the threat landscape.

The practical response is not to ban AI or assume an AI detector can identify every fake. It is to build layered security: verify identity out of band, protect credentials, limit AI permissions, treat external content as untrusted, validate outputs before execution, monitor behaviour, test AI systems and prepare to contain incidents quickly.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#1-ai-is-changing-the-economics-of-cybercrime)1\. AI is changing the economics of cybercrime

For years, the effectiveness of a cyberattack depended partly on the time, expertise and resources available to the attacker. Artificial intelligence does not remove the need for access, opportunity or operational skill. It does, however, reduce the effort involved in several supporting tasks.

A criminal can use generative tools to draft fluent messages, translate them, create synthetic profile images, summarise public information about a target or help interpret technical material. The [FBI warned in December 2024](https://www.ic3.gov/psa/2024/psa241203) that criminals were using generative AI to make fraud schemes more believable and to increase the scale of their activity. The agency described AI-generated text, images, audio and video being used in social engineering, impersonation and financial fraud.

This changes the economics of deception. A message that once looked suspicious because of poor grammar may now read naturally. A fake profile may have a plausible image. A voice message may sound familiar. None of these signals proves that an attack will succeed, but they make simple visual or linguistic checks less dependable.

AI also helps defenders. Security teams can use machine learning to correlate alerts, detect unusual behaviour, summarise incidents and prioritise vulnerabilities. The technology is therefore not inherently offensive or defensive. Its effect depends on who controls it, what data it can access and what actions it is allowed to take.

**The strategic shift:** security teams must assume that attackers can produce convincing content cheaply, while ensuring that critical actions still require strong evidence of identity and authority.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#2-how-criminals-use-ai-to-attack-people-and-organisations)2\. How criminals use AI to attack people and organisations

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#more-convincing-phishing-and-social-engineering)More convincing phishing and social engineering

Phishing remains a method for manipulating people into revealing credentials, opening malicious content or authorising an action. Generative AI can help attackers tailor language to a person's role, industry, language or current business context.

The defence should not rely on spotting spelling mistakes. Staff should verify unusual requests through a separate, previously established channel, especially when a message asks for credentials, changes to payment details, access approvals or urgent transfers.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#voice-cloning-and-deepfake-video)Voice cloning and deepfake video

Synthetic audio and video can imitate a colleague, executive, family member or public figure. The danger is not that every deepfake is perfect. It is that an attacker may only need a convincing enough imitation to create urgency and bypass normal checks.

In June 2024, the [Hong Kong Government reported a case](https://www.info.gov.hk/gia/general/202406/26/P2024062600192.htm) in which a staff member joined what appeared to be a video conference with senior colleagues. Police believed the conference was pre-recorded and generated using publicly available video clips and voices. The employee authorised transfers to five local bank accounts, resulting in a reported loss of approximately HK$200 million.

The lesson is procedural: a familiar face or voice is not an authentication factor. High-value payments and sensitive instructions need independent verification, dual control and pre-agreed approval workflows. The case, and the controls it calls for, are examined in [Can You Still Trust Your Eyes?](https://joviantan.com/insights/ai-deepfakes-financial-trust-infrastructure)

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#faster-reconnaissance-and-vulnerability-research)Faster reconnaissance and vulnerability research

AI tools can help analyse public documentation, source code, configuration information and security findings. Used responsibly, similar capabilities help defenders review code and prioritise remediation. In the hands of an attacker, they may accelerate the preparation required before exploiting a weakness.

This does not mean AI can autonomously break into any system. Real compromise still depends on exploitable weaknesses, credentials, misconfigurations, human error or other access paths. Organisations should focus on reducing those opportunities through asset inventory, timely patching, secure configuration and continuous testing.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#malware-and-ransomware-support)Malware and ransomware support

Generative AI can assist with researching malware concepts, explaining code, adapting scripts and creating supporting material for an attack. It should not be assumed that every AI-generated script is functional or that AI has removed the technical barriers to sophisticated malware.

The more immediate organisational risk is often the combination of social engineering, stolen credentials, exposed remote services and unpatched systems. [Verizon's 2025 DBIR](https://www.verizon.com/business/resources/reports/2025-dbir-data-breach-investigations-report.pdf) reported ransomware in 44% of the breaches it analysed, up from 32% in the prior report. This is a general breach finding, not evidence that AI caused those ransomware incidents.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#scaling-fraud-across-languages-and-channels)Scaling fraud across languages and channels

AI can translate and adapt content quickly, enabling campaigns to reach people across languages and platforms. Fraudsters may combine synthetic identities, fake investment material, cloned voices and conversational chatbots to maintain contact with victims.

The defence is to verify the underlying organisation, payment destination and transaction independently. Do not treat a polished website, fluent conversation or realistic profile as proof of legitimacy.

[![Six cards headed Two directions of AI cyber risk: AI used to attack (phishing, impersonation, synthetic media, faster research); AI systems attacked (prompt injection, data leakage, poisoned retrieval, tool abuse); human trust exploited (urgency, authority, familiarity and process bypass); infrastructure exposed (credentials, APIs, cloud services, endpoints and data stores); AI used to defend (alert triage, anomaly detection, investigation and response); and the core principle: verify identity, restrict authority and monitor every critical action](https://joviantan.com/images/insights/ai-hack-threat-landscape.png)View the image full size](https://joviantan.com/images/insights/ai-hack-threat-landscape.png) 

AI is both an attacker's tool and a new target.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#3-attackers-are-also-targeting-ai-systems-themselves)3\. Attackers are also targeting AI systems themselves

Using AI to help conduct a conventional cyberattack is only half the story. When organisations connect AI assistants to internal documents, email, databases, software tools and APIs, the AI application itself becomes part of the attack surface.

The [2026 OWASP Top 10 for Large Language Model Applications](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10) places prompt injection, sensitive information disclosure and excessive agency among its leading risk categories. The framework is useful because it treats AI security as an application and systems-engineering problem, not simply a model-quality problem.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#prompt-injection-malicious-instructions-hidden-in-content)Prompt injection: malicious instructions hidden in content

Prompt injection occurs when attacker-controlled content influences an AI system to ignore its intended task or perform an unintended action. In a direct attack, the malicious instruction is supplied by the user. In an indirect attack, it is embedded in a webpage, email, document, ticket or other content that the AI later reads.

[MITRE ATT&CK documents indirect prompt injection](https://attack.mitre.org/techniques/T1698/) and [direct prompt injection](https://attack.mitre.org/techniques/T1699/). An important implication is that an AI assistant may process hostile text while performing an otherwise legitimate task, such as summarising an email or searching internal documents.

**Defence:** treat retrieved content as data, not authority. Keep system instructions and untrusted content logically separated where possible; restrict tools; enforce permissions outside the model; and require approval before consequential actions. Prompt wording alone is not a security boundary.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#sensitive-information-disclosure)Sensitive information disclosure

An AI application may reveal information that a user should not be able to access if retrieval and permission checks are poorly designed. This can happen when a system searches a shared document store without applying the requesting user's access rights, or when sensitive data is placed in prompts, logs or outputs without appropriate controls.

**Defence:** enforce access control at the data layer, apply user-level permissions to retrieval, minimise sensitive data sent to models, redact secrets, protect logs and test whether one user can retrieve another user's information.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#excessive-agency-when-an-assistant-has-too-much-power)Excessive agency: when an assistant has too much power

An AI agent may be connected to tools that send email, modify records, run code, approve workflows or call payment APIs. If it has broad permissions, a manipulated or mistaken output can lead to real-world consequences.

The risk is not limited to malicious prompts. Hallucinations, ambiguous instructions, software defects or compromised integrations can also trigger unintended actions.

**Defence:** apply least privilege. Give agents narrow, task-specific permissions; separate read from write access; use transaction limits; require human approval for high-impact actions; use short-lived credentials; and make revocation immediate.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#data-poisoning-and-compromised-supply-chains)Data poisoning and compromised supply chains

AI systems rely on models, datasets, plugins, libraries, retrieval indexes and external services. If these components are compromised or contaminated, system behaviour may be altered or information returned to the model may become unreliable.

**Defence:** maintain an inventory of models and dependencies, use trusted sources, review third-party integrations, protect data pipelines, validate updates, monitor changes and retain the ability to roll back a model or dataset.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#insecure-output-handling)Insecure output handling

An AI model's output should not automatically be treated as safe code, a valid database query or an authorised instruction. If an application passes generated output directly to another system without validation, it can create injection and execution risks.

**Defence:** validate and sanitise outputs, use structured schemas, apply allow-lists, parameterise database operations and keep execution environments isolated.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#retrieval-systems-and-hidden-context)Retrieval systems and hidden context

Retrieval-augmented generation (RAG) systems search external or internal content and pass relevant material to a model. Weaknesses in document ingestion, access control, chunking, metadata or vector-store permissions can expose information or allow manipulated content to influence answers.

**Defence:** authenticate sources, preserve document-level permissions through retrieval, track provenance, isolate tenants, scan ingested files and test retrieval boundaries.

[![Six numbered cards headed Where an AI application can be attacked: input and prompts (user prompts, webpages, emails and uploaded files); data and retrieval (knowledge bases, vector stores, permissions and provenance); model and context (system instructions, context windows, model supply chain); tools and APIs (email, databases, code execution and payment integrations); output and action (validation, policy checks, approvals and audit records); operations (logging, monitoring, incident response and rollback). Below: the model is only one component; the whole workflow needs security controls](https://joviantan.com/images/insights/ai-hack-attack-surface.png)View the image full size](https://joviantan.com/images/insights/ai-hack-attack-surface.png) 

The model is only one part of the attack surface.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#4-what-real-incidents-reveal)4\. What real incidents reveal

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#case-study-the-hong-kong-deepfake-video-conference-fraud)Case study: the Hong Kong deepfake video-conference fraud

The [Hong Kong Government's 26 June 2024 response to a legislative question](https://www.info.gov.hk/gia/general/202406/26/P2024062600192.htm) described a fraud case reported to police at the end of January 2024. An employee received a phishing email impersonating the UK-based chief financial officer of the employee's head office and joined a group video conference that appeared to include colleagues. Police believed the conference was pre-recorded and created using publicly available video and voice material. The employee was instructed to transfer funds to five local bank accounts and the reported loss was about HK$200 million.

The case demonstrates a failure of process as much as a failure of technology. The employee was persuaded by a convincing representation of authority. A robust payment process should not allow a video call, email or messaging thread to serve as the sole approval for a large transfer.

**What organisations should learn:**

- Confirm payment changes and urgent transfers using a known, independent channel.
- Require dual approval for material payments.
- Use pre-registered beneficiary details and change-control procedures.
- Train employees that voice and video can be synthetically generated.
- Preserve evidence and escalate suspicious requests immediately.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#the-broader-breach-picture-ai-is-not-the-only-threat)The broader breach picture: AI is not the only threat

The [2025 Verizon Data Breach Investigations Report](https://www.verizon.com/business/resources/reports/2025-dbir-data-breach-investigations-report.pdf) analysed more than 12,000 confirmed breaches and reported the following known initial access vectors in its non-error, non-misuse breach dataset: credential abuse (22%), exploitation of vulnerabilities (20%) and phishing (16%). These are not AI-attributed figures. They are a reminder that familiar security weaknesses remain highly consequential.

[![Bar chart headed Known initial access vectors in the 2025 DBIR, on a common scale: credential abuse 22%, exploitation of vulnerabilities 20%, phishing 16%. Callout: not AI-specific; general breach entry vectors, not attacks caused by AI; ransomware was present in 44% of breaches, against 32% a year earlier. Source: Verizon, 2025 Data Breach Investigations Report](https://joviantan.com/images/insights/ai-hack-dbir-initial-access.png)View the image full size](https://joviantan.com/images/insights/ai-hack-dbir-initial-access.png) 

General breach statistics, not attributed to AI.

The same report found ransomware present in 44% of breaches reviewed, compared with 32% in the previous year's report. Again, this is not a measure of AI-driven ransomware. It is useful context for why backup resilience, identity protection, patching and incident response remain essential.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#5-how-to-defend-an-eight-layer-framework)5\. How to defend: an eight-layer framework

No single product can prevent every AI-enabled attack. A resilient programme combines identity, access, data, application, infrastructure, monitoring, governance and recovery controls.

[![Eight numbered cards headed The eight-layer AI cybersecurity defence: identity (phishing-resistant MFA and verified channels); least privilege (narrow scopes, short-lived credentials, fast revocation); approval gates (dual control for payments and high-impact actions); data protection (classification, encryption, access-aware retrieval); secure AI design (untrusted-input handling and output validation); monitoring (logs, anomaly detection and response ownership); testing (red teaming, prompt-injection and permission tests); recovery (backups, containment and agent shutdown procedures)](https://joviantan.com/images/insights/ai-hack-eight-layers.png)View the image full size](https://joviantan.com/images/insights/ai-hack-eight-layers.png) 

Security is a system of controls, not a single product.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-1-verify-identity-not-appearance)Layer 1 — Verify identity, not appearance

Use phishing-resistant multi-factor authentication where practical, such as FIDO2 security keys or passkeys. Protect administrator accounts separately. For sensitive requests, verify the person through a known phone number or a second trusted channel, not the contact details supplied in the suspicious message.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-2-enforce-least-privilege)Layer 2 — Enforce least privilege

Give employees, applications and AI agents only the access required for their task. Separate read and write permissions. Restrict access to production systems, customer records, payment functions and administrative interfaces. Review permissions regularly and revoke access promptly when roles or requirements change.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-3-put-approval-gates-around-high-impact-actions)Layer 3 — Put approval gates around high-impact actions

Require human approval or dual control for high-value payments, beneficiary changes, customer-data exports, account recovery, production deployments and destructive operations. Define thresholds and escalation paths before an incident occurs.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-4-protect-data-throughout-its-lifecycle)Layer 4 — Protect data throughout its lifecycle

Classify information, encrypt it in transit and at rest, minimise data sent to external models, redact secrets and apply data-loss prevention controls. Ensure that AI retrieval systems respect the same user and tenant permissions as the underlying source repositories.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-5-secure-the-ai-application-architecture)Layer 5 — Secure the AI application architecture

Treat prompts, retrieved documents and tool outputs as untrusted inputs. Use allow-listed tools, strict schemas, input and output validation, isolated execution environments and server-side authorisation. Never rely on a model's own statement that an action is permitted as proof of permission.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-6-monitor-behaviour-and-transactions)Layer 6 — Monitor behaviour and transactions

Collect logs from identity providers, endpoints, cloud services, AI gateways, data stores and critical applications. Look for unusual access patterns, unexpected tool calls, abnormal data exports, new integrations, privilege changes and transactions that deviate from normal behaviour. Ensure alerts reach a team that can act.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-7-test-the-system-including-the-ai-layer)Layer 7 — Test the system, including the AI layer

Perform conventional vulnerability assessments and penetration tests, then add AI-specific testing: direct and indirect prompt injection, data leakage, retrieval permission boundaries, tool misuse, model and dependency provenance, and unsafe output handling. Red-team exercises should test the complete workflow, not only the model's chat interface.

### [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#layer-8-prepare-to-contain-and-recover)Layer 8 — Prepare to contain and recover

Maintain tested offline or immutable backups, incident-response playbooks, credential-revocation procedures and clear authority to suspend an AI agent or integration. Practise scenarios involving a compromised account, manipulated AI workflow, data leakage and fraudulent payment request.

[![Diagram headed A practical incident response sequence, in six steps: detect abnormal access, tool calls, exports or payments; contain by suspending the agent, account, integration or endpoint; revoke credentials, tokens, sessions and API keys; investigate by preserving logs, establishing scope and tracing actions; recover trusted systems and verify integrity; improve by fixing the root cause, updating controls and rehearsing again. Below: rehearse the sequence before you need it](https://joviantan.com/images/insights/ai-hack-incident-response.png)View the image full size](https://joviantan.com/images/insights/ai-hack-incident-response.png) 

Decide who can act, and how, before an incident.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#6-what-financial-institutions-and-digital-asset-businesses-should-prioritise)6\. What financial institutions and digital-asset businesses should prioritise

The consequences of an AI-enabled compromise can be particularly severe where systems control money, identity or digital assets.

Banks, payment providers, fintech platforms and cryptocurrency exchanges should pay special attention to:

- **Customer and staff identity:** phishing-resistant authentication, secure account recovery and privileged-access controls.
- **Transaction authorisation:** withdrawal limits, beneficiary controls, dual approval, cooling-off periods where appropriate and anomaly monitoring.
- **Wallet and custody operations:** segregation of duties, multi-party approvals, key-management controls and tested recovery procedures.
- **KYC and onboarding:** liveness and document checks supported by human review for higher-risk cases; synthetic media detection should be one signal, not the sole decision.
- **AML and fraud monitoring:** use AI to prioritise suspicious patterns, while maintaining explainability, escalation and human investigation.
- **AI integrations:** do not connect agents directly to production payment or custody functions without strict scopes, independent policy enforcement and approval gates.

For an exchange, an AI assistant that can explain a withdrawal policy is materially different from an agent that can approve withdrawals. The second needs a much stronger control environment. Model confidence is not transaction authorisation.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#7-the-standards-and-regulatory-landscape)7\. The standards and regulatory landscape

Several established resources can help organisations turn principles into controls.

**OWASP GenAI Security Project.** Its [2026 Top 10 for Large Language Model Applications](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10) identifies leading application risks including prompt injection, sensitive information disclosure, excessive agency, supply-chain weaknesses, data and model poisoning, hidden context exposure, and vector and embedding weaknesses in retrieval systems. It is a practical risk taxonomy for developers and security teams.

**CISA and international partners.** On 1 May 2026, [CISA announced joint guidance on the careful adoption of agentic AI services](https://content.govdelivery.com/accounts/USDHSCISA/bulletins/41544ff), developed with cybersecurity agencies from Australia, Canada, New Zealand and the United Kingdom, as well as the US NSA. [The guidance](https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services) emphasises limiting agent autonomy, layered defence, strong identity management, oversight, threat modelling and continuous monitoring.

**NIST AI Risk Management Framework.** [NIST's Generative AI Profile (AI 600-1)](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence), published on 26 July 2024 and updated on its publication page in April 2026, provides a voluntary, lifecycle-based approach to identifying and managing generative AI risks. It complements rather than replaces cybersecurity controls.

**MITRE ATT&CK and ATLAS.** MITRE documents adversary techniques relevant to AI-enabled systems, including direct and indirect prompt injection, and maintains [ATLAS](https://atlas.mitre.org/), a knowledge base of attacks on AI systems. On 27 October 2026, [MITRE announced plans to add AI-enabled systems as a platform in ATT&CK Enterprise](https://www.mitre.org/news-insights/news-release/mitre-attck-enterprise-add-ai-platform), with a release planned for early 2027. This reflects the growing need to describe attacks against AI systems in a structured way.

For organisations in Malaysia, the relevant obligations depend on sector, data handled, contractual commitments and whether the organisation falls within a regulated or designated critical-infrastructure category. AI security should be integrated into existing privacy, cybersecurity, financial-services and incident-reporting obligations. This article is general information, not legal advice.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#8-a-practical-30-day-action-plan)8\. A practical 30-day action plan

Period

Priority

Actions

Days 1–7

Discover

Inventory AI tools, agents, models, plugins, data sources and connected APIs. Identify systems that can send money, change records or access sensitive data.

Days 8–14

Restrict

Remove unnecessary permissions, separate read/write access, enable strong MFA and establish independent verification for payment and account changes.

Days 15–21

Test

Test prompt injection, data leakage, retrieval permissions, unsafe outputs and tool misuse. Review logging and alert coverage.

Days 22–30

Prepare

Define approval gates, incident owners, agent shutdown procedures, credential revocation and recovery exercises. Train staff using realistic deepfake and phishing scenarios.

Small businesses can begin with the basics: protect email accounts, enable MFA, patch internet-facing systems, maintain reliable backups, verify payment instructions by phone using a known number and restrict who can change supplier banking details. These controls are not glamorous, but they address common routes into organisations.

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#conclusion-in-a-world-of-synthetic-trust-verification-becomes-infrastructure)Conclusion: in a world of synthetic trust, verification becomes infrastructure

AI is making parts of cybercrime more scalable and making AI-enabled applications a new target. It can generate persuasive content, assist reconnaissance and help attackers exploit weaknesses. It can also help defenders detect anomalies, investigate incidents and respond faster.

The outcome is not predetermined. The organisations that adapt will not be those that simply buy the newest AI security product. They will be those that understand what their systems can access, restrict what they can do, verify who is asking, monitor what happens and know how to recover when controls fail.

A convincing voice is not identity. A familiar face is not authority. A model response is not permission. And an automated action is not automatically a safe action.

**As AI becomes more capable of imitating people and interacting with systems, trust must be independently verified and engineered into every critical workflow.**

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#frequently-asked-questions)Frequently asked questions

**Can AI really hack a computer by itself?** AI can assist parts of cyber operations, such as analysing code, generating content or supporting vulnerability research. A successful compromise still depends on the target's weaknesses, access conditions and the surrounding tools and processes. Claims that AI can autonomously hack any system are misleading.

**What is the biggest AI cybersecurity threat to ordinary people?** For many people, the immediate risk is more convincing impersonation and fraud: phishing, fake investment opportunities, synthetic voices and fabricated identities. Verify urgent requests independently and never share one-time passcodes or recovery codes.

**What is prompt injection?** Prompt injection is an attack in which malicious instructions supplied directly or embedded in content influence an AI system to behave outside its intended task. It is especially concerning when the AI can access private data or use tools.

**Can deepfake detectors stop voice and video scams?** Detection tools can help, but they are not reliable as a sole control. Media quality, new generation techniques and adversarial adaptation can reduce their effectiveness. Independent identity verification and payment approval controls are more dependable safeguards.

**How should a business secure AI agents?** Start with narrow permissions, separate read and write access, server-side authorisation, strict tool allow-lists, output validation, logging, human approval for high-impact actions and tested shutdown procedures.

**Does using AI for cybersecurity make a business safer?** It can improve detection and response, but it does not automatically make an organisation secure. AI tools need secure configuration, good data, human oversight and integration with established security controls.

* * *

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#sources-and-further-reading)Sources and Further Reading

1. **Federal Bureau of Investigation, Internet Crime Complaint Center, "Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud", public service announcement I-120324-PSA, 3 December 2024.** How criminals use AI-generated text, images, audio and video to make fraud more believable and to commit it on a larger scale. [Read the announcement](https://www.ic3.gov/psa/2024/psa241203).
    
2. **Hong Kong Government, "LCQ9: Combating frauds involving deepfake", 26 June 2024.** The pre-recorded video conference, the transfers to five local bank accounts and the reported loss of about HK$200 million, as described in the Secretary for Security's written reply. [Read the reply](https://www.info.gov.hk/gia/general/202406/26/P2024062600192.htm).
    
3. **Verizon Business, "2025 Data Breach Investigations Report", 2025.** Credential abuse (22%), exploitation of vulnerabilities (20%) and phishing (16%) among known initial access vectors, and ransomware in 44% of breaches, up from 32%. General breach statistics, not attributed to AI. [Read the report](https://www.verizon.com/business/resources/reports/2025-dbir-data-breach-investigations-report.pdf).
    
4. **OWASP GenAI Security Project, "OWASP Top 10 for Large Language Model Applications", 2026 release, 4 August 2026.** The ten application risk categories, from prompt injection, sensitive information disclosure and excessive agency to hidden context exposure, vector and embedding weaknesses and improper output handling. [See the project](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10).
    
5. **CISA, NSA, Australian Signals Directorate's Australian Cyber Security Centre, Canadian Centre for Cyber Security, New Zealand NCSC and UK NCSC, "Careful Adoption of Agentic Artificial Intelligence (AI) Services", 1 May 2026.** Joint guidance on agentic AI: limited autonomy, layered defence, identity management, oversight, threat modelling and continuous monitoring. [Read the guidance](https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services).
    
6. **Cybersecurity and Infrastructure Security Agency, "CISA and Partners Release Guidance on Adopting Agentic AI Services", 1 May 2026.** The announcement of the joint guidance and its key mitigations. [Read the announcement](https://content.govdelivery.com/accounts/USDHSCISA/bulletins/41544ff).
    
7. **National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" (NIST AI 600-1), 26 July 2024; page updated 8 April 2026.** A voluntary, lifecycle-based approach to managing generative AI risks. [Read the publication](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence).
    
8. **MITRE ATT&CK, "Indirect Prompt Injection", technique T1698, created 18 September 2026, last modified 3 November 2026.** Malicious instructions embedded in webpages, documents, emails, tickets or other content an AI system processes. [Read the technique](https://attack.mitre.org/techniques/T1698/).
    
9. **MITRE ATT&CK, "Direct Prompt Injection", technique T1699, created 18 September 2026, last modified 22 September 2026.** Malicious prompts supplied directly to an AI interface to manipulate its behaviour. [Read the technique](https://attack.mitre.org/techniques/T1699/).
    
10. **MITRE, "MITRE ATT&CK Enterprise to Add AI Platform", 27 October 2026.** The plan to add AI-enabled systems as a platform in ATT&CK Enterprise, with a release planned for early 2027. [Read the release](https://www.mitre.org/news-insights/news-release/mitre-attck-enterprise-add-ai-platform).
     
11. **MITRE, "ATLAS".** A living knowledge base of adversary tactics and techniques against AI and machine-learning systems. [See ATLAS](https://atlas.mitre.org/).
     

* * *

## [#](https://joviantan.com/insights/ai-learning-how-to-hack-cybersecurity-defence#editorial-note)Editorial note

This article is for general information and education. It is not legal, financial or technical security advice, and it does not replace an organisation-specific risk assessment, professional penetration test or incident-response engagement. The views expressed are editorial analysis.

Statistics are attributed to their original datasets. The Verizon figures are general breach statistics and should not be read as evidence that AI caused those incidents. The Hong Kong case is described as the Hong Kong Government reported it while it was under investigation.

Sources last checked on 7 November 2026.

Share this article

[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fjoviantan.com%2Finsights%2Fai-learning-how-to-hack-cybersecurity-defence) [X](https://x.com/intent/post?text=AI%20Is%20Learning%20How%20to%20Hack.%20Are%20We%20Ready%3F&url=https%3A%2F%2Fjoviantan.com%2Finsights%2Fai-learning-how-to-hack-cybersecurity-defence) 

About the author

[Dr. Jovian Joris Tan Chern Chian](https://joviantan.com/about), professionally known as Dr. Jovian Tan, is a Kuala Lumpur-based technology entrepreneur and the founder and Chief Technology Officer of Bitara, working across blockchain, Web3 infrastructure, AI infrastructure and digital financial infrastructure.

## Related insights

- ![Illustration titled Can You Still Trust Your Eyes? How AI deepfakes are breaking the world's trust infrastructure. A man's face is split down the middle, one half real skin and the other a glowing digital wireframe, beside a laptop showing a bank icon and stacks of coins](https://joviantan.com/_img/images/insights/ai-deepfakes-financial-trust-infrastructure-1920.webp)
    
    AI & Emerging Technology · 6 November 2026· 15 min read
    
    ### [Can You Still Trust Your Eyes?](https://joviantan.com/insights/ai-deepfakes-financial-trust-infrastructure)
    
    AI deepfakes are reshaping financial fraud: voice cloning, synthetic identities, real cases, Malaysia's warnings and the controls banks and businesses need.
    
    By Dr. Jovian Tan
    
- ![Illustration titled Ueno Bank's Collapse: What It Really Tells Us About Crypto Infrastructure. A cracked classical bank building labelled Ueno Bank faces a glowing glass vault with blockchain cubes across a bridge, with a city skyline and a rising candlestick chart behind](https://joviantan.com/_img/images/insights/ueno-bank-collapse-crypto-infrastructure-1920.webp)
    
    Digital Assets & Finance · 8 November 2026· 17 min read
    
    ### [Ueno Bank’s Collapse: What It Really Tells Us About Crypto Infrastructure](https://joviantan.com/insights/ueno-bank-collapse-crypto-infrastructure)
    
    What the resolution of Paraguay's Ueno Bank does and does not show about crypto infrastructure: governance, related-party risk, custody, Proof of Reserves and customer protection.
    
    By Dr. Jovian Tan
    
- ![Illustration titled The Financial System Is Going 24/7. But Is It Ready for a Bank Run? with the line As finance goes always-on, can liquidity, trust and regulation keep up? A glowing globe circled by a clock-like arrow stands beside a columned bank building, above glass panels holding coins, a house, gold bars and a certificate](https://joviantan.com/_img/images/insights/financial-system-24-7-bank-run-liquidity-risk-1920.webp)
    
    Digital Assets & Finance · 5 November 2026· 16 min read
    
    ### [The Financial System Is Going 24/7. But Is It Ready for a Bank Run?](https://joviantan.com/insights/financial-system-24-7-bank-run-liquidity-risk)
    
    Stablecoins, tokenised assets and instant settlement promise always-on finance. The liquidity, redemption and systemic risks that must be addressed first.
    
    By Dr. Jovian Tan
    

[All insights](https://joviantan.com/insights)