As AI makes phishing campaigns faster and easier to scale, defenders need data platforms that can investigate suspicious domains at threat speed. Ermes used DLP indexing to accelerate key cybersecurity workloads while preserving its existing Lakehouse architecture.
faster domain queries
faster DBT materialisation
application changes
|
Customer
|
Industry
|
Platform
|
|---|---|---|
|
Ermes
|
Cybersecurity and threat intelligence
|
Delta Lake, Apache Spark and DBT on object storage
|
Generative AI has changed the economics of phishing. Attackers can produce lookalike domains, cloned pages and targeted lures at scale, compressing the time available for detection and response.
Ermes had already established a modern Lakehouse foundation for web-scale crawling and threat intelligence. As data volumes and access patterns grew, domain lookups and recurring data transformations became increasingly costly and time-consuming. The requirement was clear: improve speed and efficiency without rewriting DBT models, SQL, Spark logic or downstream applications.
DLP indexed the underlying Delta data across submission timestamp and domain, changing the physical layout while leaving the consuming workloads intact. The evaluation used a representative 2 TB sample and the same runtime environment for the comparison, including Spark 3.5.2, Delta 3.1.0, DBT 1.9.4 and eight on-demand R6a.2xlarge executors.
|
Workload
|
Before
|
With DLP
|
Outcome
|
|---|---|---|---|
|
Domain-level queries
|
30.4 sec
|
2.5 sec
|
More than 10× faster
|
|
DBT materialisation
|
Partitioned Delta
|
DLP-indexed Delta
|
More than 90% reduction
|
|
Experimental shuffle elimination
|
689 sec
|
325 sec
|
52.83% faster
|
|
Shuffle data
|
Present
|
Zero
|
Eliminated
|
Close the detection window
Reducing domain-level latency enables automated systems to assess more suspicious domains, more frequently, with richer contextual checks.
Improve analyst productivity
Threat researchers can move from a single-domain lookup to wider campaign analysis, including related infrastructure, historical activity and brand impersonation patterns.
Scale AI and ML workflows
More efficient access to large crawl tables supports faster iteration for classification, reputation scoring, anomaly detection, embeddings and retrieval-augmented analytics.
Create platform capacity
Lower elapsed time and CPU demand allow platform teams to support additional detection rules, customer views and analytical workloads without proportional infrastructure growth.
Cybersecurity analytics frequently filters on high-cardinality identifiers such as domains, URLs, hosts, IP addresses, certificate hashes and timestamps. DLP’s multidimensional indexing helps organise data across several access dimensions, making it a strong fit for threat intelligence environments where one large dataset must serve analysts, pipelines, ML systems, dashboards and customer-facing services.
Ermes demonstrated that web-scale threat intelligence can become materially faster and more cost-efficient without a disruptive platform rewrite. The result is a stronger foundation for detecting AI-powered phishing earlier, supporting richer investigations and scaling security analytics with less compute overhead.
