- Customer interactions: Support emails, call recordings, chat logs.
- Product information: Catalog images, spec PDFs, manuals.
- Operations: Sensor data, video surveillance, satellite imagery.
- AI applications: RAG (Retrieval-Augmented Generation) requires vectorising unstructured documents.
- Compliance: Contracts and legal documents must be searchable and auditable.
Companies that can extract value from unstructured data — through search, classification, summarisation, or generation — have a significant competitive advantage.
Common Unstructured Data Pitfalls
Watch out for these traps when handling unstructured data:
- Treating object stores as databases: S3 is not a database. Listing 10 million files in a single bucket is slow. Use proper partitioning (prefix hierarchies or partitioning tables like Iceberg/Delta).
- Skipping data classification: Many unstructured files contain PII (faces in photos, names in audio transcripts). Use AWS Macie or Azure Purview to scan and classify.
- No deduplication strategy: The same image or PDF can exist in many places. Use content hashes (MD5, SHA-256) to detect and dedupe.
- Ignoring metadata: EXIF data in photos, author info in PDFs, and GPS in videos are valuable metadata. Extract and store alongside the file.
- No cost control: Unstructured data grows fast. Set lifecycle policies (move to Glacier after 90 days, delete after 7 years). Untracked S3 bills are the #1 source of cloud cost overruns.
Quick Reference — Cheatsheet
- Image and video = unstructured. CSV = structured. JSON = semi-structured.
- 80–90% of enterprise data is unstructured — high business value.
- Object stores (S3, ADLS) are the cheapest storage.
- Vector databases (Pinecone, Milvus) power RAG and similarity search.
- NLP + computer vision + speech-to-text cover 95% of unstructured processing.
Unstructured Data Quick-Wins
Apply these patterns to handle unstructured data efficiently:
- Always store with metadata: Save file path, content hash, upload date, source, and tags alongside the file.
- Use lifecycle policies: Move cold data (no recent access) to cheaper storage tiers like Glacier.
- Generate thumbnails for images: Store a small (256px) thumbnail for fast preview while keeping the full-size original.
- Use content-hash deduplication: Compute MD5/SHA on upload; reject duplicates or use them to dedupe.
- Pre-extract features: Run OCR, embedding generation, and metadata extraction at upload time — not at query time.
Frequently Asked Questions
Which two files can be categorized as unstructured data?
Any two files that don't follow a fixed schema — typically an image (JPG/PNG) and a video (MP4), or a PDF and an MP3.
What is unstructured data?
Information without a predefined data model — images, videos, audio, PDFs, free text. ~80–90% of enterprise data is unstructured.
Structured vs semi-structured vs unstructured data?
Structured: tables. Semi-structured: JSON/XML/YAML. Unstructured: images, videos, PDFs, free text.
How is unstructured data stored?
Object stores (S3, ADLS, GCS), data lakes (Delta Lake, Iceberg), search indexes (Elasticsearch), and vector databases (Pinecone, Milvus).
How do you process unstructured data?
NLP for text (spaCy, Hugging Face), computer vision for images (OpenCV, YOLO), speech-to-text (Whisper), PDF extraction (Textract, Tesseract).
Why is unstructured data important?
It holds 80–90% of enterprise business value and powers modern AI applications (RAG, image search, voice assistants).
Fazlur Rahman
