Fake News Detection Using Stylometry

CSC1032 Group 3 project for Data Warehousing and OLAP Achieved: First/Highest in Class

The goal was to find out whether a news article can be classified as real or fake by looking only at how it is written, not what it says. No Natural Language Processing is used. The model reads measurable writing habits such as character ratios, punctuation density, bigram frequencies and entropy, then feeds them to a Random Forest classifier.

The wider point of the project turned out to be data leakage: how easy it is to build a model that looks almost perfect while learning nothing useful.

Dataset

The ISOT Fake News dataset from the University of Victoria.

File	Articles	Label
True.csv	21,417	1 (real)
Fake.csv	23,481	0 (fake)

Total of 44,898 articles, roughly 52.3% of them fake, so the classes are reasonably balanced. Each row has a title, body text, subject and publication date, covering January 2016 to December 2017. Most articles deal with politics and international affairs.

Files

File	Purpose
main.py	Full pipeline: loading, cleaning, feature extraction, training, evaluation
True.csv	Real articles from Reuters
Fake.csv	Fake articles from flagged outlets
CSC1032 - Group 3.pdf	Written report with all figures and tables

The leakage problem

Three separate leaks were found in the raw data.

The subject column. Subject values were unique to each file. politicsNews and worldnews only ever appeared in the real set, while left-news, Middle-east, US_News and others only appeared in the fake set. A model given that column just reads the subject and skips the article entirely, so the column was dropped.

Source identifiers in the text. 99.21% of the real articles carried a (Reuters) tag. Fake articles carried their own markers such as "21st Century Wire says", "via [source]" and Twitter handles, at 33.84% and 24.4% respectively. Any model trained on this is a source detector, not a fake news detector. These are stripped out by clean_leaky_text, remove_source_vocabulary and normalize_structure, and the equivalent title patterns by clean_leaky_title.

Overly biased features. Some of the engineered features carried the answer on their own. title_uppercase_ratio alone scored around 0.58 importance and title_length around 0.17, meaning the forest was deciding almost entirely on shouty headlines rather than writing style.

Features

All features are numeric and derived from the text and title.

Category	Features	Idea behind it
Structural	text_length, title_length	Fake articles vary more in length
Character ratios	uppercase_ratio, digit_ratio, punctuation_ratio, whitespace_ratio	Fake writing leans on capitals and ellipses
Punctuation counts	exclamation_count, question_count	Captures urgency and clickbait tone
Title ratios	title_uppercase_ratio, title_exclamation, title_question	Headlines behave differently to body text
Bigrams	frequency of th, he, in, er, an, re, ed, nd, on, en	Real writing follows a stable English fingerprint
Information theory	char_entropy, unique_chars	Higher entropy suggests richer vocabulary
Readability	avg_sentence_length	Fake articles tend to use simpler sentences

Bigrams were chosen over unigrams and trigrams on purpose. A unigram has no context at all, since "tan" and "ant" look identical to it. A trigram is too sparse for a dataset this size.

Method

1. Label the two files, combine them, drop the subject column.
2. Clean source identifiers from text and titles with regex.
3. Extract the numeric features twice, once from the raw text and once from the cleaned text.
4. Split 80/20 with stratify=y and random_state=42 so the class balance holds and results are reproducible.
5. Scale with StandardScaler, fitted on the training set only and applied to the test set, so no test information leaks into training.
6. Train a Random Forest with 100 trees, max_depth=10 and min_samples_split=5.
7. Evaluate with a confusion matrix, classification report, ROC curve, AUC and feature importance.

Two studies were run so the effect of the biased features could be isolated:

• Study I: all features on the original data.
• Study II: cleaned data with exclamation_count, question_count, whitespace_ratio, title_uppercase_ratio and title_length removed.

Running it

pip install pandas numpy scikit-learn matplotlib seaborn
python main.py

Keep True.csv and Fake.csv in the same folder as main.py. The script prints both classification reports and opens the plots as it goes.

Results

Metric	Study I (original)	Study II (cleaned)
Accuracy	99.10%	79.21%
AUC	~1.00	0.875
Precision (fake / real)	0.99 / 0.99	0.81 / 0.78
Recall (fake / real)	0.99 / 0.99	0.79 / 0.79
F1 (fake / real)	0.99 / 0.99	0.80 / 0.78

Study I looks flawless. A near perfect AUC on a problem this hard is a warning sign rather than a success, and the feature importance chart confirmed it. The model was riding two features that gave the answer away.

Study II drops 19.89 percentage points of accuracy. That drop is the whole point. It is the size of the shortcut the first model was taking.

Conclusion

The cleaned model is the honest one. It scores 79.21% accuracy with an AUC of 0.875, precision and recall sitting between 0.78 and 0.81, and F1 scores between 0.78 and 0.80. Those numbers are close together across both classes, which means the model is not favouring one label over the other. Importance is spread across bigram_on, text_length, uppercase_ratio and bigram_th, so it really is reading writing style.

Stylometry does work for fake news detection. Real and fake articles carry different measurable fingerprints in their bigram distribution, length, capitalisation, punctuation and character entropy. Just as importantly, the project shows that a 99% result is worth less than a 79% result if the 99% came from a leak.

Limitations

Stylometry is not foolproof. A fake article written by a competent journalist can imitate the style of a real one, so in practice this would be paired with an NLP model rather than used alone. The dataset itself is also narrow, covering two years of mostly political news, with all the real articles coming from a single source. For consistency, uppercase_ratio arguably should have been removed alongside title_uppercase_ratio in Study II.

References

Dataset from H. Ahmed, I. Traore and S. Saad, University of Victoria. Full reference list is in the report PDF.