In the early 2000s, after a career as a journalist and editor, I had been offered a job working for the British government as a counter-terrorism communications consultant. It was, in some ways, the traditional tap on the shoulder – an old colleague approached me directly, asking if I would be interested in the work. I found the offer too intriguing to turn down. My job was to explain to people what intelligence agencies were doing and how they worked; to get past the Hollywood myths and make spies real.

Before I could start, I had to get security clearance. This was a mysterious process in which I answered about a dozen questions and then everything went quiet for weeks until my government pass arrived. I was in – or so I thought. But once I was through the secure glass doors of the Westminster office, I had very little to do. Nobody said anything but I got the impression that they were waiting for something.

It was then that I was befriended by a young woman. She was about 28 years old, with wide blue eyes. She said that she had just started working in the legal department. We had a brief conversation while making tea one afternoon. I didn’t think about her again until a few days later, when I ran into her on my bus after work. We sat together during the journey and decided to go for a coffee the next day.

Over the next few weeks, we often had lunch together. She asked about my family and background. I’m a sharer and I chatted away happily, right up until the moment she disappeared. And I mean she really disappeared: her phone number, her email address… everything was gone. When I asked around the office, nobody could remember her. It was as though I had met a ghost.

Almost immediately though, things changed for me. I was invited to meetings that had been closed to me before. My work began in earnest. And gradually I realised that the friendly blue-eyed office worker had been a spy. She was the last step in my background check. Of course, I’d known from the start that I would be working with spies – that was the point – but I had never suspected that she might be one. This was because fiction had never shown me a spy like her.

What little we know about spies we learn from books and films. In fact, most of what we believe comes from just two writers: Ian Fleming, who created the Bond fantasy of the dashing, charming, Etonian spy; and John le Carré, who brought us the realpolitik of bitter, jaded men in rumpled suits and unheated offices, undermining each other while duelling with Soviet agents.

There have been other successful spy novelists, of course, but our collective vision of spies still largely comes from the work of those two British men and, to a lesser extent, the books of Len Deighton and Graham Greene. Their novels form the accepted canon of the genre against which all modern spy novels are still judged. And yet all those authors failed to write believable women characters.

Fleming’s women were blow-up sex dolls with ludicrous names such as Pussy Galore and Holly Goodhead. The women in Le Carré novels were absurd in a different way, presented as either the “mothers” – his word for secretaries in mi6 offices – or Connie Sachs, an ageing, sex-mad alcoholic. Aside from an occasional minor character or traitorous wife, we don’t see other women in the books lauded as the greatest spy novels. They simply aren’t there.

I believe this is why I am constantly asked whether there really are female spies like the ones I write into my novels – normal women doing extraordinary jobs. Over and over again, I tell people they exist. I explain that the actual Q (the technical genius in the Bond novels whose job exists in real life) is a woman. And that three of the four current division heads at mi6 are female. I can’t blame them for being surprised. After all, I was surprised to meet so many women spies.

Naturally, mi6 is aware of this. Recently, it has been recruiting more women, and one of the biggest blocks it finds is that women don’t realise they could be good at spying or that they would be welcome in intelligence. One advertisement that the service ran a few years ago showed a mother with a small child and the words, “Secretly, we’re just like you.” And they are. The women I met during those years were of all backgrounds and classes. Some were approachable and open, others incredibly intimidating and formidable. I met a lot of male spies too, of course, but the scariest spy I ever met was a woman.

She had a freezing stare that seemed to stab into the most shallow and insignificant parts of my soul. I was certain she could see all my faults through my skin. She was in her sixties, weathered and disdainful. I might never have met her, but she was invited to a meeting about a major forthcoming event. Everyone else in the room – all male intelligence officers – deferred to her. She was, I suspect, the most senior person I met in intelligence during my career, although her rank was never revealed to me. All I knew for certain was that she had no time for someone like me telling her that spies should talk publicly about their work. I’m not easily intimidated but in that meeting I found myself fumbling my words and saying inane things. I couldn’t wait to get out of the room.

She was entirely different from the first female spy I encountered. In that case, it was her sheer ordinariness that made me never question her. She was so normal in her brown skirt and slightly scuffed boots, with her hair pulled back. She had a vicious sense of humour and was so convincing that I believed everything she told me. And yet every word she said was a lie. I’ve wondered for years how she could have been so smoothly deceptive at such a young age.

If I had known my history then I wouldn’t have been surprised. Female spies, especially young women, have been critical to intelligence for decades. Consider Nancy Wake, nicknamed “White Mouse” by the Germans during the Second World War. Wake was a British spy who joined the French Resistance aged 29. Working first as a courier and then as a member of the Escape Network, she helped numerous Allied airmen slip out of occupied France to safety in Spain. After fleeing France in 1943, she later parachuted back in to help British intelligence organise French guerilla groups. She was never captured and died in 2011 at the ripe old age of 98.

Then there was Virginia Hall, an American socialite who lost a leg in a hunting accident before the Second World War but still travelled to France to offer her services to the Resistance. Fiercely intelligent and utterly without fear, she eluded the Nazis throughout the war, carrying critical messages for British intelligence and uniting rebel groups, running operations that changed history. The Germans knew her only as die hinkende Frau (“the limping woman”). She became their most wanted Allied spy but was never caught.

Authors Ian Fleming and Graham Greene both served in intelligence alongside Virginia Hall during the war. They would have known about her and Nancy Wake, and the other female spies who risked – and often lost – their lives. Even though he came into the firm later, a scholar such as Le Carré would certainly have heard of them.

They knew about these female spies. Some even met them. And then they wrote those women out of their books. It was so consistent across all the male espionage writers that it couldn’t be an accident. But why would men who knew about these fascinating women choose not to represent them in novels about spies? Of course, we will never know the full truth, but I believe it was fear.

Consider that Graham Greene was born in 1904; Ian Fleming in 1908; Len Deighton in 1929; John le Carré in 1931. They lived at a time when women were in no way equal, for most of their lives unable to buy a home without a man’s permission and unable to have a bank account of their own. Often they were fired as soon as they were married. Then the war came along and everything changed.

Everyone was needed and it turned out that, among many other things, women were very good at spying. How intimidating that must have been for someone like Fleming, a writer with minimal ambition who got a job in intelligence thanks to his mother’s connections. Imagine him in a room with Virginia Hall. She must have had an icy stare like that older female spy I once met. Ruthless, brilliant, unforgiving.

When Fleming and his generation penned their novels years later, the women they wrote weren’t brave or intelligent. They were sex toys or traitors. Because these writers had worked in intelligence, people assumed they wrote what they knew. But they didn’t; they wrote what they wanted us to believe. But there’s an oddly positive twist. By ensuring nobody knew that female spies existed, those authors unintentionally made life considerably easier for the women who really hold those jobs. Yes, it’s galling to be written out of fiction but if people don’t know you exist, they never see you coming. You can slip in and out of their lives unnoticed. Like a ghost. And that’s the best gift you could give a spy.