Why is it so difficult to discover new antibiotics?

Over-reliance on and misuse of antibiotics has led to warnings of a future without effective medicines. Why is it so difficult for scientists to discover new drugs?

It's a tale of scientific discovery taught the world over: the serendipitous find of a mould that revolutionised modern medicine.

Almost 90 years ago, Alexander Fleming returned from holiday to find Penicillium on Petri dishes left in his basement laboratory at St Mary's Hospital in London.

By the 1950s, the golden age of antibiotic discovery, an array of new medicines was being found.

Today, scientists are searching for a new breakthrough, testing microbes in sources as diverse as soil, caves and Komodo dragon blood, as well as developing new, lab-made synthetic drugs.

Yet despite these remarkable advances, we are running out of effective antibiotics - the drugs that fight infection and are essential for everything from organ transplants to the treatment of food poisoning.

Deadly bacteria resistant to penicillin, or the more than 100 different antibiotics since developed, are already killing 700,000 people every year.

Unchecked, the global toll could rise to 10 million a year by 2050.

If the problem is so serious, why, in this age of incredible medical and scientific endeavour and advance, is it so difficult to get the new antibiotics the world so desperately needs?

The answer lies partly in scientific challenge and partly in the broken economy of research and development work.

Perhaps the less well known part of Fleming's story is the long period of research and collaboration which followed, before, in the 1940s, Penicillium became the world's first antibiotic.

Or that Fleming himself cautioned from the earliest days that bacteria could become resistant to drugs.

As a patient, antibiotics can seem such a simple treatment for infection, but the pills have a complex relationship with the very bacteria they are designed to destroy.

All microorganisms evolve and those that develop defences against antibiotics will survive, while the defenceless will be killed.

The more antibiotics we use, the faster the process of bacteria developing resistance becomes.

The result of misuse and overuse, in human and animal health, is a continual race to stay ahead of the superbugs.

It's easy to find chemicals that kill bacteria.

The challenge is that it's much more difficult to discover and develop substances that are not also toxic to humans.

The path from discovery to clinically approved medicine is necessarily long and the failure rate is high.

The process starts with basic research to identify organisms which produce antibiotic substances.

Thousands of possibilities will be screened - a process which in itself can take years.

Scientists look at different chemicals, combinations of chemicals and ways to weaken bacteria.

Some might try to attack the cell wall; others interfere with the way the bacterial cell functions, or with its metabolism.

When a candidate is found this must be tested on known infectious bacteria.

Then, if the results are promising, it will be tested for its possible toxicity to humans and must be produced at scale.

Only then can the years of clinical trials begin.

In total it takes around 10 to 20 years from discovery to medicine.

Of course, with complexity and uncertainty comes cost.

This is where the broken economy comes into play.

Antibiotics are not only complex to develop, the most innovative new products also cannot be sold freely.

Instead, they must be put on the shelf in reserve for serious cases - as is the case with colistin, the "drug of last resort".

This doesn't present an appealing investment opportunity, and over the past 30 years pharmaceutical companies have significantly decreased their work developing new antibacterial therapies.

No new classes of antibiotics have been invented for decades.

In fact, all the antibiotics brought to the market in the past 30 years have been variations on existing drugs discovered by 1984.

Most worryingly, it was as long ago as 1962 that the last new class of antibiotics to treat those infected by the most resistant gram-negative superbugs was discovered.

These include multi drug-resistant bacteria which can cause severe and often deadly bloodstream infections and pneumonia.

They pose a particular threat in hospitals, nursing homes and for patients treated with devices such as ventilators and catheters.

Other priorities include increasingly drug-resistant bacteria that cause more common diseases such as gonorrhoea and food poisoning caused by salmonella.

In recent years, as awareness of drug-resistant infections has increased and politicians have taken heed of the warnings long given by doctors and scientists, the public and private sectors have begun to work together to find solutions.

As of May 2017, a total of 51 antibiotics were in the clinical pipeline - around a third targeting priority pathogens, 12 families of bacteria seen as posing the greatest threat to human health.

But only a small number are innovative products - those not based on existing classes of antibiotics.

New drugs are vital but they are only part of the solution.

We also need to explore the potential for vaccines to protect against infection in the first place.

And better, more accurate diagnosis of infections could help doctors know as quickly as possible the best and most appropriate treatments.

We also need a better understanding of where drug-resistant infections are spreading, not just in people, but also in animals and the environment.

Improving hygiene in hospitals, clinics and communities across the world would help stop infection taking hold in the first place.

If we are to succeed in getting and staying ahead of superbugs we cannot rely on Fleming's luck in 1928.

More needs to be done to ensure industry and governments work together to test promising treatments and bring them to market.

Perhaps most importantly of all, we must give this miraculous and marvellous medicine the respect it deserves.

Antibiotics, old and new, are a valuable resource, to be used only when necessary for protecting and improving health.

About this piece

This analysis piece was commissioned by the BBC from an expert working for an outside organisation.

Tim Jinks is head of drug-resistant infections at the Wellcome Trust, which describes itself as a global charitable foundation working to improve health for everyone.

The Wellcome Trust is supporting antibiotic discovery through its US partnership, CARB-X. In the past year it has announced funding for 18 projects targeting the most urgent drug-resistant gram-negative bacteria, including eight potential new classes of antibiotics.

Edited by Duncan Walker