Astronauts of the underworld: The scientists venturing into the deep, dark Earth

Blind spiders and whip scorpions, the future of climate change, photosynthesis in the dark – here's what the deepest places on Earth can reveal about life and the universe.

Seated, I slide through a curved tunnel of stone that glistens like the inside of a throat. I feel as if I am being swallowed as I disappear into the darkness of the underworld. I grab pustule-like stalagmite globules as I crawl, now on my knees. Spiked stalactites threaten to bite, while calcite capillaries scatter over the rock walls. Caves, Phil Short tells me, are "alive". They breathe – though their entrances are often small, they exchange gases with the outside world.

I'm in Wookey Hole cave, part of a network of underground caverns in the British village of Wookey Hole, Somerset. 

"Today's a hot day," explains Short, one of the world's foremost cave explorers and divers, and underwater missions lead at Deep Research Labs. "The atmospheric pressure outside is high – but yesterday was a cold day so the atmospheric pressure in here is low." If air pressure outside the cave is greater than inside, he says, air moves into the cave and vice versa. "Another day it might be freezing cold outside and still warm in here – and it breathes the other direction."

I squeeze my body through a gap – clambering on my hands, knees and stomach – until the slanted gash in the Earth opens into a small cavern. Perched on a rock, Short looks as comfortable as you or I might be slouched in a comfy armchair. And this might not be far from the mark, since he describes the Wookey Hole cave as his "spiritual home".

Becca Burne, caving guide for Wild Wookey instructs us to turn off our head torches. We sit in complete darkness. "People swear to me they can see their hand in front of their face – but they can't," laughs Burne. Down here, under the Earth, there is utter darkness. It is still and quiet, and gifts a sense of calm only possible if you remove all the incessant stimuli of life above ground.

"[Caving] is a slow, controlled activity," says Short, "not an adrenaline sport. You work slowly, clipping on one cow's tail, past the belay, securing it, then moving the second."

Currently, tens of thousands of caves are known globally, and more are being discovered every day. In fact, many of the world's caves – including those of the UK's Mendip Hills, where I find myself now – remain unexplored.

"If you go back to the golden era of exploration – the search for the source of the Nile, the race for the South Pole – there were no satellites, no aeroplanes," says Short. "Caving is the last realm where true, pure exploration is possible. [When you enter an unexplored cave], you're going to a place on the planet where nothing – no drone, no modern technology – has been before."

And in caves, adds Short, treasures can be found, "new species, new cures for disease". Some caves are so big they are reported to have their own weather system. Some are so deep, we have yet to reach the bottom. Caves contain the secrets of human evolution, of the life that came before us, and of millennia of climate impacts. And caves are not just repositories of distant memories, but hotspots of biodiversity and endemism – entire ecosystems teaming with life.

This is what entomologist Leonidas-Romanos Davranoglou, lead entomologist for Expedition Cyclops and postdoctoral fellow at Oxford University, found when he trekked to the Cyclops Mountains in the Papuan region of Indonesia.

If you stand still for a moment, Davranoglou tells me, leeches – glistening and dark as the forest floor – approach from all directions. They are hunting you, following minute vibrations in the earth, your shadow, your breath. "In Papua, because it's so wet leeches live absolutely everywhere; on the trees, on the ground, in the bushes," Davranoglou says. In the jungle of the Cyclops Mountains all manner of venomous snakes and spiders, disease-carrying mosquitos and ticks, stalk the few that brave this largely untrodden land.

Despite the dangers, the team were determined to "undertake the most comprehensive survey this ecosystem" Davranoglou explains, and in doing so collected the first-ever photographic evidence confirming the survival of the Attenborough Echidna. They also rediscovered a bird lost to science for over 15 years, found a new genus of tree-dwelling shrimp, as well as countless new species of insects and – when one of the team fell into a sinkhole – a previously unknown cave system.

"We could see that it went far deeper. We had to crawl and, the moment we entered, bats started flying out hysterically. We thought, okay, this is a really good sign. Then we started seeing cave crickets," says Davranoglou. Cave crickets, he explains, are peculiar looking insects. They have extremely long legs and antennae, and tiny eyes. "They feel their way through the dark," he says. "Cave crickets are a telltale sign that there's a rich cave ecosystem inside."

Davranoglou and expedition leader, James Kempton returned to the cave numerous times and on the third occasion, Kempton was alone underground scouting out routes for further exploration when the Earth began to shake. Dust fell from the cracks, and the bats began flying in panic.

"Papua is one of the most tectonically active areas in the world," says Davranoglou. "We experienced earthquakes all the time. You'd see massive boulders tumbling down [the mountainside]. Then there was this massive earthquake, and [Kempton] was being shaken within an extremely narrow cavern that was full of boulders. Our students were waiting outside the cave with extreme fear to see if he would come out – they cried with happiness when he was safe."

The team discovered a "treasure trove" of underground species, including blind spiders, blind harvestman and a whip scorpion – all new to science. "We were absolutely elated because we discovered a hidden ecosystem that has so much potential. And given that we explored only the first 40m (131ft) – we just scratched the surface – who knows what lives deeper inside."

Back in Oxford, Davranoglou shows me a tray of dung beetle specimens, iridescent beauties sporting huge spiked horns. We are standing in a room full of metal cabinets, part of the Oxford University Natural History Museum's Life Collections, which hold 5.5 million insect specimens. Now the expedition is done, Davranoglou tells me, the work really begins.

"Papua is the most biodiverse island in the world," says Davranoglou. A new level of understanding of its biodiversity, he hopes, will help to inform conservation measures to protect this precious ecosystem.

"The discovery of each new species can tell you about the evolution of a whole lineage," says Davranoglou. "It can help you understand how organisms were distributed in the past, which ecological and geological factors drove the speciation of different groups. And by using this data, you can also understand factors that might affect species distributions – and their fate – in the future," he says.

In 2013, Short completed a three-month cave exploration, spending a total of 45 days underground. 12km (7 miles) long and 1.2km (0.7 miles) deep, the J2 cave system is hidden deep within the Sierra Juárez mountains of Southern Mexico.

Hauling three months' worth of expedition equipment – rebreather scuba cylinders, regulators, food and camping equipment – through dense rainforest, was a challenge in itself. "On the top flat of the mountain, we made base camp – tents scattered through the jungle, walkways in between and a big tarpaulin communal covered area with a fire pit," says Short. "About an hour's walk downhill, there was a tiny shelf in the mountainside with the entrance to the cave, about a foot and a half wide (45cm) and four feet high (1.3m)."

This unassuming entrance gave no hint of the enormous maze hidden beneath the Earth. Within just a couple of meters from the entrance, however, Short found himself abseiling 70m (230ft), carrying 40kg (88lbs) of diving equipment on his back. "Down, and again, and again, and again. Eventually, after about 700m (2,300ft) of descent, we ended up at a small chamber where [previous team members] had hung hammocks. There was a stove and some supplies," says Short.

At camp two, a tent had been prepared, and at camp three was the dive base. The first flooded section was 200m (656ft) long. "Now, it was only the dive team," says Short. Of a team of 44 people from 15 different countries, now there were just two – Short and fellow explorer, Marcine Gala – alone, for nine days.

After a further 600m (1,969ft) dive into the unknown, Short and Gala surfaced to hear the roar of a waterfall. "We found this giant, beautiful multicoloured calcite curtain. We squeezed around it and saw the river backed up like it was dammed. There was mist, like a waterfall in the jungle, in this huge chamber – where the entire river of J2 was just pouring down into the bowels of the earth,” says Short.

Using an electric drill to attach bolts to the rock wall, the pair rappelled to its base. They followed the river until it "closed down into a gap you couldn't even put your hand in – the end of the cave", says Short.

Although exploration – and not science – was the reason for being there, says expedition leader Bill Stone, mapping cave systems like this paves the way for future scientific expeditions.

"Caves need to be protected," says Hazel Barton, professor of geological sciences at the University of Alabama. Barton is a geomicrobiologist studying microbes that dwell in some of the most extreme environments on Earth – and one of the scientists that followed Stone's footsteps into the Sierra Juárez mountains.

More like this:

• The photographer at the bottom of the ocean

• The city with a labyrinth of underground caves

• Where we might find aliens in the next decade

For more than 20 years, Barton has been studying microscopic life, found deep underground, that can survive extreme starvation. Her research is furthering our understanding of antimicrobial resistance (AMR), and even what the ability to photosynthesise – in what would appear to human eyes as pitch darkness – might tell us about the possibility of extraplanetary life.

"A kilometre from the entrance, and there's still photosynthesis," says Barton, "but it's pushed into the near-infrared wavelength. There are stars that only emit in those wavelengths. So, this could help us understand how life is possible on other planets," she says.

Cave exploration, says Barton, is the closest you can come to being an astronaut without going to space. “You're the first person to see something, yours are the first footprints. 10,000 years from now, the footprints I made in Lechuguilla Cave in New Mexico, or the Tepui Caves of Venezuela, could still be there.”

Emerging from Wookey Hole into the sunlight, I find my senses exaggerated – the smell of damp foliage, the echoes of birdsong, the breeze on my skin, the warmth of the Sun. I feel as if I have awoken from a dream. "Nowadays, it's hard to impress people," says Short, "but you can come here and see something different every time."

Hundreds of cave entrances are known on Earth, the Moon, even Mars. Many have never been explored. If we dare to peer into the darkness, what might we find hiding beneath the surface?

You can hear Phil Short speaking to the BBC podcast More Wow here.

—

If you liked this story, sign up for The Essential List newsletter – a handpicked selection of features, videos and can't-miss news, delivered to your inbox twice a week

For more science, technology, environment and health stories from the BBC, follow us on Facebook, Xand Instagram.